[client] agentproxy PoC: in-process DNS notes + startup/request logging

Clarify and make visible that the embedded client resolves the upstream entirely in-process. In userspace/netstack mode NetBird's DNS server runs "via memory" (packet hooks on the netstack device), so hostname lookups go over the tunnel and are answered from the network map with no changes to the host resolver and no root. Add operational visibility: - initialize the shared NetBird logger (NB_LOG_LEVEL, default info) - log overlay IP/FQDN, management/signal state and peer count on connect - preflight: resolve+dial the upstream over the tunnel once at startup - per-request access logs and per-dial resolution/connection logs Document the resolution mechanism and troubleshooting in the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GAryvqRK7SP8JWZvVSxyfd
2026-08-30 18:41:37 -07:00 · 2026-08-27 09:05:49 +00:00
parent a285f64d89
commit 97fdb70afd
2 changed files with 170 additions and 14 deletions
--- a/client/embed/example/agentproxy/README.md
+++ b/client/embed/example/agentproxy/README.md
@@ -85,10 +85,65 @@ AI agent ──HTTP──▶ 127.0.0.1:8080 (agentproxy)
 ```

 The reverse proxy's `Transport.DialContext` is the embedded client's dialer, so
-every upstream connection is established inside the WireGuard tunnel. The
-upstream hostname is resolved by NetBird's magic DNS via the netstack dialer,
-and the outgoing `Host` header is set to the upstream host so TLS SNI and the
-Agent Network's identity-aware routing see the real endpoint.
+every upstream connection is established inside the WireGuard tunnel, and the
+outgoing `Host` header is set to the upstream host so TLS SNI and the Agent
+Network's identity-aware routing see the real endpoint.
+
+### DNS resolution happens *inside* the process — no root, no `/etc/resolv.conf`
+
+This is the important part for locked-down hosts. The proxy **never touches the
+system resolver**. In userspace/netstack mode NetBird runs its DNS server "via
+memory": it registers UDP/TCP packet hooks on the in-process netstack device
+(`ServiceViaMemory`) for a reserved tunnel DNS IP. When the netstack dialer
+resolves the upstream hostname it sends the DNS query *over the tunnel* to that
+IP, where the hook answers it from the NetBird network map:
+
+```
+client.Dial("mirror.netbird.ai:443")
+  └─ netstack Net.LookupContextHost  (in-process)
+       └─ DNS query to the reserved tunnel DNS IP
+            └─ ServiceViaMemory packet hook answers from the network map
+                 └─ overlay IP → netstack dials it through WireGuard
+```
+
+Because the resolver lives in the netstack and is fed by the network map, the
+process resolves NetBird names on its own — it does **not** rely on the host's
+`/etc/resolv.conf`, systemd-resolved, or any change that would need root. (The
+`netbird` daemon's `noop host manager` in netstack mode only means it skips
+editing the OS resolver; the in-memory DNS server still answers in-process.)
+This is the same mechanism the WASM client and the Agent Network reverse proxy
+rely on.
+
+For the upstream name to resolve, it must be known to your NetBird DNS (a peer
+FQDN, a custom DNS record/zone, or a matching nameserver group) and a policy
+must grant this peer access to the agent network.
+
+## Logs and troubleshooting
+
+The process is deliberately noisy about what it is doing:
+
+- On connect it logs the assigned overlay IP/FQDN, management/signal state, and
+  the peer count.
+- A **preflight** step resolves and dials the upstream over the tunnel once at
+  startup and logs the result, so a DNS/policy problem shows up immediately
+  rather than on the first request.
+- Each proxied request logs a `-> METHOD /path` / `<- METHOD /path STATUS (dur)`
+  pair, and each tunnel dial logs the resolved local/remote tunnel addresses.
+
+Example startup:
+
+```
+connected to NetBird network: ip=100.72.0.5 fqdn=agent-proxy.netbird.cloud management=true signal=true peers=3
+preflight: resolving and dialing mirror.netbird.ai:443 over the tunnel ...
+preflight: reached mirror.netbird.ai:443 (tunnel 100.72.0.5:52344 -> 100.72.0.9:443)
+listening on http://127.0.0.1:8080 -> https://mirror.netbird.ai (over NetBird)
+```
+
+Set `NB_LOG_LEVEL=debug` (or `trace`) to surface the embedded client's own
+management/signal/relay handshakes, DNS setup, and per-peer connection detail.
+If the preflight cannot reach the endpoint, the two most common causes are that
+the upstream name is not resolvable in your NetBird DNS, or that no policy grants
+this peer access to the agent network.

 > **Proof of concept.** Intended to demonstrate the embedded-client + local
 > forward-proxy pattern for rootless environments. Not hardened for production.
--- a/client/embed/example/agentproxy/main.go
+++ b/client/embed/example/agentproxy/main.go
@@ -95,7 +95,16 @@ func run() error {
 	mgmtURL := firstNonEmpty(*mgmtFlag, os.Getenv("NB_MANAGEMENT_URL"), profilemanager.DefaultManagementURL)
 	deviceName := firstNonEmpty(*deviceFlag, os.Getenv("NB_DEVICE_NAME"), "agent-proxy")
 	setupKey := os.Getenv("NB_SETUP_KEY")
-	logLevel := firstNonEmpty(os.Getenv("NB_LOG_LEVEL"), "warn")
+	logLevel := firstNonEmpty(os.Getenv("NB_LOG_LEVEL"), "info")
+
+	// Initialize the shared NetBird logger so the embedded engine's own logs
+	// (management/signal/relay handshakes, DNS setup, peer connectivity) are
+	// formatted and level-filtered consistently and land on stderr. Set
+	// NB_LOG_LEVEL=debug (or trace) to see in-tunnel DNS resolution and per-peer
+	// connection detail.
+	if err := util.InitLog(logLevel, util.LogConsole); err != nil {
+		return fmt.Errorf("init log: %w", err)
+	}

 	opts := netbird.Options{
 		DeviceName:    deviceName,
@@ -134,7 +143,12 @@ func run() error {
 	if err := client.Start(startCtx); err != nil {
 		return fmt.Errorf("start netbird client: %w", err)
 	}
-	log.Printf("connected to NetBird network")
+	logConnected(client)
+
+	// Preflight: resolve and dial the upstream through the tunnel once, up
+	// front, so connectivity and in-process DNS resolution are verified (and
+	// visible in the logs) before any real request arrives.
+	preflight(client, upstream)

 	proxy := newAgentProxy(client, upstream)

@@ -177,7 +191,10 @@ func run() error {
 // client so the connection travels inside the encrypted overlay. Hostnames are
 // resolved through NetBird's magic DNS by the netstack dialer, so a private
 // name such as mirror.netbird.ai reaches the reverse proxy peer directly.
-func newAgentProxy(client *netbird.Client, upstream *url.URL) *httputil.ReverseProxy {
+//
+// The returned handler logs every request (method, path, upstream status,
+// duration) so it is obvious the proxy is receiving and forwarding traffic.
+func newAgentProxy(client *netbird.Client, upstream *url.URL) http.Handler {
 	proxy := httputil.NewSingleHostReverseProxy(upstream)

 	// Preserve the standard single-host director but force the outgoing Host
@@ -195,7 +212,7 @@ func newAgentProxy(client *netbird.Client, upstream *url.URL) *httputil.ReverseP
 	// client.NewHTTPClient so we keep full control of the reverse-proxy
 	// transport knobs.
 	proxy.Transport = &http.Transport{
-		DialContext:           dialWithTimeout(client.DialContext, dialTimeout),
+		DialContext:           loggingDialer(client.DialContext, dialTimeout),
 		ForceAttemptHTTP2:     true,
 		MaxIdleConns:          100,
 		IdleConnTimeout:       90 * time.Second,
@@ -208,19 +225,103 @@ func newAgentProxy(client *netbird.Client, upstream *url.URL) *httputil.ReverseP
 		http.Error(w, "agent network upstream unreachable: "+err.Error(), http.StatusBadGateway)
 	}

-	return proxy
+	return accessLog(proxy)
 }

-// dialWithTimeout applies a bounded timeout to the connection-establishment
-// phase only, leaving the request/response body streaming unbounded.
-func dialWithTimeout(dial func(ctx context.Context, network, addr string) (net.Conn, error), d time.Duration) func(ctx context.Context, network, addr string) (net.Conn, error) {
+// accessLog wraps a handler and logs one line per request with the resulting
+// upstream status code and total duration.
+func accessLog(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		start := time.Now()
+		rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}
+		log.Printf("-> %s %s", r.Method, r.URL.Path)
+		next.ServeHTTP(rec, r)
+		log.Printf("<- %s %s %d (%s)", r.Method, r.URL.Path, rec.status, time.Since(start).Truncate(time.Millisecond))
+	})
+}
+
+// statusRecorder captures the response status code for the access log.
+type statusRecorder struct {
+	http.ResponseWriter
+	status int
+}
+
+func (s *statusRecorder) WriteHeader(code int) {
+	s.status = code
+	s.ResponseWriter.WriteHeader(code)
+}
+
+// loggingDialer wraps the embedded client's dialer. It applies a bounded
+// timeout to connection establishment (not the full request lifetime) and logs
+// each dial — including the tunnel-side local/remote addresses on success —
+// which surfaces the in-process DNS resolution and reachability of the target.
+func loggingDialer(dial func(ctx context.Context, network, addr string) (net.Conn, error), d time.Duration) func(ctx context.Context, network, addr string) (net.Conn, error) {
 	return func(ctx context.Context, network, addr string) (net.Conn, error) {
-		ctx, cancel := context.WithTimeout(ctx, d)
+		dialCtx, cancel := context.WithTimeout(ctx, d)
 		defer cancel()
-		return dial(ctx, network, addr)
+
+		start := time.Now()
+		conn, err := dial(dialCtx, network, addr)
+		if err != nil {
+			log.Printf("dial %s %s failed after %s (resolution/connection over tunnel): %v",
+				network, addr, time.Since(start).Truncate(time.Millisecond), err)
+			return nil, err
+		}
+		log.Printf("dial %s %s ok in %s (tunnel %s -> %s)",
+			network, addr, time.Since(start).Truncate(time.Millisecond),
+			conn.LocalAddr(), conn.RemoteAddr())
+		return conn, nil
 	}
 }

+// logConnected prints a concise summary of the tunnel state right after the
+// client reports up: the local overlay IP/FQDN, control-plane connectivity, and
+// how many peers are in the network map.
+func logConnected(client *netbird.Client) {
+	status, err := client.Status()
+	if err != nil {
+		log.Printf("connected to NetBird network (status unavailable: %v)", err)
+		return
+	}
+	log.Printf("connected to NetBird network: ip=%s fqdn=%s management=%v signal=%v peers=%d",
+		status.LocalPeerState.IP, status.LocalPeerState.FQDN,
+		status.ManagementState.Connected, status.SignalState.Connected, len(status.Peers))
+}
+
+// preflight resolves and dials the upstream endpoint once over the tunnel so
+// any DNS or connectivity problem is reported at startup rather than on the
+// first real request. It never fails the process — a proxy that starts but
+// cannot yet reach the endpoint is still useful once policy/peers converge.
+func preflight(client *netbird.Client, upstream *url.URL) {
+	addr := hostPort(upstream)
+	ctx, cancel := context.WithTimeout(context.Background(), dialTimeout)
+	defer cancel()
+
+	log.Printf("preflight: resolving and dialing %s over the tunnel ...", addr)
+	conn, err := client.DialContext(ctx, "tcp", addr)
+	if err != nil {
+		log.Printf("preflight: could not reach %s yet: %v", addr, err)
+		log.Printf("preflight: this is not fatal; check that %q resolves in your NetBird DNS "+
+			"and that a policy grants this peer access to the agent network", upstream.Host)
+		return
+	}
+	log.Printf("preflight: reached %s (tunnel %s -> %s)", addr, conn.LocalAddr(), conn.RemoteAddr())
+	_ = conn.Close()
+}
+
+// hostPort returns the upstream host with an explicit port, defaulting to the
+// scheme's well-known port when the URL omits it.
+func hostPort(u *url.URL) string {
+	if u.Port() != "" {
+		return u.Host
+	}
+	port := "443"
+	if u.Scheme == "http" {
+		port = "80"
+	}
+	return net.JoinHostPort(u.Hostname(), port)
+}
+
 // interactiveLogin drives the standard NetBird OAuth flow (PKCE where a
 // desktop session is available, device-code flow otherwise) and returns the
 // JWT to hand to the embedded client. This is the same flow the `netbird