--- title: "HTTP Proxy vs SOCKS5: Who Owns the TCP Connection" date: 2026-09-17T10:00:00.000Z lang: "en" canonical: "https://pokgak.xyz/notes/http-proxy-vs-socks5-connection-reuse/" tags: ["networking","tailscale","http","kubernetes","performance"] --- # HTTP Proxy vs SOCKS5: Who Owns the TCP Connection A latency regression that looked like a proxy being slow, but was really about which process owned the TCP connection — and therefore which connection pool got to reuse it. ## The setup A Python service (httpx, async) forwards requests to backends that live on another network, reachable only over a [Tailscale](https://tailscale.com) tailnet. The backends sit behind a reverse proxy roughly 150 ms away. **Before:** every service pod ran a Tailscale sidecar container in kernel mode. Tailnet routes landed in the pod's own network namespace, so the app dialled the backend directly. The client's own connection pool held those connections and reused them — dozens of requests per connection. **The problem with that:** each sidecar registers its own tailnet device. With autoscaling, device count tracked pod count, and the tailnet has a device cap. Scale up enough times and you hit it. **After:** one shared gateway Deployment with a fixed replica count runs `tailscaled` instead. Pods reach the tailnet through it via `HTTP_PROXY`. Device count now tracks replicas, not pods. Problem solved. Latency went up by about a second. ## Two things have to be true at once The regression needs both of these, and neither is obvious on its own. **1. httpx does not use `CONNECT` for `http://` URLs.** With a proxy configured, the shape of the connection depends entirely on the scheme: ```python import httpx, httpcore c = httpx.AsyncClient(base_url="http://10.0.0.5:8000") # HTTP_PROXY set pool = c._transport_for_url(httpx.URL("http://10.0.0.5:8000/v1/x"))._pool pool.create_connection(httpcore.Origin(b"http", b"10.0.0.5", 8000)) # -> AsyncForwardHTTPConnection pool.create_connection(httpcore.Origin(b"https", b"example.com", 443)) # -> AsyncTunnelHTTPConnection ``` - `https://` gets a **tunnel**: one `CONNECT`, then an opaque byte pipe. The client's connection reaches the origin end to end. - `http://` gets **forward proxying**: the client sends `POST http://host:port/path HTTP/1.1` with an absolute URI, and the proxy *terminates* that request and re-originates it. Forward proxying means one request becomes two TCP connections with two different owners. The client pool only ever sees the near one. This is easy to get backwards, because testing a proxy with `curl -p` exercises the `CONNECT` path — which does reuse connections perfectly well. It just isn't the path a plaintext `http://` client takes. **2. `tailscaled`'s forward proxy pools two connections per host.** The relevant code in `cmd/tailscaled/proxy.go` is short enough to quote: ```go rp := &httputil.ReverseProxy{ Director: func(r *http.Request) {}, // no change Transport: &http.Transport{ DialContext: dialer, }, } ``` A `http.Transport` constructed as a struct literal takes Go's defaults for everything unset. The one that matters: ```go const DefaultMaxIdleConnsPerHost = 2 ``` So each gateway replica keeps **two** idle upstream connections per destination host. A third concurrent request dials a new one, and when it finishes it is closed rather than pooled. At any real concurrency, that means nearly every request opens a fresh TCP connection across the 150 ms link, paying a handshake and TCP slow start it should not have paid. ## The difference, drawn
app podgateway replicabackendBEFORE — per-pod sidecarclient pool+ tailscaled TUN:8000one TCP stream — the client pool reuses itmany requests per connectionHTTP PROXY — forward modeclient poolkeepalive NReverseProxyMaxIdleConnsPerHost 2:8000leg A — local~4 ms coldleg B — remote150 ms + slow startHTTP terminated and re-originated hereSOCKS5 — relayclient poolkeepalive Nsocks5.Serverrelays 1:1:8000one TCP stream again — the client pool governs reuse

The gateway is not the problem; the termination is. Under SOCKS5 the gateway still carries every byte, but it relays the client's stream instead of ending it and dialling its own.

## Watch it happen Both lanes get the same request stream. The only difference is who owns the connection crossing the slow link. Push concurrency past two and the HTTP proxy lane stops reusing anything.
HTTP proxy forward mode

Gateway owns the far leg and pools two idle connections per host.

remote leg
0requests
0dials
—req / conn
SOCKS5 relay

Client owns the stream end to end; its own pool does the reuse.

remote leg
0requests
0dials
—req / conn
Red is the handshake and slow start on a freshly dialled connection. The ratio on the left converges on ~1.0 — which is exactly the symptom that showed up in the backend proxy's `downstream_rq_total / downstream_cx_total`. Measured on a real link: the same large POST took **~1000 ms** on a fresh connection against **~200 ms** on a reused one. Around 800 ms of that is TCP slow start. Slow start is per-connection, so it only becomes visible when connections stop being reused. ## Why SOCKS5 fixes it SOCKS5 is not a smarter HTTP proxy. It is not an HTTP proxy at all — it negotiates a destination, then relays bytes: ```go ss := &socks5.Server{ Logf: logger.WithPrefix(logf, "socks5: "), Dialer: dialer.UserDial, } ``` One client TCP stream maps 1:1 onto one backend TCP stream. The proxy never parses or terminates HTTP, so keep-alive is negotiated between the client and the backend, and the client's pool is the only pool in the path. In httpx that means: ```python pool = client._transport_for_url(url)._pool # AsyncSOCKSProxy pool.create_connection(origin) # AsyncSocks5Connection ``` `AsyncSOCKSProxy` keys its pool on the *target* origin, not the proxy, so `max_keepalive_connections` and `keepalive_expiry` apply per destination, as you would want. Two practical notes: - It needs an extra dependency (`httpx[socks]`, which pulls `socksio`). httpx raises `ImportError` at *client construction* against a `socks5://` proxy without it — so the dependency has to ship before the config flips. - Setting `HTTP_PROXY=socks5://…` as an ambient env var affects every library in the process. `requests` needs PySocks for SOCKS and will raise `InvalidSchema` without it. Passing `proxy=` explicitly to the one client that needs it is safer than overloading the environment. ## The second-order bug: `keepalive_expiry` Once the client owns the connection again, a setting that was previously irrelevant starts to matter. httpx defaults `keepalive_expiry` to **5 seconds**. Passing an explicit `httpx.Limits(...)` without naming it inherits that default silently: ```python # 5s expiry, whether you meant it or not limits = httpx.Limits(max_connections=2000, max_keepalive_connections=70) # what you probably want for a low-rate, high-RTT upstream limits = httpx.Limits(max_connections=2000, max_keepalive_connections=70, keepalive_expiry=300.0) ``` Whether 5 s is enough depends on the gap between requests **per client instance, per destination** — not on aggregate throughput. A service handling thousands of requests per second can still have each individual pod sitting idle for a minute at a time, if it is scaled wide enough. For roughly Poisson arrivals, the share of requests landing on a cold connection is `exp(-T / gap)`:

cold — redial, handshake, slow startwarm — reused

Note what the third row shows: past a certain sparsity, no timeout saves you. A client seeing one request every ~9 minutes for a destination will be cold essentially always. That is a capacity-shape problem — too many instances for the traffic — not a timeout problem. ## Takeaways - **`http://` through an HTTP proxy is not tunnelled.** The proxy terminates and re-originates. Your client's pool stops at the proxy, and whatever pooling happens past it is the proxy's business, not yours. - **Test the path your client actually takes.** A `curl -p` test exercises `CONNECT` and will look healthy while the real traffic churns connections. - **Go's `http.Transport` defaults are conservative.** `DefaultMaxIdleConnsPerHost = 2` is fine for a browser-shaped workload and badly wrong for a shared egress proxy. Any struct literal that omits it inherits it. - **SOCKS5 is the right tool when you want the client to keep owning its connection.** It is lower-level than an HTTP proxy, and here that is precisely the advantage. - **`keepalive_expiry` is sized by the per-instance idle gap**, not by request duration or aggregate rate. - **Connection reuse is invisible until it stops.** Handshake and slow-start costs are per-connection; a change that quietly ends reuse reads as "the network got slower".