HTTP Proxy vs SOCKS5: Who Owns the TCP Connection

A latency regression that looked like a proxy being slow, but was really about which process owned the TCP connection — and therefore which connection pool got to reuse it.

The setup

A Python service (httpx, async) forwards requests to backends that live on another network, reachable only over a Tailscale tailnet. The backends sit behind a reverse proxy roughly 150 ms away.

Before: every service pod ran a Tailscale sidecar container in kernel mode. Tailnet routes landed in the pod's own network namespace, so the app dialled the backend directly. The client's own connection pool held those connections and reused them — dozens of requests per connection.

The problem with that: each sidecar registers its own tailnet device. With autoscaling, device count tracked pod count, and the tailnet has a device cap. Scale up enough times and you hit it.

After: one shared gateway Deployment with a fixed replica count runs tailscaled instead. Pods reach the tailnet through it via HTTP_PROXY. Device count now tracks replicas, not pods. Problem solved.

Latency went up by about a second.

Two things have to be true at once

The regression needs both of these, and neither is obvious on its own.

1. httpx does not use CONNECT for http:// URLs.

With a proxy configured, the shape of the connection depends entirely on the scheme:

import httpx, httpcore

c = httpx.AsyncClient(base_url="http://10.0.0.5:8000")   # HTTP_PROXY set
pool = c._transport_for_url(httpx.URL("http://10.0.0.5:8000/v1/x"))._pool

pool.create_connection(httpcore.Origin(b"http",  b"10.0.0.5",    8000))
# -> AsyncForwardHTTPConnection

pool.create_connection(httpcore.Origin(b"https", b"example.com", 443))
# -> AsyncTunnelHTTPConnection

• https:// gets a tunnel: one CONNECT, then an opaque byte pipe. The client's connection reaches the origin end to end.
• http:// gets forward proxying: the client sends POST http://host:port/path HTTP/1.1 with an absolute URI, and the proxy terminates that request and re-originates it.

Forward proxying means one request becomes two TCP connections with two different owners. The client pool only ever sees the near one.

This is easy to get backwards, because testing a proxy with curl -p exercises the CONNECT path — which does reuse connections perfectly well. It just isn't the path a plaintext http:// client takes.

2. tailscaled's forward proxy pools two connections per host.

The relevant code in cmd/tailscaled/proxy.go is short enough to quote:

rp := &httputil.ReverseProxy{
    Director: func(r *http.Request) {}, // no change
    Transport: &http.Transport{
        DialContext: dialer,
    },
}

A http.Transport constructed as a struct literal takes Go's defaults for everything unset. The one that matters:

const DefaultMaxIdleConnsPerHost = 2

So each gateway replica keeps two idle upstream connections per destination host. A third concurrent request dials a new one, and when it finishes it is closed rather than pooled.

At any real concurrency, that means nearly every request opens a fresh TCP connection across the 150 ms link, paying a handshake and TCP slow start it should not have paid.

The difference, drawn

Watch it happen

Both lanes get the same request stream. The only difference is who owns the connection crossing the slow link. Push concurrency past two and the HTTP proxy lane stops reusing anything.

Red is the handshake and slow start on a freshly dialled connection. The ratio on the left converges on ~1.0 — which is exactly the symptom that showed up in the backend proxy's downstream_rq_total / downstream_cx_total.

Measured on a real link: the same large POST took ~1000 ms on a fresh connection against ~200 ms on a reused one. Around 800 ms of that is TCP slow start. Slow start is per-connection, so it only becomes visible when connections stop being reused.

Why SOCKS5 fixes it

SOCKS5 is not a smarter HTTP proxy. It is not an HTTP proxy at all — it negotiates a destination, then relays bytes:

ss := &socks5.Server{
    Logf:   logger.WithPrefix(logf, "socks5: "),
    Dialer: dialer.UserDial,
}

One client TCP stream maps 1:1 onto one backend TCP stream. The proxy never parses or terminates HTTP, so keep-alive is negotiated between the client and the backend, and the client's pool is the only pool in the path.

In httpx that means:

pool = client._transport_for_url(url)._pool   # AsyncSOCKSProxy
pool.create_connection(origin)                # AsyncSocks5Connection

AsyncSOCKSProxy keys its pool on the target origin, not the proxy, so max_keepalive_connections and keepalive_expiry apply per destination, as you would want.

Two practical notes:

• It needs an extra dependency (httpx[socks], which pulls socksio). httpx raises ImportError at client construction against a socks5:// proxy without it — so the dependency has to ship before the config flips.
• Setting HTTP_PROXY=socks5://… as an ambient env var affects every library in the process. requests needs PySocks for SOCKS and will raise InvalidSchema without it. Passing proxy= explicitly to the one client that needs it is safer than overloading the environment.

The second-order bug: keepalive_expiry

Once the client owns the connection again, a setting that was previously irrelevant starts to matter.

httpx defaults keepalive_expiry to 5 seconds. Passing an explicit httpx.Limits(...) without naming it inherits that default silently:

# 5s expiry, whether you meant it or not
limits = httpx.Limits(max_connections=2000, max_keepalive_connections=70)

# what you probably want for a low-rate, high-RTT upstream
limits = httpx.Limits(max_connections=2000, max_keepalive_connections=70,
                      keepalive_expiry=300.0)

Whether 5 s is enough depends on the gap between requests per client instance, per destination — not on aggregate throughput. A service handling thousands of requests per second can still have each individual pod sitting idle for a minute at a time, if it is scaled wide enough.

For roughly Poisson arrivals, the share of requests landing on a cold connection is exp(-T / gap):

Note what the third row shows: past a certain sparsity, no timeout saves you. A client seeing one request every ~9 minutes for a destination will be cold essentially always. That is a capacity-shape problem — too many instances for the traffic — not a timeout problem.

Takeaways

• http:// through an HTTP proxy is not tunnelled. The proxy terminates and re-originates. Your client's pool stops at the proxy, and whatever pooling happens past it is the proxy's business, not yours.
• Test the path your client actually takes. A curl -p test exercises CONNECT and will look healthy while the real traffic churns connections.
• Go's http.Transport defaults are conservative. DefaultMaxIdleConnsPerHost = 2 is fine for a browser-shaped workload and badly wrong for a shared egress proxy. Any struct literal that omits it inherits it.
• SOCKS5 is the right tool when you want the client to keep owning its connection. It is lower-level than an HTTP proxy, and here that is precisely the advantage.
• keepalive_expiry is sized by the per-instance idle gap, not by request duration or aggregate rate.
• Connection reuse is invisible until it stops. Handshake and slow-start costs are per-connection; a change that quietly ends reuse reads as "the network got slower".