Not getting blocked: headers, TLS fingerprints and request pacing

When a customer tells us "your IPs are blocked", we ask for a sample request. In the large majority of cases we investigate, the address is fine and the block comes from the client: a Python default user agent, headers no browser would send, a TLS handshake that announces "I am a script", or a perfectly regular request every 500 milliseconds.

Modern bot detection scores each request on many signals at once. The IP is one of them. This article goes through the others in the order in which a server sees them.

1. The TLS handshake

Before any HTTP header is sent, your client opens a TLS connection with a ClientHello message. It lists the TLS version, cipher suites, extensions, supported groups and signature algorithms — in a specific order. Each TLS library produces a characteristic ClientHello, and detection vendors fingerprint it (JA3 has been the common format for years; JA4, published this autumn, is designed to be more robust to extension reordering).

The consequence: a request claiming User-Agent: Chrome/119 but carrying the ClientHello of Python's ssl module or Go's crypto/tls is trivially inconsistent. No proxy can fix this, because TLS is end-to-end between your client and the target — the proxy only relays encrypted bytes.

What to do:

• Use a real browser (Playwright, Puppeteer) when the target is strict. Its handshake is genuine by definition.
• Use a client that impersonates browser handshakes for high-volume HTTP work. Libraries built on patched curl builds, such as curl_cffi in Python, can reproduce the ClientHello of current Chrome, Edge or Safari versions.
• Keep the claimed browser and the handshake consistent. If you impersonate Chrome 119, send Chrome 119 headers.

2. HTTP/2 settings

Most browsers talk HTTP/2 to sites that support it. The opening HTTP/2 frames — the SETTINGS values, the initial window update, the order of pseudo-headers (:method, :authority, :scheme, :path) — are also fingerprintable, and they differ between Chrome, Firefox, Safari and common libraries.

Many HTTP libraries still default to HTTP/1.1. Sending HTTP/1.1 with a Chrome user agent to a site where every real Chrome visitor uses HTTP/2 is another inconsistency. Browser impersonation libraries handle this too; plain requests cannot.

3. Headers: values and order

Browsers send a predictable set of headers, in a predictable order. A recent desktop Chrome navigating to a page sends roughly:

sec-ch-ua: "Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
sec-fetch-site: none
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
accept-encoding: gzip, deflate, br
accept-language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7

Common mistakes:

• Missing client hints. Chrome sends sec-ch-ua* headers; a "Chrome" without them stands out.
• Wrong sec-fetch-* values. A navigation and an XHR request carry different values. Copy them from the browser's network panel for the exact request type you reproduce.
• Accept-Language that contradicts the IP. A French mobile IP with en-US only is possible, but at scale it is a pattern. Match the language to the exit country.
• Header order. Many libraries sort headers or put Host and User-Agent first. Use a client that preserves insertion order and insert them in browser order.
• Stale versions. A Chrome version that is a year old is a weak signal individually and a strong one across thousands of requests. Update your user agents with each browser release.

4. Cookies and state

Real users accept cookies. A client that never returns the cookies a site sets on the first response — consent cookies, load-balancer cookies, anti-bot tokens — looks nothing like a browser. Use a cookie jar per identity, keep it for the life of the session and start a new one when you start a new identity (and, on mobile proxies, typically rotate the IP at the same moment).

5. Pacing and concurrency

Timing is the signal most often overlooked. Humans do not request a page every 500 ms for an hour, and they do not open 40 parallel connections from one address.

• Limit concurrency per IP. For a residential exit, one or two concurrent connections per session. For a dedicated mobile proxy, a handful — remember that real CGNAT addresses carry many users, but each user is slow.
• Add jitter. Replace fixed delays with random ones in a range.
• Respect the target's own signals. Retry-After headers, 429 responses and robots.txt crawl-delay directives are free information about the rate the operator tolerates.

A simple per-identity pacing loop in Python:

import random, time

class Pacer:
    """At most `rate` requests per minute for one identity, with jitter."""
    def __init__(self, rate=12, jitter=0.4):
        self.interval = 60.0 / rate
        self.jitter = jitter
        self.next_at = 0.0

    def wait(self):
        now = time.monotonic()
        if now < self.next_at:
            time.sleep(self.next_at - now)
        spread = self.interval * self.jitter
        self.next_at = time.monotonic() + self.interval + random.uniform(-spread, spread)

pacer = Pacer(rate=12)
for url in urls:
    pacer.wait()
    fetch(url)

On 429, back off exponentially and honour Retry-After when present. Hammering through throttles is how addresses end up on block lists — for you and for everyone else sharing them.

6. Navigation patterns

Detection systems also look at the shape of a session: did the visitor load the page's CSS and images, arrive via a link with a plausible Referer, visit category pages before product pages? You do not need to simulate a full human journey for every request, but you should avoid the most artificial patterns — thousands of deep links with no referrer, at uniform intervals, never fetching a static asset.

A pre-flight checklist

Before blaming the IP, check:

1. Does the TLS fingerprint match the browser you claim to be?
2. Are you speaking HTTP/2 to an HTTP/2 site?
3. Are headers complete, current and in browser order?
4. Does Accept-Language fit the exit country?
5. Do you keep and return cookies within an identity?
6. Is concurrency per IP low, with jittered delays?
7. Do you back off on 429 and honour Retry-After?

If all seven are in order and you are still blocked, then it is time to look at the IP type — and at that point, the difference between datacentre, residential and mobile ranges becomes very visible. The limits and best practices documentation lists the concurrency and rate figures we recommend for each of our products.