Web scraping

The problem

Most websites rate-limit or block datacentre ranges outright, and many fingerprint the request pattern of a single IP. A crawler that sends thousands of requests from a few addresses ends up collecting block pages instead of data.

Spreading requests across a large residential pool keeps the per-IP request rate close to that of a human visitor, while sticky sessions keep multi-page flows such as pagination on one address.

Configuration

Product

Rotating residential, mobile proxies for the strictest domains

Rotation

Per request for listing and detail pages

Session

-session-<id>-ttl-5 for paginated or stateful flows

Targeting

Country of the target site; eu when location does not matter

Retries

Up to 2 retries on 403/429/5xx, each through a new IP

Protocol

HTTP for HTTP clients, SOCKS5 for headless browsers

Example

import { ProxyAgent, fetch } from 'undici';

const LOGIN = 'px7h2k9m4q';
const PASSWORD = 'Rq4Tn8Vw2Lx6Bz9C';
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36';

function agent(country = 'de') {
  return new ProxyAgent(`http://${LOGIN}-country-${country}:${PASSWORD}@resi.gw.proxuno.com:7000`);
}

async function get(url, attempt = 1) {
  const res = await fetch(url, { dispatcher: agent(), headers: { 'user-agent': UA } });
  if ([403, 429, 502, 503].includes(res.status) && attempt <= 2) {
    await new Promise((r) => setTimeout(r, 1000 * attempt));
    return get(url, attempt + 1);
  }
  return { status: res.status, body: await res.text() };
}

console.log((await get('https://example.com/catalogue?page=1')).status);

Example credentials. Replace them with yours from the dashboard.

Best practices

• Honour robots.txt and the site's terms, and cache pages you have already fetched.
• Keep a realistic header set (user agent, Accept, Accept-Language) and keep it consistent within a session.
• Measure success per domain. A falling success rate is a signal to slow down, not to add more IPs.
• Compress responses (Accept-Encoding: gzip, br) — residential traffic is billed on the bytes transferred.