# Web scraping
Source: https://proxuno.com/use-cases/web-scraping
Updated: 2026-10-01

Use case

Collect public data at scale with per-request rotation across a residential pool, and keep mobile modems in reserve for the targets that block everything else.

Rotating residential Rotating per request; sticky for paginated flows

## The problem

Most websites rate-limit or block datacentre ranges outright, and many fingerprint the request pattern of a single IP. A crawler that sends thousands of requests from a few addresses ends up collecting block pages instead of data.

Spreading requests across a large residential pool keeps the per-IP request rate close to that of a human visitor, while sticky sessions keep multi-page flows such as pagination on one address.

## Configuration

**Product**
Rotating residential, mobile proxies for the strictest domains

**Rotation**
Per request for listing and detail pages

**Session**
-session-<id>-ttl-5 for paginated or stateful flows

**Targeting**
Country of the target site; eu when location does not matter

**Retries**
Up to 2 retries on 403/429/5xx, each through a new IP

**Protocol**
HTTP for HTTP clients, SOCKS5 for headless browsers

## Example

Node.js Copy

```javascript
import { ProxyAgent, fetch } from 'undici';

const LOGIN = 'px7h2k9m4q';
const PASSWORD = 'Rq4Tn8Vw2Lx6Bz9C';
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36';

function agent(country = 'de') {
return new ProxyAgent(`http://${LOGIN}-country-${country}:${PASSWORD}@resi.gw.proxuno.com:7000`);
}

async function get(url, attempt = 1) {
const res = await fetch(url, { dispatcher: agent(), headers: { 'user-agent': UA } });
if ([403, 429, 502, 503].includes(res.status) && attempt <= 2) {
await new Promise((r) => setTimeout(r, 1000 * attempt));
return get(url, attempt + 1);
}
return { status: res.status, body: await res.text() };
}

console.log((await get('https://example.com/catalogue?page=1')).status);
```

Example credentials. Replace them with yours from the dashboard.

## Best practices

- Honour robots.txt and the site's terms, and cache pages you have already fetched.
- Keep a realistic header set (user agent, Accept, Accept-Language) and keep it consistent within a session.
- Measure success per domain. A falling success rate is a signal to slow down, not to add more IPs.
- Compress responses (Accept-Encoding: gzip, br) — residential traffic is billed on the bytes transferred.

Compliance

Scrape public data only. Do not bypass authentication or paywalls, and handle any personal data you collect under the GDPR, including a lawful basis and a retention limit.

[Read the acceptable use policy](/acceptable-use)

Recommended product

Rotating residential

Pay per GB, Europe-wide pool, city targeting and sticky sessions.

From $3.80per GB

[Order](/signup?next=%2Fdashboard%2Fbuy%3Fproduct%3Dresi-rotating%26gb%3D10) [Product details](/residential-proxies)

Alternative

[Mobile 4G/5G](/mobile-proxies)

For a handful of strict targets that block residential ranges, route those domains through dedicated mobile proxies and rotate on 403 or 429 responses.

## Other use cases

[Use cases](/use-cases)

- [SEO and SERP tracking Rotating residential](/use-cases/seo-serp-tracking)
- [Social media management Mobile 4G/5G](/use-cases/social-media-management)
- [Ad verification Rotating residential](/use-cases/ad-verification)
- [E-commerce price monitoring Rotating residential](/use-cases/ecommerce-price-monitoring)
- [Geo-testing and localisation QA Mobile 4G/5G](/use-cases/geo-testing)
- [Market research Static ISP](/use-cases/market-research)

## Run your first request in under five minutes

Create an account, add funds in crypto and start with a 24-hour mobile proxy — paid from your balance and delivered instantly.

[Create an account](/signup) [Compare pricing](/pricing)