Practical guidance for fetching pages through an API

Choosing a region

Pick the region closest to the audience the target serves, not the one closest to you. A retailer's European storefront will show different prices, different currency and sometimes a different catalogue depending on where the request came from, and if you are comparing prices over time you want that variable held still.

Read GET /v2/proxy/regions at start-up rather than hardcoding a list. It is health-gated, so it tells you what can serve right now. Then log region_used from every response: it is normally the region you asked for, and the times it is not are exactly the times your comparison is no longer apples to apples.

If you are measuring geographic variation deliberately, fetch the same URL from two regions in the same few seconds and compare. Doing it hours apart mixes geography with whatever the site changed in between.

Retries, and the one header that makes them safe

A network timeout tells you nothing about whether the work happened. Retrying blindly means you may fetch twice, be billed twice, and — if the target counts requests — look twice as aggressive as you are.

Send an Idempotency-Key header on every fetch: any unique string up to 128 characters, derived from whatever identifies that logical unit of work in your own system. For 24 hours a repeat with the same key returns the stored response, flagged with an Idempotency-Replayed: true header, without re-fetching or billing again.

Rate limiting yourself

The published ceilings are 120 requests a minute on the fetch endpoint, 60 a minute to any single target host, and 300 a minute across the rest of the API. They are per account, not per key, so minting more keys does not raise them.

The per-host ceiling is the one real jobs meet first, and it is deliberate: polling one site faster than about once a second is how an egress address gets blocklisted, which costs you and every other customer using that node. Spread work across targets rather than hammering one, and treat a Retry-After header as an instruction rather than a suggestion.

Design your own concurrency below the ceiling rather than at it. A job that runs at 90 percent of a limit has no headroom for the day a target gets slow and your workers pile up.

robots.txt, terms of service and the law

Read the target's robots.txt and honour it. It is a machine-readable statement of what the operator wants automated clients to leave alone, and ignoring it is the fastest way to turn a technical relationship into a legal one.

Read the target's terms as well. robots.txt and terms of service disagree more often than people expect, and the terms are the document a lawyer will quote. Collect public information, not material behind a login or a paywall, and do not collect personal data because it happens to be reachable.

You are responsible for what you fetch through us. Our terms say so, and the answer does not change because the request came out of our address rather than yours. If a site asks you to stop, stop.

Redirects, and reading the response honestly

The egress node follows redirects for you and reports where it ended up. Compare final_url with the URL you asked for before you parse anything: a product page that has quietly become a category page or a consent interstitial will still arrive as a 200 with real HTML in it, and a parser that finds nothing is a much harder bug than one that never ran.

SSRF checks are enforced after DNS resolution on every hop, not just the first, so a redirect chain cannot walk a fetch into private address space. A refusal on a later hop arrives as target_not_permitted.

Remember which status is which. Our 200 means the fetch happened; the status field inside the envelope is what the target said. Track that field over time and a rising rate of 403s tells you you are being blocked long before your parser starts failing.

What a datacenter address can and cannot reach

Our addresses are AWS addresses, and any site that checks can see that. For public APIs, documentation, feeds, listing pages and most product catalogues this is exactly the right tool. A plain HTTP fetch does not pay for a browser it never renders in, and what you are billed for is a request count you can predict from your own crawl plan.

For sites that specifically refuse cloud ranges, it is the wrong tool, and no amount of rotation inside our range fixes it. Search engines are the clearest case: they answer datacenter addresses with a JavaScript challenge, which is why we withdrew our search endpoint instead of selling one that returns nothing.

Before concluding a target is unreachable, check whether the page you want is assembled from a JSON endpoint. Sites that look hostile to a plain HTTP fetch very often have a clean, stable API underneath the rendering, and that API is a far better thing to depend on than their markup.

Housekeeping that pays for itself

Cap your own response sizes and expect response_too_large on the occasional enormous page. Store the raw body you fetched, not just the parsed result, so a parser bug is a reprocessing job rather than a re-fetch. And keep your key in an environment variable or a secret store, never in the repository.

Questions

How often should I poll a single site?

No faster than once a second, and slower if the data does not change that often. The per-host ceiling is 60 requests a minute per account and there is a global per-host ceiling shared across all customers as well; if you need sustained volume against one site, talk to us before building around it.

Should I rotate user agents?

That is your side of the line, so decide deliberately. The node sends the User-Agent you sent it; if you send none, the request goes out as DataProxy-Internal-Fetcher/1.0, which is rarely what you want. GET /v2/user-agents/ is a catalog you can read and pick from — it needs a key with the read scope. Rotating identity does not make an unwelcome scraper welcome, though. Rate and volume matter far more than headers.

Can I scrape a site that requires a login?

The API takes a URL and no session state, so there is nothing to authenticate with. Beyond the mechanics, content behind a login is content someone agreed to terms to see, and collecting it is a decision with consequences that are not technical.

What is the best way to test a target before committing?

Fetch one URL from each region and look at four things: the envelope status, the target's status field, final_url, and whether the body contains the text you expected. Ten minutes of that answers more than any amount of planning.

Related