Use cases
Building a price checker on top of the fetch API
This is the implementation page. If you want the business framing — why anyone monitors prices and what it is worth — read the price monitoring solution page. What follows is the shape of the code and the four places it usually breaks.
What we do not offer
No JavaScript rendering, so a page that hydrates its price in the browser hands you the empty shell. No parsing or extraction service — we return the response, not fields. No scheduler or managed crawl jobs. And no marketplace partner APIs: fetching a public page is not the same as having a seller feed, and we do not have one.
1. Get a URL list you can defend
Everything downstream depends on this and almost nobody spends long enough on it. You want one canonical URL per thing you are tracking, at the granularity you actually price at — usually a variant, not a product. A URL that redirects to a different variant will quietly track the wrong item for months.
Before you settle on the HTML page, open the site with your browser's network tab and look for the JSON request the page makes for its own price and stock. If that request is a GET, its URL is a better target than the page: smaller, already structured, and far less likely to break when someone changes a stylesheet.
2. One call per URL
POST to /v2/proxy/fetch with region as a query parameter, your key in the x-api-key header, and a body of {"url": "..."}. That is the whole request surface. The node performs a GET, follows redirects, and hands you back the target's response.
There is no method choice, no request body toward the target and no custom header set. The one header that is forwarded is your User-Agent, so if you want the target to see a particular one, send it on your call to us. A price that only exists behind a form POST, a logged-in session or a stored postcode is not reachable this way.
3. Decide whether the response is a price
Gate on the response fields before you hand anything to a parser. status is the target's, so a 403 block page arrives inside a 200 envelope and a naive parser will happily extract nothing from it and record a price of zero.
content_type tells you whether you got the HTML or JSON you expected. bytes tells you whether the body is the usual size; a sudden collapse to a few hundred bytes is a challenge page. final_url tells you whether you landed where you aimed. A parser that runs only when all four look normal is the difference between a dataset and a rumour.
4. Retry without paying twice
Send an Idempotency-Key with each fetch. If your client times out and you retry with the same key, the request is neither re-executed nor billed a second time, and the replay carries Idempotency-Replayed: true so you can tell in your own logs.
Distinguish the two rate-limit codes when you back off. target_rate_limited means the retailer is pushing back and you should slow that target down specifically. rate_limited is ours and applies per account rather than per key, so minting a second key to run a second crawler does not buy more headroom.
5. Do the arithmetic before you write the loop
Requests are the unit you are billed in, and the multiplication is unforgiving. Five thousand variants checked twice a day is ten thousand requests a day and around three hundred thousand in a month. The same list checked hourly is twelve times that.
Most catalogues do not need uniform cadence. Fast-moving lines hourly, the long tail weekly, and the total drops by an order of magnitude without losing anything you would have acted on. The included monthly allowance and the rate beyond it are on the pricing page, which reads them live rather than repeating them here.
What you still have to build
The parser, the storage, the comparison, the alerting and the thing that decides when to run. We return the target's response and meter it. Everything after that is yours, which is either the point or a dealbreaker depending on what you came here for.
Questions
- Can you handle a site that renders prices with JavaScript?
- Not by rendering it. We have no headless browser. The workaround that usually works is fetching the JSON endpoint the page itself calls, which you can find in your browser's network tab. If the price only exists after client-side execution and there is no such endpoint, we are the wrong tool.
- What happens when a retailer blocks us?
- You see it: the target's status comes back in the response rather than being hidden behind a 200. Our addresses are datacenter addresses, so a site that blocks cloud ranges blocks us, and rotating between three nodes we own does not change that answer.
- How do I get a subscription for a crawl this size?
- For a crawl this size, by email rather than by card: write to billing@roamingproxy.com with the shape of your workload — targets, cadence, roughly how many requests a month — and the plan is activated after a short exchange. The estimate you did in step 5 is exactly what that exchange needs. Card checkout is being enabled separately through our payment provider, for workloads that do not need the conversation.
Get a key
Create an account and mint an API key in the dashboard. The full endpoint reference — request shapes, parameters and error codes — is published at https://api.roamingproxy.com/v2/docs.