Use cases
Watching a competitor's public surfaces on a schedule
Most competitive intelligence is a diff loop. You fetch a handful of pages a rival publishes, you store what came back, and you compare it with last time. We are the fetching part of that loop, and this page is about the parts that go wrong.
What we do not offer
No search rankings. Search engines serve a JavaScript challenge to datacenter addresses and every address we have is one, so we cannot tell you where a competitor ranks and we do not sell a product that pretends otherwise. There is also no ad-spend data, no automatic summarising or SWOT generation, and no scheduler — the polling loop runs on your side.
The pages that actually change
A competitor's website is mostly static marketing. The signal is concentrated in a small number of URLs that get edited by someone whose job is to edit them, and polling those beats crawling the whole site.
The pricing page
Tier names, limits and what moved between tiers. Often the first place a strategy change becomes public.
The changelog or release notes
Dated, append-only, and usually small enough to fetch often without being rude about it.
The API reference
New endpoints appear here before they appear in marketing. If it is served as JSON or OpenAPI, it diffs cleanly.
The careers page
Open roles describe what a company is building next more honestly than its homepage does.
Read the response before you diff it
The envelope answering 200 means the fetch happened, not that the target was pleased about it. A block page or a login wall comes back as the target's own status with its own body in result. If you diff that against yesterday's real page, you get an alert that says a competitor rewrote their entire site overnight.
Three fields kill that class of false alarm. status is the target's HTTP status. final_url is where the fetch ended after redirects, so a quiet bounce to a consent or login path is visible without parsing anything. bytes is the body size, and a page that normally weighs tens of kilobytes arriving at two is an interstitial, not a redesign.
Two regions, two answers
Plenty of sites vary by where the request came from: currency, tier availability, regional wording, sometimes an entirely different pricing page. Fetching the same URL with region=us-east and again with region=europe turns that into two documents you can compare, and region_used in the response tells you which region actually served each one — normally the region you asked for, and something else only if that region's endpoints were all failing when you called.
We run three regions and no others, so this technique reaches as far as North America and Europe and stops there.
You own the history
Our egress nodes do not log target URLs or response bodies. That is deliberate, and it has a consequence you should plan around: we cannot show you what a page looked like last month, because we never kept it. The archive is yours to write, on your own storage, from the responses we hand back.
What we do keep is the meter: per-request usage you can query back over the API. That is what a support conversation is reconstructed from, together with the region, target and timestamp you send us.
Questions
- How often should I poll a competitor's page?
- As rarely as your question allows. A pricing page checked daily catches everything that matters and costs thirty requests a month, where the same page checked hourly costs seven hundred and twenty. Polite intervals also keep you out of the target_rate_limited path, which is the code you get back when the target itself starts pushing back.
- Can you tell me a competitor's search rankings or traffic?
- No. Rankings would need search-engine results we cannot fetch, and traffic figures are not public data at all. Anyone selling you either is modelling, not measuring.
- Do you keep previous versions of the pages I fetch?
- No. Nodes do not log target URLs or bodies, so version history exists only where you store it. Most people hash the body, keep the hash plus a copy of anything that changed, and throw the rest away.
Get a key
Create an account and mint an API key in the dashboard. The full endpoint reference — request shapes, parameters and error codes — is published at https://api.roamingproxy.com/v2/docs.