Solutions
Content aggregation, and which part of it we are
Pulling many independent sources into one view is mostly retrieval work repeated at volume, and retrieval is the part we sell. This page draws the line between the call we answer and the aggregator you assemble around it, because that line is where most of the work actually sits.
What we do not offer
We are not an aggregation platform. There is no scheduler, no crawler, no HTML parser, no full-text index, no alerting and no storage. We do not render JavaScript either, so a source that assembles its article body in the browser comes back as the shell its server sent. What we sell is the fetch in the middle: you supply a URL, you get the target's response, you keep the pieces.
The problem, stated plainly
You need a single view of information that is spread across sources with no interest in being aggregated. Each publishes on its own schedule, in its own shape, at its own URL structure, and none of them will tell you when something changed. The work is not glamorous: you have to go and look, repeatedly, at a lot of places.
Two things break first when you do that from one machine. The source starts rate-limiting the address you are calling from, and the source serves something different depending on where the request appears to come from. Both are address problems, and the address is what we operate.
What we contribute
One HTTP call. POST /v2/proxy/fetch with a region and a URL, and the target's response comes back in a JSON envelope: the body, the status the target itself answered with, its content type, the URL that actually produced the body after redirects, and how many hops it took.
The envelope is deliberately more than the body. Without the target's own status you cannot distinguish a real article from a well-formed block page, and a pipeline that cannot tell those apart fails silently rather than loudly, which is the more expensive of the two.
Region per request
us-east, us-west or europe, chosen as a query parameter on each call rather than fixed for your account.
Idempotency-Key
Send one and a retry after a timeout is neither re-executed nor billed twice; the replay is marked Idempotency-Replayed: true. On a collection run that is the difference between a duplicate in your corpus and a clean retry.
Metering you can query back
Usage is recorded per request and readable over the API, so a run that quietly doubled in size is visible before the invoice is.
Feeds and structured sources before HTML
The best-behaved sources already publish machine-readable output. RSS and Atom feeds, JSON APIs, sitemaps and export endpoints are all just URLs, and fetching them through the same call saves you the parser you would otherwise maintain against someone else's markup changes.
Three social sources are readable directly instead, through a beta social API bundled with the proxy plan. Coverage is narrow on purpose — only platforms whose public APIs verifiably answer our addresses — which today means Bluesky, Mastodon across a fixed set of five instances, and Hacker News. Query GET /social/platforms for the current coverage and, more useful to an aggregator, the exclusions and their reasons: Bluesky search is refused upstream, and Reddit is not covered at all.
What you still build
Everything that makes it an aggregator rather than a pile of responses. None of this is difficult in isolation, and all of it is yours.
The source list
Which URLs, and how you discover new ones. We do not crawl and we cannot search, so the list starts with you.
Extraction
Pulling the headline, the body, the author and the date out of each source's markup. We return the response verbatim.
Deduplication and change detection
Syndicated stories arrive many times over. Comparing today's fetch to yesterday's needs yesterday's, which means a store.
Scheduling and alerting
When to check and who to tell. There is no cron here and no notification product.
Rights stay with you
Our terms forbid bypassing paywalls or access controls, and forbid scraping in violation of a site's terms where you have been put on notice of them. Deciding what you may collect and republish is your call before the fetch, not ours after it.
Egress nodes do not log target URLs or response bodies. That is a privacy property, and it has a consequence worth stating: we hold no record of what you collected and cannot reconstruct it for you.
Questions
- Can you run my collection on a schedule?
- No. There is no scheduler, no cron and no managed job runner. You call the API when you want a fetch; the timing loop lives in your system.
- Do you extract the article text from the page?
- No. You get the response body as the target sent it. Parsing, boilerplate removal and metadata normalisation are yours to write.
- What happens with sources that need JavaScript?
- You get the HTML the server sent, which for those sources is usually a shell without the content. We run no headless browser, so there is no way around that here. Check whether the source has a feed or an API instead; most do.
Get a key
Create an account and mint an API key in the dashboard. The full endpoint reference — request shapes, parameters and error codes — is published at https://api.roamingproxy.com/v2/docs.