Tools
Scheduled collection, when the scheduler is yours
Most collection work is recurring: the same list of URLs, on some cadence, for as long as the project lasts. This page is about running that well against a metered fetch API — choosing the interval, deciding where the timer lives, and the few details that decide whether a run costs what you expected.
What we do not offer
There is no scheduler in this product. Nothing here runs a job on a cron, retries it overnight, fans it out across workers, alerts you when it fails or keeps a run history. No workflow designer, no dependency graph, no managed queue. An earlier version of this page advertised all of that; it did not exist. The loop is yours to run and yours to watch, and we serve the individual fetch inside it.
Choose the interval on purpose
Interval is the largest single lever on what a scheduled job costs, and it is usually picked out of habit. Hourly is twenty-four times daily for the same coverage; every fifteen minutes is ninety-six times. The arithmetic is one multiplication — URLs, times runs per day, times days — and it is worth doing before the first run rather than at the end of the first billing period.
Then ask how often the source actually changes. A page that moves twice a week does not repay hourly polling; a busy feed does. Most lists split into a small hot set worth checking often and a long tail that is fine daily, and splitting them saves more than any optimisation inside the loop. Usage is metered per request and queryable over the API, so a schedule that quietly doubled is visible days before the invoice is.
Where the timer lives
Anything that can make an HTTPS request on a timer can drive this, so pick whatever you already operate rather than something new.
cron
On a machine that is genuinely always on. Simple and dependable; a laptop that sleeps is not that machine.
systemd timers
Persistent= catches a run missed while the host was down, and output lands in the journal with your other logs.
CI schedules
No server to keep. Runs on shared CI can be delayed under load and are often paused on inactive repositories, so avoid them for time-critical work.
A managed cloud scheduler
A timer that calls your endpoint or function, paired with a small worker holding the URL list.
Airflow, Prefect, Dagster
Worth their weight once you have real dependencies, backfills and per-task retries. Before that, overhead.
A queue and workers
Separate when from what: the schedule enqueues URLs, workers drain the queue at a rate you control.
Spread the work, do not fire it all at :00
A schedule that starts everything on the hour makes a spike and then idles. Spread the URLs across the interval, or give each one a small random offset, so the load is flat. It is easier on the targets and it keeps you clear of your own ceiling.
That ceiling is per account, not per key, so extra keys do not raise it. Cap concurrency deliberately and back off when a target pushes back: its status code is passed through untranslated, so a 429 arrives as a 429 with its body — a real signal about that site rather than a message from us.
Make a rerun safe
A run that half-finished is the ordinary failure, and rerunning it is the ordinary fix. Send an Idempotency-Key on every fetch and a repeat is neither re-executed against the target nor billed twice; the replay carries Idempotency-Replayed: true so you can tell it from a fresh fetch. Reusing a key for a different URL or region is refused with a stable error code rather than quietly returning the wrong body.
Derive the key instead of randomising it — the run's date and the URL is enough — because a random key makes every retry a new fetch, which is the thing you were avoiding. Keys are remembered for twenty-four hours, so tomorrow's run of a daily job fetches fresh content even for the same URL, which is what you want.
A run is not finished because it returned
Check the envelope rather than the absence of an exception. status is the target's own code. content_type tells you whether you got the JSON you expected or an HTML error page wearing a 200. final_url is where the fetch actually landed, which is how a listing page that started redirecting to a consent wall can look like success for a month. And bytes halving across a whole run is a layout change, not a coincidence.
Assert on region_used too if the exit location matters: a failed-over fetch succeeds from a different region, and that field is the only place it says so. Keep your own log of what each run saw — we do not record target URLs or response bodies, so yours is the only history there is.
Questions
- Do you have a cron, a job runner or a managed queue?
- No, none of the three. We serve one fetch per API call. The timer, the retry policy, the concurrency and the alerting are all yours, and we would rather say so than describe a control panel that does not exist.
- Will you tell me when a scheduled run fails?
- No. We have no notion that you have a job at all — we see individual requests. Alerting belongs with whatever runs your schedule, and it should watch your own success counts rather than only process exit codes.
- Can I run several keys to get more throughput?
- No. Rate limits are enforced per account. Keys exist so you can scope and revoke them individually — one per environment, say — not to multiply throughput.
- Does an Idempotency-Key stop my daily job re-fetching?
- Only within twenty-four hours, and only for the same key. That is what you want: put the run date in the key and a same-day retry replays without a second charge, while tomorrow's run fetches fresh content.
- What is the cheapest way to watch many URLs?
- Poll fewer things more often and more things less often. Group the list by how fast it really changes, and prefer a feed or a JSON endpoint over a heavy page where the site offers one — fewer bytes and a far easier parse.
Get a key
Create an account and mint an API key in the dashboard. The full endpoint reference — request shapes, parameters and error codes — is published at https://api.roamingproxy.com/v2/docs.