Polling feeds and articles, and the arithmetic of cadence

Feeds first, articles second

Fetch the RSS or Atom feed, not the homepage. A feed is small, structured, dated, and stable across redesigns that would break any selector you wrote against the site. It also tells you exactly which items are new, which means you fetch article HTML only for things you have not seen.

That two-stage pattern is what keeps the request count sane. One feed fetch per publisher per interval, plus one article fetch per genuinely new item, instead of re-fetching a section page full of articles you already have.

Conditional requests are not available

This is the constraint that shapes the whole design, so it is better stated than discovered. The fetch body is a URL and nothing else. You cannot attach If-Modified-Since or If-None-Match, which means you cannot get a cheap 304 back from a publisher who would happily give you one.

Every poll is therefore a full fetch and a full billed request. Deduplicate on your side: keep each item's GUID or link, hash the feed body, and skip the parse when the hash is unchanged. The network cost is already paid but the processing cost is not.

What a consent wall looks like from here

News sites are the worst offenders for interstitials, and most of them return 200. The reliable tells are in the response envelope rather than the body: final_url has moved to a consent or subscribe path, redirects is higher than usual, and bytes is a fraction of what that publisher normally sends.

Treat those as a distinct outcome in your pipeline rather than as an article with no text. An aggregator that silently ingests consent boilerplate produces a corpus nobody can trust later.

Region changes the edition

Large publishers serve different editions, different consent regimes and sometimes different headlines depending on where the request came from. Fetching with region=europe and again with region=us-east is a legitimate way to capture both, and region_used in the response records which region actually served each copy, and device_id which egress node did the work, so your corpus stays honest about provenance.

Budget the cadence

Two hundred feeds polled every fifteen minutes is nineteen thousand two hundred requests a day before you fetch a single article. The same two hundred feeds at hourly is four thousand eight hundred. Breaking news rarely justifies the first number, and a tiered cadence — minutes for the handful of sources that matter, hours for the rest — is almost always the right shape.

Politeness and cost point the same direction here, which is convenient. A publisher whose feed you hit four times a minute will eventually stop answering you, and that arrives as a target_rate_limited refusal.

Questions

Do you have an endpoint that returns articles already parsed?

No. The proxy returns the publisher's own response and you parse it. The one place we return structured data is the social data API, which covers Bluesky, Mastodon and Hacker News and is a different product surface.

How do I avoid hammering a publisher?

Poll the feed rather than the site, space the interval to the pace the publisher actually updates at, and back off properly when a target starts refusing. Read the publisher's terms — some state a polling interval explicitly.

Can you fetch sites that only render articles in JavaScript?

No. There is no headless browser and no rendering step. Many such sites still publish a working feed, and that is usually the way in.

Related