Property data, and the parts of it we cannot reach

Start with the sources that want to be read

County and municipal open data portals, assessor and recorder search pages, planning and permit registers, and national or regional statistics offices publish property information because publishing it is their job. Many expose CSV or JSON directly. They rarely block cloud addresses, and the data is often better than what a listing site would have shown you: assessed values, lot dimensions, transfer records, permit history.

The work here is unglamorous — every jurisdiction has its own URL scheme and its own idea of what a parcel identifier looks like — but it is work that keeps producing results, because nobody is trying to stop you.

The large listing portals are a different problem

Major property portals treat automated collection as an adversary. They check the address range first, and every address we have belongs to AWS. When they refuse, you see it honestly: the target's 403 arrives as status 403 inside a 200 envelope, with the block page in result. Rotating among three nodes we own does not change a judgement made about the whole range.

Their terms of use are usually explicit about it too. Read them before you build. We cannot give you permission you do not have, and a supplier who implies otherwise is selling you a problem.

Change detection on listings

Where a source does answer, the useful signal is change rather than content: a price reduction, a status flip to under offer, a listing that disappears. Key everything on the source's own listing identifier, store a hash of the fields you care about, and compare.

Watch final_url specifically. A withdrawn listing very often redirects to a search page or a generic "no longer available" page while still answering 200, and the redirect is the cleanest signal that the listing is gone.

Pagination is yours to build

Result sets are paginated and each page is a separate fetch and a separate billed request. A forty-page result at twenty-five listings a page is forty requests before you touch a single detail page. Filter at the source — by area, by price band, by date listed — so you are paging through the subset you care about rather than the whole county.

Region choice is simple for this work: us-east or us-west for North American records, europe for European portals. Those are the three we run, and GET /v2/proxy/regions is the live list.

Questions

Can I scrape a particular listing portal?

Check its terms of use first, then expect its address checks to refuse a datacenter address. We will show you the refusal rather than hide it, but we cannot get around it and would rather you spent your evaluation on sources that answer.

Do you have MLS data?

No. We have no property data of our own at all. We fetch URLs you choose, and MLS content is not on a public URL.

Which region should I use for US public records?

us-east or us-west. Some county systems are slow enough that proximity is worth a little; otherwise either works. If you ask for a region we do not run, the request fails fast with unknown_region rather than being served from somewhere else.

Related