We return the HTML. The parsing is yours.

What comes back

result holds the document as text. bytes is its size, content_type is what the target declared including its charset, status is the target's own status code, and final_url is where the fetch landed after any redirects — which is often the most useful field on the page, because a listing URL that silently redirects to a consent wall looks like a successful fetch until you check.

The node decodes the body using the declared encoding, and applies size and timeout caps so one enormous or very slow page cannot occupy a node indefinitely.

No JavaScript runs anywhere in this path

You get the HTML the server sent. If a page renders its content client-side, you get the shell and an empty container where the data would be. There is no headless browser tier, and adding one is not a queue we are working through.

The five-second test before you build anything: fetch the URL with curl, or open view-source, and look for the data. If it is in the source, we can fetch it reliably. If it appears only after scripts execute, no proxy of any kind changes that, and a vendor who tells you otherwise is selling you a browser farm under a different name.

No selectors, no extraction rules

There is no extraction layer in this product at all. No CSS or XPath targeting, no selector builder, no rules that adapt themselves to a layout change, no pagination engine, no output mapping to CSV or JSON records. The omission is deliberate rather than pending: it is not a feature we are working toward, and you should choose another vendor knowingly rather than discover this after you have integrated.

Parsing belongs in your code, with your test fixtures, under your version control. An extraction service that silently starts returning empty strings after a site changes its markup is worse than no extraction service, because it fails without failing. Use your language's parser — the mature ones are excellent — and assert on what you extracted.

Being a reasonable client

The request identity you present is the User-Agent header you set on the call, chosen from the catalog we publish. Beyond that, pacing is on you: read one page at a time unless you know the target tolerates more, back off when you see a 429, and honour the terms of the site you are reading.

Our terms make that your responsibility rather than a shared ambiguity, because the alternative is a service that quietly encourages hammering someone else's server.

Questions

Can you return clean structured JSON instead of HTML?

No. We return the target's response. Turning a page into records is the part that is specific to your project, and we do not offer it as a service.

Can you log into a site for me?

No. There is no cookie handling, no credential storage and no form submission. The outbound request is a GET carrying a fixed header set plus your chosen User-Agent.

Can I collect Google or Bing rankings?

No. Search engines serve a JavaScript challenge to datacenter addresses, ours are datacenter addresses, and we do not run a browser to answer the challenge. Any SERP result you got back would be the challenge page.

What happens with an anti-bot challenge page?

You receive it as HTML with whatever status the target sent. We do not solve challenges or CAPTCHAs, so check status and content before assuming a fetch succeeded.

Related