Detect the Render Mode Before You Write the Scraper

Every pipeline that pulls structured data off the open web eventually asks the same question, once per source: does the page already contain what I need, or is a script going to build it after the fact? The two answers lead to completely different code paths, and picking the wrong one is the single most common way to make a scraping job ten to a hundred times slower and more expensive than it needs to be. This is a method for answering that question in under a minute, before you write any extraction logic at all — and for keeping the two paths behind one interface so the rest of the pipeline never has to care which one ran.

The setup that makes this worth writing down: any time you need structured data from an online catalog — a store, a directory, a listings site — you're going to hit both kinds of page in the same project, sometimes on the same domain. Treating them identically either wastes a browser on a page that never needed one, or silently returns empty results from a page that did.

§ 01One catalog, two kinds of page

Say you need prices and product details for a few dozen search terms across two catalog sites. store-a.example is built as a single-page app: the server sends a near-empty HTML shell, and a JavaScript bundle fetches product data from an API and paints it into the DOM after the page loads. store-b.example is a more conventional server-rendered storefront: the server builds the full product grid — names, prices, SKUs — into the HTML response before it ever reaches the browser.

Functionally, both pages look identical once loaded. A person browsing either site sees a grid of products and can't tell the difference without opening developer tools. That's exactly the trap: it's tempting to write one extraction method, point it at both sites, and assume it'll work everywhere. It won't, and the failure mode is quiet — a fetch-and-parse job against store-a.example won't error out, it'll just return a shell with no products in it, because the data genuinely isn't there yet.

§ 02How to tell which you're facing

Before writing a single selector, run three quick checks. Each one takes seconds and together they're close to conclusive:

• View-source vs. the rendered DOM. Open the raw response — curl the URL, or use the browser's "view page source" rather than the inspector — and search for a piece of text you can see on the live page, like a product name or price. If it's there in the raw response, the server rendered it. If it only shows up in the DevTools Elements panel (which reflects the DOM after scripts have run) and not in the raw source, a script built it client-side.
• The Network tab. Reload the page with DevTools open and watch for XHR or fetch requests that fire after the initial document loads and return JSON that matches what's on screen. That pattern — HTML shell first, data fetched separately, DOM updated afterward — is the clearest tell for client-side rendering.
• JavaScript disabled. Turn off JS for the page and reload. If the product grid is still there, it was server-rendered (or pre-rendered as static HTML). If the page goes blank or shows only a loading skeleton, the content depends on script execution.

One caveat worth internalizing: a page can look server-rendered in view-source — a full HTML document, sensible tags, even a JSON blob embedded in a script tag — and still be a client-rendered app underneath, because many frameworks ship a static shell plus embedded state that JavaScript then "hydrates" into an interactive tree. If the product data you actually need lives inside an embedded state blob rather than in ordinary HTML tags, treat it as a variant of the client-rendered case: you can sometimes parse the JSON directly out of that script tag without a browser, which is worth checking before assuming you need one.

§ 03Server-rendered: fetch it, parse it, done

For a page like store-b.example, the entire extraction is an HTTP request and an HTML parse — no browser, no JavaScript execution, no per-item clicking. Request the category or search URL directly, and the response already contains every product node you need:

// detect.js — decide before you pay for a browser
async function detectRenderMode(url, marker) {
  const html = await fetch(url).then(r => r.text());

  // marker (a selector, SKU, or price string) present in raw HTML?
  if (html.includes(marker)) return "ssr";

  // nothing there yet — content only exists after scripts run
  return "csr";
}

// route each source down the cheap or expensive path
const mode = await detectRenderMode(url, ".product-item");
mode === "ssr"
  ? parseStatic(html)          // cheerio, no browser
  : await renderAndRead(url);  // headless browser, read post-hydration DOM

Once you've confirmed the page is server-rendered, a lightweight HTML parser — something in the cheerio/BeautifulSoup family — does the rest: select the repeating product node (.product-item, or whatever the markup actually uses), pull the fields you need from its children, and move to the next search term. There's no waiting for a page load, no network idle timeout, no headless process to launch and tear down per request. You can run dozens of these requests concurrently against the same site's search endpoint and finish a full item list in the time a single browser-driven page load would take.

§ 04Client-rendered: let the browser do the work

For store-a.example, the raw response genuinely doesn't contain the data — a browser has to load the page, execute the bundle, wait for the data-fetching calls to resolve, and only then does the product grid exist anywhere. There's no shortcut around this that doesn't involve reverse-engineering the site's internal API, which is a valid option when the payoff justifies the extra fragility, but the default, dependable approach is to drive a real (headless) browser: navigate to the page, wait for the content to settle, then read the fully hydrated DOM from inside the page context, the same way the site's own JavaScript would.

In practice that means using a browser automation library — Playwright and Puppeteer are the common choices — to open the page, wait for a specific selector or for network activity to go idle, and then evaluate a small script in the page's own JavaScript context to collect the fields you need out of the rendered nodes, rather than trying to guess at an internal API response shape from the outside. It's slower and heavier than a plain HTTP request by design: you're paying for an entire browser process, a full page load, and script execution, because that's the minimum needed for the data to exist at all.

§ 05Why the distinction is worth the thirty seconds

The gap between the two paths isn't marginal. A plain HTTP request plus an HTML parse typically finishes in well under a second and uses a trivial amount of memory; launching a headless browser, loading a page, and waiting for scripts to finish executing routinely takes several seconds and an order of magnitude more memory and CPU per page. Multiply either cost across dozens of search terms and multiple sources, and the difference between the two paths is the difference between a job that finishes in a couple of minutes and one that takes the better part of an hour — and, if you're running it on rented compute, the difference in the bill is proportional.

That's also why the check in §02 belongs at the very start of the process rather than being skipped in favor of "just always use a browser, it'll work everywhere." A browser will work against a server-rendered page too — but only because it's silently doing a full page render and script execution to fetch data that was sitting in the plain HTML the whole time. It's correct and wasteful, which in a pipeline you run repeatedly is worse than it sounds.

§ 06Scrape like you intend to come back

Neither path is free of obligations to the site you're reading from. Check robots.txt before either approach and honor any Disallow rules and Crawl-delay directives it specifies — it isn't legally binding on its own in most jurisdictions, but ignoring it is a clear signal of bad faith if a dispute ever comes up, and a well-run pipeline shouldn't need that signal to behave. Rate-limit requests rather than firing them concurrently against one domain, identify your client honestly instead of spoofing a browser's user agent to look like organic traffic, and read the site's terms of service for an explicit no-scraping clause — a public catalog page and a page gated behind a login carry very different expectations, and "technically reachable" isn't the same as "intended to be automated against."

A scraper that respects rate limits and robots.txt isn't being cautious for its own sake — it's the difference between a pipeline you can run again next month and one that gets an IP range blocked after the first pass.

If a site exposes a documented API or a data feed for the same information, that's almost always the better source than either extraction path — it's faster, more stable across redesigns, and unambiguously sanctioned. Reserve scraping for the gap an API doesn't cover, not as the default first move.

§ 07One router, not two pipelines

The last piece is making the choice invisible to everything downstream. Rather than maintaining two separate pipelines — one that assumes every source is server-rendered, one that assumes a browser — wrap both extraction methods behind a single interface that takes a URL and a source configuration, runs the detection check from §02, and returns the same shape of result regardless of which path it took. The code that reconciles product records across sources afterward, the kind of matching problem we've written about reconciling records pulled from multiple sources, shouldn't need to know or care whether a given item came from a parsed HTML node or a browser-read DOM element.

It's the same shape of decision we've made in other adapter-style pipelines: one interface, several implementations behind it, each source routed to the cheapest one that actually works — the pattern shows up again in how we handle pulling live data from multiple providers with incompatible formats. A new source shows up, you run the thirty-second check, you register one more implementation, and nothing upstream of that boundary changes. The multi-source data pipeline pattern holds regardless of the domain — catalogs, listings, odds feeds, or anything else assembled from pages that weren't built to agree with each other.

None of this is exotic. It's thirty seconds of inspection before a line of extraction code gets written, and a router that remembers the answer so the rest of the pipeline never has to ask again.

— End of write-up. Building a pipeline that pulls structured data off pages that weren't built to be read by machines? Start a project →