How to Scrape JavaScript-Rendered Websites

A JavaScript-rendered website builds its content in the browser after the initial page load. When a scraper fetches such a page with a plain HTTP request, the response is a nearly empty HTML shell — the product listings, prices, or articles you wanted only appear after client-side scripts run. This is why a scraper that works on a static site returns nothing on a React, Vue, or Angular application. Extracting data from these sites requires either executing the JavaScript yourself or finding the data source the scripts read from.

How do you know a site is JavaScript-rendered?

• View the page source (not DevTools) — if the content you see in the browser is missing from the raw HTML, it is rendered client-side.
• Disable JavaScript and reload. A blank or skeleton page confirms it.
• Watch the network tab: single-page applications typically fetch their data from JSON endpoints after load.

Option 1: Find the hidden API

Before rendering anything, check whether the site’s own frontend calls a JSON API. In the browser’s network tab, filter by XHR/fetch and look for responses containing the data you need. Calling that endpoint directly is faster and more stable than parsing rendered HTML — no browser, clean structured responses. The trade-offs: endpoints are undocumented, may require the right headers, cookies, or signatures, and can change without notice.

Option 2: Render with a headless browser

Headless browsers — Puppeteer and Playwright driving Chromium, or Selenium driving a range of engines — load the page, execute its JavaScript, and hand you the final DOM. This works on virtually any site, but the costs are real:

• A rendered page costs 10–100× the CPU and memory of a plain HTTP request; a browser instance typically needs 100–300 MB of RAM.
• Anti-bot systems fingerprint headless browsers aggressively — you end up maintaining stealth plugins, proxy rotation, and CAPTCHA handling.
• Browser fleets need orchestration: queuing, crash recovery, version upgrades, and scaling infrastructure.

Option 3: Use a managed scraping API

A managed web scraping API runs the browsers, proxies, and anti-bot mitigation as a service. You send the target URL and the schema you want back; the service renders the page, extracts the fields, validates them, and returns typed JSON or CSV. That is the model Scrapeshop is built on — rendering JavaScript-heavy applications, paginated archives, and sites behind common anti-bot protections through one API call.

Which approach should you choose?

New to the topic? Start with What is web scraping? — and before scraping any site at scale, read Is web scraping legal?