How to Scrape and Crawl JavaScript-Rendered Websites (2026)

A JavaScript-rendered website builds its content in the browser after the initial page load. Fetch such a page with a plain HTTP request and you get a nearly empty HTML shell: a root <div>, a few script tags, and none of the products, prices, or articles you came for. Those only exist once the client-side code runs, calls its data endpoints, and paints the DOM. This is why a scraper that works on a static site returns nothing on a React, Vue, Angular, or Svelte application.

There are three ways through, and they are worth trying in order: find the JSON the page already fetches, extract state the server embedded in the HTML, or execute the JavaScript yourself in a headless browser. A fourth, sending the URL to a managed rendering API, is the right call when you need this to work at scale on sites that fight back. This guide covers all four with working code in Python and Node.js.

How to tell a page is JavaScript-rendered

Thirty seconds of diagnosis saves hours of debugging:

1. View source, not DevTools. Press Ctrl+U and search for a string you can see on the page. DevTools shows the live DOM after scripts ran; view-source shows what the server actually sent. If the text is missing in view-source, it is rendered client-side.
2. Disable JavaScript and reload. A blank page, spinner, or skeleton screen confirms it. Partial content means a hybrid: some server-rendered HTML with client-side hydration for the rest.
3. Watch the Network tab. Filter by Fetch/XHR and reload. Single-page applications almost always pull their data from JSON endpoints after the first paint. Those endpoints are your best entry point.
4. Check with curl. curl -s URL | grep -c "product-card" returning zero while the browser shows fifty cards is the classic signature.

Also look for framework markers: a __NEXT_DATA__ script tag (Next.js), data-reactroot, ng-version (Angular), __NUXT__ (Nuxt), or window.__INITIAL_STATE__. Each hints at where the data lives.

Option 1: Call the hidden API directly

Before rendering anything, check whether the site's own frontend calls a JSON API. Open DevTools, go to Network, filter by Fetch/XHR, and reload. Look for responses whose preview contains the data you want. Right-click the request and choose Copy as cURL; you now have the exact URL, query parameters, and headers the browser sent. Reproducing that call is faster and more stable than parsing rendered HTML: no browser, clean structured responses, and usually pagination built into the query string.

import httpx

# Reproduce the request the frontend makes (found in DevTools > Network > Fetch/XHR)
url = "https://shop.example.com/api/v2/products"
params = {"category": "laptops", "page": 1, "pageSize": 48}
headers = {
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/128 Safari/537.36",
    "Referer": "https://shop.example.com/laptops",
}

with httpx.Client(http2=True, timeout=20) as client:
    r = client.get(url, params=params, headers=headers)
    r.raise_for_status()
    for item in r.json()["items"]:
        print(item["sku"], item["name"], item["price"]["amount"])

What to watch for when replaying frontend API calls:

• Required headers. Some endpoints check Referer, Origin, an X-Requested-With header, or a CSRF token pulled from the page. Copy the full header set first, then remove headers one at a time to find the minimal set.
• Signed or short-lived tokens. A bearer token in the request often comes from an earlier call or from the HTML. Trace where it originates; if it is derived from a hash of the request, you may need the browser after all.
• GraphQL. Look for a single /graphql endpoint with POST bodies. You can usually reuse the persisted query hash and variables verbatim, and request only the fields you need.
• Undocumented means unstable. Internal endpoints change without notice. Validate responses against a schema and alert on shape changes rather than discovering them in a downstream report.

Option 2: Extract embedded state from the HTML

Many frameworks serialise the initial data into the HTML so the client can hydrate without a second round trip. Next.js puts page props in a __NEXT_DATA__ script tag; Nuxt uses window.__NUXT__; Redux and Vuex apps commonly expose window.__INITIAL_STATE__ or __PRELOADED_STATE__; Shopify themes embed product JSON; and a surprising number of sites ship schema.org JSON-LD with full product or article data. In all of these cases a plain HTTP request plus a JSON parse gets you clean data with no browser involved.

import json, re
import httpx
from bs4 import BeautifulSoup

html = httpx.get("https://www.example.com/products/123").text
soup = BeautifulSoup(html, "html.parser")

# Next.js pages-router sites embed the server-side props here
tag = soup.find("script", id="__NEXT_DATA__")
if tag:
    data = json.loads(tag.string)
    product = data["props"]["pageProps"]["product"]
    print(product["title"], product["price"])

# Other frameworks: look for window.__INITIAL_STATE__ / __PRELOADED_STATE__
m = re.search(r"window\.__INITIAL_STATE__\s*=\s*(\{.*?\});", html, re.S)
if m:
    state = json.loads(m.group(1))

Newer Next.js app-router sites stream React Server Component payloads into self.__next_f.push(...) calls instead. The data is still there, but it is chunked and escaped; it is often simpler to find the underlying API route those pages call.

Option 3: Render with a headless browser

When the data only exists after scripts run and you cannot reproduce the call, load the page in a real browser engine and read the finished DOM. Three tools dominate: Playwright and Puppeteer, which drive the browser over the Chrome DevTools Protocol (with Playwright adding Firefox and WebKit), and Selenium, which uses the WebDriver protocol and supports every major engine. Whatever you choose, the same rules apply: wait for a concrete signal rather than a fixed sleep, block resources you do not need, and reuse browser contexts instead of launching a fresh browser per page.

Playwright (Python)

Playwright is the best default in 2026: built-in auto-waiting, first-class Python, Node, Java and .NET bindings, browser contexts that isolate cookies cheaply, and a route API for request interception.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        viewport={"width": 1366, "height": 768},
        locale="en-US",
    )
    # Skip images, fonts and media to cut render time and bandwidth
    context.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in {"image", "font", "media"}
        else route.continue_(),
    )
    page = context.new_page()
    page.goto("https://shop.example.com/laptops", wait_until="domcontentloaded")
    page.wait_for_selector("[data-testid='product-card']", timeout=15_000)

    cards = page.locator("[data-testid='product-card']")
    for i in range(cards.count()):
        card = cards.nth(i)
        print(card.locator("h3").inner_text(), card.locator(".price").inner_text())

    browser.close()

Puppeteer (Node.js)

Puppeteer is Chromium-first and Node-only, but it has the most mature stealth ecosystem (puppeteer-extra-plugin-stealth) and the DevTools protocol gives you the lowest-level control over the browser.

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.setViewport({ width: 1366, height: 768 });

// Block heavy resources
await page.setRequestInterception(true);
page.on("request", (req) => {
  const type = req.resourceType();
  if (["image", "font", "media", "stylesheet"].includes(type)) req.abort();
  else req.continue();
});

await page.goto("https://shop.example.com/laptops", { waitUntil: "domcontentloaded" });
await page.waitForSelector("[data-testid='product-card']", { timeout: 15_000 });

const products = await page.$$eval("[data-testid='product-card']", (cards) =>
  cards.map((c) => ({
    name: c.querySelector("h3")?.textContent?.trim(),
    price: c.querySelector(".price")?.textContent?.trim(),
  })),
);
console.log(products);
await browser.close();

Selenium (Python)

Selenium is the right pick when your team already runs a Selenium grid for QA or you specifically need a non-Chromium engine such as real Safari. Expect slower per-action round trips and weaker network interception than the DevTools-based tools.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1366,768")
driver = webdriver.Chrome(options=opts)

driver.get("https://shop.example.com/laptops")
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "[data-testid='product-card']"))
)
for card in driver.find_elements(By.CSS_SELECTOR, "[data-testid='product-card']"):
    print(card.find_element(By.TAG_NAME, "h3").text,
          card.find_element(By.CSS_SELECTOR, ".price").text)
driver.quit()

Waiting correctly

The most common bug in headless scrapers is reading the DOM too early. time.sleep(5) is both too slow on fast pages and too short on slow ones. Wait for the thing you actually need:

• A selector: page.wait_for_selector(".price"). Best when the target element is distinctive.
• A network response: page.expect_response(...). Best when you know which API call populates the data. You can often skip the DOM entirely and read the JSON:

# Playwright: skip the DOM entirely and read the JSON the page fetches
with page.expect_response(lambda r: "/api/v2/products" in r.url and r.ok) as resp_info:
    page.goto("https://shop.example.com/laptops")
payload = resp_info.value.json()
items = payload["items"]

• A function: page.wait_for_function("() => document.querySelectorAll('.card').length > 20") for conditions selectors cannot express.
• Avoid networkidle on pages with analytics beacons, long polling, or websockets; it may never fire, or fire before your data arrives.

Infinite scroll and lazy loading

Pages that load more items as you scroll need a scroll loop with a stopping condition and a hard cap. Better still: the scroll triggers an API call with a page or cursor parameter. Intercept one, then call it directly with incrementing cursors and skip the browser for pages two onward.

# Playwright: scroll until no new items appear, with a hard cap
prev = -1
for _ in range(50):
    count = page.locator("[data-testid='product-card']").count()
    if count == prev:
        break
    prev = count
    page.mouse.wheel(0, 4000)
    page.wait_for_timeout(800)  # let the next batch request fire

What headless rendering really costs

Rendering works on virtually any site, and it is expensive in every dimension that matters at scale:

• Compute. A rendered page costs 10–100× the CPU and memory of a plain request. Each browser context uses roughly 100–300 MB, and a page that renders in two seconds on a laptop takes four or more on a shared cloud vCPU.
• Detection. Anti-bot vendors fingerprint headless browsers aggressively: navigator.webdriver, missing plugins, canvas and WebGL hashes, TLS signatures, and behavioural signals such as mouse movement. Bare Puppeteer or Playwright is identified within a handful of requests on protected sites. You end up maintaining stealth patches, realistic fingerprints, residential proxy rotation, and CAPTCHA handling, which we cover in advanced web scraping techniques.
• Operations. Browser fleets crash, leak memory, and need upgrades every few weeks as Chromium releases. You need a queue, per-host concurrency limits, health checks, and retries with backoff. Our guide to web scraping best practices lists the operational rules.

Option 4: Use a managed rendering or scraping API

A managed web scraping API runs the browsers, proxies, and anti-bot mitigation as a service. You send the target URL, a wait condition, and the schema you want back; the service renders the page, extracts the fields, validates them, and returns typed JSON or CSV. That is the model Scrapeshop is built on: rendering JavaScript-heavy applications, paginated archives, and sites behind common anti-bot protections through one API call. An illustrative request:

curl -X POST https://api.scrapeshop.com/v1/extract \
  -H "Authorization: Bearer $SCRAPESHOP_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example.com/laptops",
    "render": true,
    "wait_for": "[data-testid=product-card]",
    "schema": {
      "products": [{ "name": "string", "price": "number", "currency": "string" }]
    }
  }'

The trade is straightforward: a usage-based fee in exchange for not owning a browser fleet, a proxy pool, or the detection arms race. It makes sense when scraping is a means to data rather than your product, when targets run serious protection, or when volume has outgrown a self-hosted fleet. For a deeper look at what these services do internally, read what a JavaScript rendering API is.

Can Scrapy handle JavaScript rendering?

Scrapy is the most common crawling framework in Python, and the most common reason it returns empty items is that it never executes JavaScript. Three ways to fix that, in order of preference:

1. Point Scrapy at the hidden API. If the page loads data from a JSON endpoint (Option 1 above), yield requests to that endpoint and parse JSON. Fastest, cheapest, no browser.
2. Use scrapy-playwright. The plugin routes selected requests through a Playwright browser, supports waiting for selectors, and returns the rendered HTML to your normal parse callback. Mark only the requests that need rendering with the playwright request meta flag so listing pages that are server-rendered stay fast.
3. Route through a rendering API. Keep Scrapy for scheduling, deduplication and item pipelines, and swap the download step for a request to a rendering or scraping API. This removes browser management from your crawler entirely.

Which approach should you choose?

Decision order: try the hidden API, then embedded state, then a headless browser. Move to a managed API when the browser fleet becomes a project of its own. For a head-to-head on the browser tools, see Puppeteer vs. Selenium vs. scraping API.

Checklist before you ship

1. Confirmed the data is client-rendered with view-source and a JS-off reload.
2. Searched the Network tab for a JSON endpoint and tried replaying it.
3. Searched the HTML for __NEXT_DATA__, __INITIAL_STATE__, or JSON-LD.
4. If rendering: waiting on a selector or response, not a sleep.
5. If rendering: blocking images, fonts and media; reusing contexts.
6. Validating output against a schema and alerting on null rates.
7. Rate limiting per host and honouring robots.txt.
8. Reviewed the legal position, especially for personal data. Start with is web scraping legal?

New to the topic entirely? Begin with what is web scraping?