Puppeteer vs. Selenium vs. Scraping API

Puppeteer, Selenium, and managed scraping APIs solve the same problem, getting data out of JavaScript-heavy websites, at three different layers. Puppeteer is a Node.js library that drives Chromium over the Chrome DevTools Protocol. Selenium is a cross-browser automation standard built on the W3C WebDriver protocol, with clients in most major languages. A scraping API is a service that runs the browsers, proxies, and anti-bot handling for you and returns structured data. Playwright, Microsoft's successor to Puppeteer, belongs in the comparison too, because it is now the default recommendation for new headless work.

The short version: Puppeteer and Playwright are faster and give finer control; Selenium has the broadest browser and language reach; and a managed API removes the operational half of the problem entirely. Which one fits depends on volume, how hostile your target sites are, and how much infrastructure you want to own. The rest of this article works through each dimension, with a benchmark you can run against your own targets.

How they work under the hood

The architectural difference explains most of the performance and detection differences that follow.

Benchmark: measuring what actually matters

Published head-to-head numbers are mostly noise, because the result depends on the page, the wait strategy, and whether images are blocked far more than on the driver. What is consistent across our own runs and community benchmarks is the shape of the result, not the exact milliseconds:

Treat those as ranges observed on commodity cloud VMs, not guarantees. The right benchmark is your target pages on your infrastructure. The three scripts below run the same task, load a listing page, wait for the product cards, count them, twenty times each, and print p50 and p95. Swap in your URL and selector.

# bench_playwright.py  -- pip install playwright && playwright install chromium
import time, statistics
from playwright.sync_api import sync_playwright

URL = "https://example.com/products?page=1"
N = 20

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    ctx = browser.new_context()
    ctx.route("**/*", lambda r: r.abort()
              if r.request.resource_type in {"image", "font", "media"} else r.continue_())
    page = ctx.new_page()
    times = []
    for _ in range(N):
        t0 = time.perf_counter()
        page.goto(URL, wait_until="domcontentloaded")
        page.wait_for_selector(".product-card")
        page.locator(".product-card").count()
        times.append(time.perf_counter() - t0)
    browser.close()

print(f"p50 {statistics.median(times):.2f}s  p95 {sorted(times)[int(N*0.95)-1]:.2f}s")

// bench_puppeteer.mjs -- npm i puppeteer
import puppeteer from "puppeteer";
const URL = "https://example.com/products?page=1", N = 20;
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on("request", (r) =>
  ["image", "font", "media"].includes(r.resourceType()) ? r.abort() : r.continue());
const times = [];
for (let i = 0; i < N; i++) {
  const t0 = performance.now();
  await page.goto(URL, { waitUntil: "domcontentloaded" });
  await page.waitForSelector(".product-card");
  await page.$$eval(".product-card", (els) => els.length);
  times.push((performance.now() - t0) / 1000);
}
await browser.close();
times.sort((a, b) => a - b);
console.log(`p50 ${times[Math.floor(N / 2)].toFixed(2)}s  p95 ${times[Math.ceil(N * 0.95) - 1].toFixed(2)}s`);

# bench_selenium.py  -- pip install selenium (Selenium Manager fetches the driver)
import time, statistics
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL = "https://example.com/products?page=1"
N = 20
opts = Options(); opts.add_argument("--headless=new")
opts.add_argument("--blink-settings=imagesEnabled=false")
driver = webdriver.Chrome(options=opts)
times = []
for _ in range(N):
    t0 = time.perf_counter()
    driver.get(URL)
    WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card")))
    len(driver.find_elements(By.CSS_SELECTOR, ".product-card"))
    times.append(time.perf_counter() - t0)
driver.quit()
print(f"p50 {statistics.median(times):.2f}s  p95 {sorted(times)[int(N*0.95)-1]:.2f}s")

When is Puppeteer the right choice?

Puppeteer talks to the browser directly over CDP, which makes it fast and precise: fine-grained network interception, request blocking, CPU and network throttling, coverage, tracing, and full control of the page lifecycle. It is the natural pick for Node.js teams scraping a handful of known sites at moderate volume, and it has the deepest stealth ecosystem via puppeteer-extra-plugin-stealth and the Rebrowser patches.

• Strengths: lowest-level Chromium control, mature stealth tooling, tiny per-command latency, excellent documentation.
• Limits: Node-only, Chromium-centric (Firefox support via WebDriver BiDi is newer and less complete), no auto-waiting, and a bare instance is fingerprinted within a few requests on protected sites.

When is Playwright the right choice?

Playwright was built by the original Puppeteer team after they moved to Microsoft, and it fixes Puppeteer's two biggest gaps: it drives Firefox and WebKit as well as Chromium, and it ships official Python, Java, and .NET bindings. Every action auto-waits for the element to be visible and stable, which removes a whole class of flaky-scraper bugs. Browser contexts give you cheap, isolated sessions with separate cookies, which is ideal for running many proxy identities in one process.

• Strengths: cross-browser, multi-language, auto-waiting, route-based interception, tracing, codegen for quick prototyping.
• Limits:stealth ecosystem is smaller than Puppeteer's (though playwright-extra and patched builds exist); WebKit and Firefox builds are Playwright-specific, which itself is a fingerprint.

When is Selenium the right choice?

Selenium's strength is breadth: real Firefox, Safari, and Edge engines, bindings for Python, Java, C#, Ruby, JavaScript and Kotlin, and Selenium Grid for distributing sessions across machines. Since version 4.6, Selenium Manager downloads the matching driver automatically, removing the most hated setup step. It was designed for browser testing, and it shows: the WebDriver protocol adds an HTTP round trip per action, and scraping-specific needs such as network interception and fingerprint control are weaker than in the CDP-based tools, although WebDriver BiDi is closing that gap.

• Strengths: broadest browser and language support, huge community, Grid for distribution, the default choice when your organisation already runs Selenium for QA.
• Limits: slowest per action, easiest to detect (ChromeDriver leaves well-known artifacts and navigator.webdriver is true), weaker interception. Projects such as undetected-chromedriver and SeleniumBase UC mode patch the obvious signals but become a maintenance job of their own.

Anti-bot detection: the dimension that decides real projects

On friendly sites all three tools work indefinitely. On sites behind Cloudflare, DataDome, Akamai, PerimeterX or Kasada, the driver matters less than everything around it. Detection systems look at the browser, the network, and behaviour:

// What a page can read to spot a bare automation browser
const signals = {
  webdriver: navigator.webdriver,                 // true in Selenium and default Puppeteer/Playwright
  plugins: navigator.plugins.length,              // 0 in old headless Chrome
  languages: navigator.languages,                 // [] or mismatched with Accept-Language
  cdp: !!window.__playwright || !!window.__puppeteer_evaluation_script__,
  chrome: !!window.chrome,                        // missing in headless builds
  webgl: (() => { const c = document.createElement("canvas");
                  const gl = c.getContext("webgl");
                  const d = gl.getExtension("WEBGL_debug_renderer_info");
                  return gl.getParameter(d.UNMASKED_RENDERER_WEBGL); })(), // "SwiftShader" = headless
};

• Browser signals. navigator.webdriver, missing plugins and languages, SwiftShader WebGL renderer, inconsistent screen and viewport, and CDP-specific artifacts. Stealth plugins patch these; vendors update their checks every few weeks.
• Network signals.TLS fingerprint (JA3/JA4), HTTP/2 frame ordering, and the IP's reputation. A perfect browser fingerprint from a datacenter IP still gets blocked. Residential or ISP proxies are the usual answer.
• Behavioural signals. No mouse movement, instant clicks, identical timing between requests, and impossible navigation paths.

This is where self-hosted stacks accumulate cost: stealth patches, fingerprint generators, proxy providers, CAPTCHA solvers, and an engineer who watches block rates. We go deeper in advanced web scraping techniques.

When is a scraping API the right choice?

All three libraries leave you owning the hard operational half: browser fleets, proxy pools, anti-bot evasion, retries, and selector maintenance. A managed scraping API absorbs that layer. You send a URL and a schema; it returns validated JSON or CSV. That trade makes sense when scraping is a means to data rather than the product itself, when target sites run serious anti-bot protection, or when volume outgrows a self-hosted browser fleet. This is the layer Scrapeshop operates at; the mechanics are explained in what is a JavaScript rendering API?

• Strengths: no infrastructure, anti-bot and proxies included, structured output, predictable per-request cost, works from any language.
• Limits: less control over unusual flows, vendor dependence, and a per-request fee that can exceed self-hosting at very high volume on easy targets.

Full comparison table

Common mistakes with all three

• Launching a browser per page. Browser start-up is the single most expensive step. Keep one browser per worker, open a fresh context or tab per job, and recycle the browser every few hundred pages to contain memory leaks.
• Fixed sleeps instead of conditions. sleep(5) is too slow on fast pages and too short on slow ones. Wait for a selector, a network response, or a predicate function.
• Loading everything. Images, fonts, video and third-party analytics are usually 60–80% of bytes and add nothing to extraction. Block them at the request layer.
• Rendering when a JSON endpoint exists. Use the browser once to discover the data path, then move traffic to direct requests. Rendering should be the exception, not the default.
• Ignoring rate limits and robots.txt. Browsers make it easy to hammer a site with parallel tabs. Throttle per host; the operational and legal rules are in web scraping best practices.
• Measuring the wrong thing. A driver benchmark on a friendly site says nothing about block rate on a protected one. Track success rate and p95 per target, not just latency.

Decision guide

1. Can you avoid the browser? Check for a hidden JSON API or embedded state first; see how to scrape JavaScript-rendered websites. If yes, none of these tools is needed for most pages.
2. Prototyping or a few friendly sites? Playwright in your team's language, or Puppeteer if you are already in Node and want Chromium specifics.
3. Existing Selenium grid or need real Safari/Edge? Selenium. Do not migrate a working QA estate just to scrape.
4. Protected targets, many sites, or a team that should be building product rather than proxies? A managed API. The economics are laid out in build vs. buy: the real cost of in-house web scraping.