Advanced Web Scraping Techniques: Fingerprints, Proxies, CAPTCHAs, and Hidden APIs

The introductory scraping tutorial, requests plus BeautifulSoup, works on sites that do not care whether you are a browser. A growing share of commercially interesting sites do care, and they invest in telling humans and bots apart at several layers: the TLS handshake, the HTTP connection, the browser environment, the IP address, and behaviour over time. This article covers what those layers check, how practitioners work with each of them, and what each technique costs to run. It assumes you have already read the basics of respectful scraping and that you have a legitimate reason to be collecting the data.

Why requests + BeautifulSoup fails on protected sites

A plain HTTP client differs from a browser in ways a server can observe before it reads a single byte of your request body. The TLS handshake advertises cipher suites and extensions in an order characteristic of the library that generated it. The HTTP/2 connection preface carries SETTINGS frames and window sizes that vary by client. The headers arrive in a fixed order that differs from Chrome’s. None of that is spoofed by changing the User-Agent string, which is why setting it to Chrome often makes detection easier: the server now has a claim it can falsify.

Even when the request gets through, the response may be a shell that expects JavaScript to run, a challenge page that expects a proof-of-work token, or a 200 containing a CAPTCHA. Each layer below addresses one of these checks. Sites rarely use all of them, so start by identifying which one is blocking you before building around all five.

TLS and HTTP/2 fingerprinting

JA3 hashes the client’s TLS ClientHello (version, ciphers, extensions, elliptic curves, point formats) into a 32-character MD5. JA4 is the newer, more granular successor. Bot-mitigation vendors keep tables mapping fingerprints to known clients, and Python’s ssl module, Go’s crypto/tls, and Node’s default TLS stack each produce fingerprints that no browser ever does. Akamai’s HTTP/2 fingerprint does the same for the connection layer using SETTINGS values, WINDOW_UPDATE size, and pseudo-header order.

The practical fix is to use a client that impersonates a real browser’s handshake. curl_cffi wraps curl-impersonate, a patched curl that reproduces Chrome, Firefox, Safari, and Edge handshakes and HTTP/2 settings.

from curl_cffi import requests

# "chrome" tracks the latest Chrome the library supports;
# pin a version like "chrome124" if you need consistency.
r = requests.get(
    "https://example.com/products",
    impersonate="chrome",
    headers={"Accept-Language": "en-US,en;q=0.9"},
    timeout=20,
)
print(r.status_code, len(r.text))

Two details matter. First, keep the user agent consistent with the impersonated version; a Chrome 124 handshake with a Chrome 90 user agent is a mismatch. Second, do not override headers the impersonation sets for you (sec-ch-ua, Accept, ordering) unless you know what you are doing. In Node, got-scraping takes the same approach with header generation and TLS tuning.

Browser fingerprinting and headless detection

When you do need a real browser, the page can inspect it. The classic tells for headless Chromium:

• navigator.webdriver === true, set by automation protocols per the WebDriver spec.
• Missing or inconsistent window.chrome, navigator.plugins, and navigator.languages.
• Canvas and WebGL rendering hashes that match known headless GPU strings (SwiftShader, Mesa llvmpipe) rather than consumer hardware.
• Screen and viewport dimensions that are round, tiny, or identical across thousands of sessions.
• Timing signals: no mouse movement, instant form fills, scripts running before fonts load.

puppeteer-extra-plugin-stealth and playwright-stealth patch the JavaScript-visible surface: they delete navigator.webdriver, populate plugins, and override WebGL vendor strings. They handle the first generation of checks. They do not handle checks that happen outside JavaScript, such as the CDP protocol itself leaking through timing, or the fact that Chromium’s headless mode historically used a separate binary with its own behaviour. The newer --headless=new mode in Chromium runs the full browser binary and closes much of that gap.

import { chromium } from "playwright";

const browser = await chromium.launch({
  headless: true,
  args: ["--disable-blink-features=AutomationControlled"],
});
const ctx = await browser.newContext({
  viewport: { width: 1366, height: 768 },
  locale: "en-US",
  timezoneId: "America/New_York",
  userAgent:
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
});
await ctx.addInitScript(() => {
  Object.defineProperty(navigator, "webdriver", { get: () => undefined });
});
const page = await ctx.newPage();
await page.goto("https://example.com", { waitUntil: "networkidle" });

The more durable approach is a patched Chromium build (projects like undetected-chromedriver for Selenium, or commercial anti-detect browsers) that changes the fingerprint at the binary level rather than via injected scripts. It also requires rebuilding every time Chromium ships, which is the cost you are trading for.

Proxy strategy: datacenter, residential, ISP

IP reputation is the cheapest signal for a site to score and the most expensive one for a scraper to change. The three proxy classes differ in how they score:

Three rules regardless of type. Use sticky sessions: one IP per logical session for the duration of that session, because rotating mid-session breaks cookies and is itself a bot signal. Match geography to the content: a US pricing page fetched through a Vietnamese IP may get a different page or a block. And measure block rate per proxy source, because pools degrade as other customers burn them; the provider will not tell you.

The ethical dimension of residential proxies is worth a sentence: those IPs belong to real people who consented to an SDK buried in a free app. Prefer providers who document consent and let you avoid the ones who do not.

CAPTCHA handling

A CAPTCHA is the site telling you it has decided you are a bot. The order of operations is: avoid triggering it, then solve it, then give up. Most CAPTCHAs are served on a risk score, not unconditionally, so fixing your fingerprint, slowing down, and using cleaner IPs eliminates the majority. Check whether the challenge appears on the first request or only after N requests from the same IP; the latter means rate, not identity, is the trigger.

When you must solve them, services such as 2Captcha, Anti-Captcha, and CapSolver accept the site key and page URL, have a human or model solve the challenge, and return a token you inject into the form. Cost is roughly a dollar or two per thousand solves, latency 10 to 60 seconds, and success is not guaranteed on reCAPTCHA v3 or Turnstile, which score behaviour rather than a discrete puzzle. Budget for both the money and the per-request latency in your pipeline design.

Give up when solving becomes the dominant cost, when a site escalates to hardware attestation or account gating, or when the challenge appears alongside a legal notice. At that point the site has made its position clear, and continuing has moved from a technical question to a legal one.

Discovering hidden APIs and GraphQL endpoints

The highest-leverage technique in this article is not evasion at all. Most modern sites are single-page applications that fetch their data from JSON or GraphQL endpoints; the HTML is a rendering of that data. Requesting the endpoint directly gives you structured output, skips rendering, and usually bypasses the browser-level checks entirely.

The DevTools workflow

1. Open the Network tab, filter to Fetch/XHR, clear it, then perform the action that loads the data (scroll, paginate, filter).
2. Sort by size or search responses for a known value on the page, such as a price or product name. The response containing it is your endpoint.
3. Right-click the request, Copy as cURL, and replay it from a terminal. If it works, strip headers one at a time to find the minimal set.
4. For GraphQL, look for a single /graphql POST endpoint; the query and variables are in the request body. Persisted-query hashes (sha256Hash) can be replayed verbatim.

Signed requests

Some endpoints require a signature header computed client-side: an HMAC over the path and timestamp, a token minted by an earlier call, or a value derived from obfuscated JavaScript. Set a breakpoint on the XHR in the Sources tab, walk up the call stack to where the header is assembled, and reproduce it. If the logic is heavily obfuscated, running the signing function in a headless browser via page.evaluate and then making the request from a plain client is often faster than reverse engineering it.

Request interception to speed up headless browsers

A headless browser downloads everything a real one does: images, fonts, analytics beacons, ad scripts, third-party widgets. For extraction, most of that is waste. Intercepting requests and aborting the ones you do not need cuts page load time and bandwidth by a large margin and reduces the number of third-party domains that see your traffic.

await page.route("**/*", (route) => {
  const type = route.request().resourceType();
  const url = route.request().url();
  if (["image", "font", "media", "stylesheet"].includes(type)) {
    return route.abort();
  }
  if (/google-analytics|doubleclick|facebook\.net|hotjar/.test(url)) {
    return route.abort();
  }
  return route.continue();
});

Two cautions. Blocking stylesheets can change layout enough to break isVisible checks and lazy-load triggers, so test with and without. And some anti-bot scripts notice when their beacon never fires; if a site starts blocking you after you added interception, that is why.

Distributed crawling

Past a few hundred thousand pages, a single process is the bottleneck, and the architecture becomes a queue, a set of workers, and shared state. The pieces:

• URL frontier. A priority queue of URLs to fetch, deduplicated by normalized URL (lowercase host, sorted query params, stripped tracking params). Redis sorted sets, SQS, or a database table all work; the important property is atomic dequeue so two workers never fetch the same URL.
• Per-host politeness. Rate limits must be enforced across workers, not per worker. A shared token bucket in Redis (or partitioning hosts to specific workers) prevents twenty workers from each doing one request per second to the same site.
• Workers. Stateless fetch-parse-emit loops that pull from the frontier, honor the limiter, push extracted records to storage, and push discovered URLs back to the frontier. Scale horizontally.
• Session and identity affinity. If a host needs cookies or a sticky proxy, route all of its URLs to the same worker or store session state centrally.
• Checkpointing. Record the frontier and visited set durably so a crash resumes rather than restarts.

Frameworks such as Scrapy (with scrapy-redis) and Crawlee implement most of this. The failure mode to design against is not throughput but correctness: duplicate fetches, lost URLs, and politeness violations that get the whole pool banned.

Infinite scroll and shadow DOM

Infinite scroll is pagination hidden behind a scroll event. Before automating scroll, check the network tab: the scroll handler is calling an endpoint with a page number or cursor, and calling that endpoint directly is faster and more reliable than scrolling. If you must scroll, scroll to the bottom in a loop, wait for the network to idle or for the item count to increase, and stop when it does not.

Shadow DOM encapsulates a component’s subtree so that ordinary CSS selectors do not reach into it. Playwright’s locators pierce open shadow roots by default; in Puppeteer, use the >>> combinator or element.shadowRoot in page.evaluate. Closed shadow roots are not reachable from page scripts at all, but the data rendered inside them almost always arrived via a network request you can intercept.

Structured data shortcuts

Before writing a selector, check whether the page already ships its data in a machine-readable form:

• JSON-LD. <script type="application/ld+json"> blocks maintained for search engines. Product, Offer, Article, Event, and JobPosting schemas contain the fields you usually want, and they change far less often than the visible DOM.
• Framework state. Next.js embeds page props in <script id="__NEXT_DATA__">. Nuxt uses window.__NUXT__. Many React and Vue apps serialize their store into window.__INITIAL_STATE__, __PRELOADED_STATE__, or __APOLLO_STATE__. This is the complete data the page rendered from, in JSON, with no rendering required.
• Open Graph and meta tags. Coarse, but reliable for title, image, description, and canonical URL.

import json, re
from curl_cffi import requests

html = requests.get(url, impersonate="chrome").text

# Next.js
m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.S)
if m:
    data = json.loads(m.group(1))["props"]["pageProps"]

# Generic window.__INITIAL_STATE__ = {...};
m = re.search(r'window\.__INITIAL_STATE__\s*=\s*(\{.*?\});\s*</script>', html, re.S)
if m:
    data = json.loads(m.group(1))

# JSON-LD
for block in re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.S):
    ld = json.loads(block)

A scraper built on embedded state often needs no browser at all for a site that looks fully client-rendered. The techniques for deciding which route a given site needs are in How to scrape JavaScript-rendered websites.

What each layer costs

Every technique above has an ongoing cost, not a one-time one, because the sites you scrape keep changing their checks. A rough comparison:

When a managed API is the rational choice

Everything in this article is a maintenance treadmill. Bot mitigation vendors ship updates weekly; your fingerprints, proxies, and stealth patches decay against them. For a team whose product is the data rather than the scraper, that treadmill is pure overhead. A managed scraping API runs the browser fleet, proxy pools, fingerprint maintenance, and challenge handling as a service and returns extracted, schema-validated data from a single request. That is the layer Scrapeshop operates at.