Web Scraping Best Practices: 12 Rules for Reliable, Respectful Scrapers

Most scrapers are written in an afternoon and abandoned within a month. Not because the code was bad, but because nobody planned for the site changing its markup, the IP getting blocked, the output drifting into garbage, or a legal team asking what exactly was being collected. The practices below are the difference between a script that works once and a data pipeline that runs for a year. They apply whether you use plain HTTP requests, a headless browser, or a managed scraping API. They are ordered roughly by how early in a project you should think about them.

1. Read robots.txt and the terms of service first

Before writing a line of code, fetch /robots.txt and read the site’s terms. robots.txt tells you which paths the operator does not want automated access to and, via Crawl-delay, how fast they are comfortable being crawled. It is not a law, but honoring it is the cheapest possible signal of good faith, and it is the first thing an operator checks when deciding whether to block you or write to you.

The terms of service matter for a different reason: they are a contract, and contract claims are where scrapers actually lose in court. Scraping behind a login almost always means you accepted terms that forbid it. Scraping public pages without an account is on much stronger ground. The distinction is covered in depth in Is web scraping legal?

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

UA = "acme-pricebot/1.0 (+https://acme.example/bot; data@acme.example)"

if not rp.can_fetch(UA, "https://example.com/products/123"):
    raise SystemExit("Disallowed by robots.txt")

delay = rp.crawl_delay(UA) or 1.0

2. Identify yourself with a real user agent

A scraper that sends the default python-requests/2.31 user agent is announcing itself as a bot without saying who it belongs to. A scraper that spoofs a Chrome user agent while behaving nothing like Chrome is easy to fingerprint and looks deceptive when discovered. The honest middle path is a descriptive user agent with a contact URL or email, the same convention search engines use.

This buys you something concrete: when your traffic causes a problem, an operator can email you instead of blocking your entire IP range. Many long-running crawlers have avoided bans purely because the operator could see who was calling and why. If a target site actively blocks identified bots, that is information too, and it belongs in your legal review, not in a decision to pretend to be a browser.

3. Rate limit per host and back off on errors

Request volume is the single biggest cause of blocks and the single biggest source of complaints. A site that happily serves one request per second from you will treat fifty per second as a denial-of-service attempt. Rate limiting has to be per host, not global: ten sites at one request per second each is fine, one site at ten per second is not.

A token bucket is the standard implementation. Tokens refill at a fixed rate; each request consumes one; when the bucket is empty you wait. On a 429 Too Many Requests or a burst of 503s, multiply your interval by two to four and slowly return to normal. Respect the Retry-After header when present.

import asyncio, time
from collections import defaultdict

class HostLimiter:
    def __init__(self, rate_per_sec: float, burst: int = 2):
        self.rate, self.burst = rate_per_sec, burst
        self.tokens = defaultdict(lambda: burst)
        self.updated = defaultdict(time.monotonic)

    async def acquire(self, host: str):
        while True:
            now = time.monotonic()
            self.tokens[host] = min(
                self.burst,
                self.tokens[host] + (now - self.updated[host]) * self.rate,
            )
            self.updated[host] = now
            if self.tokens[host] >= 1:
                self.tokens[host] -= 1
                return
            await asyncio.sleep((1 - self.tokens[host]) / self.rate)

Concurrency and rate are different knobs. You can run twenty concurrent workers across twenty hosts while keeping each host at one request per second. Conflating the two is how people end up with a “polite” scraper that still hammers a single site.

4. Cache responses and use conditional requests

Re-downloading a page that has not changed wastes your bandwidth, the site’s bandwidth, and your rate-limit budget. HTTP already solves this. Store the ETag and Last-Modified headers from each response and send them back as If-None-Match and If-Modified-Since. A 304 Not Modified response has no body and costs almost nothing on either side.

Beyond conditional requests, keep a local cache of raw responses keyed by URL and fetch time. It lets you re-parse historical pages when you fix a bug in your extractor without hitting the site again, and it is the raw material for detecting drift later. Disk is cheap; re-crawling is not.

import httpx

cache = {}  # url -> (etag, last_modified, body)

def fetch(client: httpx.Client, url: str) -> bytes:
    headers = {}
    if url in cache:
        etag, lm, _ = cache[url]
        if etag: headers["If-None-Match"] = etag
        if lm:   headers["If-Modified-Since"] = lm
    r = client.get(url, headers=headers)
    if r.status_code == 304:
        return cache[url][2]
    r.raise_for_status()
    cache[url] = (r.headers.get("ETag"), r.headers.get("Last-Modified"), r.content)
    return r.content

5. Prefer the site’s JSON API over rendered HTML

Modern sites are mostly single-page applications that fetch their data from JSON endpoints and render it client-side. That data is what you actually want, and it is already structured. Open the browser’s network tab, filter to XHR and fetch, and look for responses containing the fields you need. Calling that endpoint directly skips the HTML entirely: no parser, no rendering, no brittle selectors, and often ten to a hundred times less bandwidth.

The caveats are real. The endpoint is undocumented, may require specific headers, cookies, or signed parameters, and can change without notice. But the same is true of the HTML, and the HTML changes more often. When a JSON endpoint exists, use it, and keep an HTML-based extractor as a fallback. The full workflow for JavaScript-heavy sites is in How to scrape JavaScript-rendered websites.

6. Write resilient selectors, not brittle paths

An XPath like /html/body/div[3]/div[2]/section/div[1]/span[2] breaks the moment a designer wraps something in an extra div. Selectors should target meaning, not position. In descending order of stability:

1. Structured data already on the page. JSON-LD in <script type="application/ld+json">, Open Graph meta tags, and microdata are maintained for search engines and change rarely. A product page with a Product schema gives you name, price, currency, and availability without touching the visible DOM.
2. Semantic and data attributes. [data-testid="price"], [itemprop="price"], and ARIA roles are tied to function, not layout.
3. Stable IDs and semantic class names. #product-title or .price-current. Avoid hashed CSS-module classes like .sc-1a2b3c; they change on every build.
4. Text anchors. Find the label “Price” and take its sibling. Ugly, but survives most restyles.

Whatever you choose, write two selectors per critical field, a primary and a fallback, and log which one fired. When the fallback starts winning, the primary has broken and you have time to fix it before both do.

7. Handle pagination explicitly and dedupe by key

Pagination is where scrapers silently lose or duplicate data. Sites paginate by page number, by offset, by cursor token, or by infinite scroll backed by a cursor API, and each has failure modes. Page-number pagination shifts when new items are inserted during the crawl, so item 20 on page 2 becomes item 21 on page 3 and gets scraped twice while a new item is missed. Cursor pagination avoids that but the cursor may expire.

Regardless of mechanism, dedupe on a stable business key, not on URL. Product SKU, listing ID, or a hash of the canonical URL plus variant. Treat “no new keys on this page” as the stop signal rather than trusting a “next” link, which some sites render even on the last page. Store the key with a first-seen and last-seen timestamp so you can also detect items that disappeared.

8. Retry with jitter and classify errors

Not every failure deserves a retry, and retrying the wrong ones makes things worse. A rough classification:

Jitter matters because synchronized retries from many workers produce a thundering herd that looks exactly like an attack. Standard formula: sleep for random(0, min(cap, base * 2 ** attempt)). Libraries like tenacity in Python or p-retry in Node implement this and are worth using instead of hand-rolling.

import httpx
from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception

def is_transient(exc: BaseException) -> bool:
    if isinstance(exc, httpx.TransportError):
        return True
    return isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code in (429, 500, 502, 503, 504)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=1, max=60),
    retry=retry_if_exception(is_transient),
)
def get(client: httpx.Client, url: str) -> httpx.Response:
    r = client.get(url, timeout=20)
    r.raise_for_status()
    return r

9. Validate every record against a schema

A scraper that emits whatever it found is a scraper that emits garbage the day the site changes. Define the output shape up front and reject or quarantine records that do not conform. Price should be a number in a known currency, not the string “$1,299.00 was $1,499.00”. Availability should be an enum. Dates should be ISO 8601. Validation catches the majority of extraction bugs at the point they occur instead of three joins downstream.

from decimal import Decimal
from typing import Literal
from pydantic import BaseModel, HttpUrl, field_validator

class Product(BaseModel):
    sku: str
    name: str
    price: Decimal
    currency: Literal["USD", "EUR", "GBP"]
    availability: Literal["in_stock", "out_of_stock", "preorder"]
    url: HttpUrl

    @field_validator("price")
    @classmethod
    def positive(cls, v: Decimal) -> Decimal:
        if v <= 0:
            raise ValueError("price must be positive")
        return v

import { z } from "zod";

export const Product = z.object({
  sku: z.string().min(1),
  name: z.string().min(1),
  price: z.number().positive(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  availability: z.enum(["in_stock", "out_of_stock", "preorder"]),
  url: z.string().url(),
});

export type Product = z.infer<typeof Product>;

Keep the invalid records. A quarantine table with the raw HTML reference and the validation error is the fastest way to see what changed on the site.

10. Monitor drift and alert on field-level null rates

HTTP status monitoring tells you when the site is down. It does not tell you when the site redesigned its product card and your price selector now matches nothing. That failure mode is silent: every request returns 200, every record validates (because the field is optional), and the dataset quietly fills with nulls.

Track, per run and per field, the percentage of records where the field is null, empty, or failed validation. Compare against a rolling baseline. A field that goes from 3% null to 40% null is a broken selector. Also track record count per run against expectation; a category page that normally yields 48 items and suddenly yields 12 has probably changed pagination. Alert on deltas, and include a link to a cached copy of the page so the person on call can see what changed without re-fetching.

• Records per run, vs. 7-day median
• Null or invalid rate per field, vs. 7-day median
• Share of requests served by fallback selectors
• Median response size (a soft-block page is usually much smaller)
• Ratio of 200s that failed a content signature check

11. Rotate proxies only when you actually need to

Proxies are a tool for scale and geography, not a default. A single well-behaved IP with a clear user agent and sane rate limit will get further on most sites than a rotating pool that looks like a botnet. Reach for proxies when you have a legitimate reason: the data is geo-specific, your volume genuinely exceeds what one address can politely request, or the site rate-limits per IP at a level below your needs.

When you do use them, prefer session stickiness (one IP per logical session) over per-request rotation, which breaks cookies and looks unnatural. Datacenter proxies are cheap and fine for sites that do not fingerprint; residential proxies are expensive and appropriate when they do. The evasion techniques, and their costs, are laid out in Advanced web scraping techniques. Using rotation to defeat a block you received after a cease-and-desist is the scenario where scrapers lose lawsuits.

12. Minimise personal data and keep a legal checklist

Collect the fields you need and nothing more. If your use case is price monitoring, you do not need reviewer names. If it is lead enrichment, you are processing personal data and the GDPR, the UK GDPR, and a growing list of US state laws apply regardless of whether the data was public. “Publicly visible” has never meant “free to process” under European law, and regulators have fined large-scale scrapers of profiles and images accordingly.

A short written checklist, reviewed once per project, prevents most problems:

• Is any target behind a login or paywall? If so, stop and get advice.
• Does the data include names, emails, faces, or other personal data? Document the lawful basis.
• Are you republishing creative content (articles, photos) or only extracting facts?
• Does robots.txt or the ToS explicitly prohibit automated access to these paths?
• Who receives cease-and-desist letters, and what is the response procedure?
• Retention: how long do you keep raw pages and extracted records, and why?

The legal landscape, including the cases people cite most, is summarised in Is web scraping legal?

The checklist

Items three through eleven are exactly the operational burden a managed service absorbs. If you would rather not own that layer, the trade-offs are quantified in Build vs. buy: the real cost of in-house web scraping. Scrapeshop handles rendering, proxies, retries, and schema validation behind one API call; rules one, two, and twelve stay with you either way.