What Is Web Scraping? A Practical Guide

Web scraping is the automated extraction of data from websites. A scraper requests a web page the way a browser does, parses the returned HTML, and pulls specific fields — prices, product details, job listings, contact information — into a structured format such as JSON or CSV. It does programmatically, at scale, what a person would otherwise do by copying and pasting.

Scraping sits underneath a large share of the modern data economy: search engines, price-comparison services, market intelligence platforms, travel aggregators, and most large machine-learning training corpora all begin with data collected from the public web. This guide covers how scraping works, the main techniques, how it differs from crawling and official APIs, what it is used for, and where it gets hard.

Web scraping definition

A precise definition has three parts. Web scraping is automated (a program, not a person, performs the requests), it targets content published for humans (web pages rather than a documented data feed), and it produces structured output (records with named fields rather than a saved copy of the page). You will also see it called data scraping, screen scraping, web data extraction, or web harvesting. The terms overlap; “screen scraping” historically meant reading text off terminal or desktop application screens, and today is mostly used interchangeably with web scraping.

The program that does the work is a scraper (or bot). The site it reads is the target. The rules that locate each field in a page — a CSS selector, an XPath expression, a JSON path — are usually called selectors or extractors.

How does web scraping work?

Almost every scraper, from a ten-line script to a distributed pipeline, follows the same five stages:

1. Fetch — an HTTP client or headless browser requests the page and receives HTML (or JSON). Sites built as single-page applications must be rendered with JavaScript before their content exists (see scraping JavaScript-rendered websites).
2. Parse — the raw HTML is turned into a document tree that can be queried.
3. Extract — selectors locate the elements holding each target field and read their text or attributes.
4. Validate — values are cleaned and typed (strings to numbers, relative URLs to absolute, dates to ISO format) and checked against an expected schema so malformed rows never reach your database.
5. Store & monitor — records land in a database, warehouse, or file, and the job is monitored for failures and for silent breakage when the target site changes.

A minimal scraper in Python

The classic Python stack is requests for fetching and BeautifulSoup for parsing. This example reads a product listing page and extracts name, price, and link from each product card:

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://example.com/products",
    headers={"User-Agent": "acme-price-bot/1.0 (ops@acme.com)"},
    timeout=15,
)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
products = []
for card in soup.select("article.product"):
    products.append({
        "name": card.select_one("h2").get_text(strip=True),
        "price": card.select_one("[data-price]")["data-price"],
        "url": card.select_one("a")["href"],
    })

print(products)

The same scraper in JavaScript (Node.js)

In Node.js the equivalent is the built-in fetch plus cheerio, which offers a jQuery-style API over parsed HTML. This is server-side JavaScript; it runs without a browser.

import * as cheerio from "cheerio";

const res = await fetch("https://example.com/products", {
  headers: { "User-Agent": "acme-price-bot/1.0 (ops@acme.com)" },
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);

const $ = cheerio.load(await res.text());
const products = $("article.product")
  .map((_, el) => ({
    name: $(el).find("h2").text().trim(),
    price: $(el).find("[data-price]").attr("data-price"),
    url: $(el).find("a").attr("href"),
  }))
  .get();

console.log(products);

Both produce the same structured output:

[
  {
    "name": "Trail Runner 2",
    "price": "89.00",
    "url": "/products/trail-runner-2"
  },
  {
    "name": "Road Racer",
    "price": "129.00",
    "url": "/products/road-racer"
  }
]

Two details in these examples matter more than they look. The descriptive User-Agent with a contact address tells the site operator who you are, which is the single cheapest way to avoid being treated as hostile. And the selectors target a semantic attribute (data-price) rather than a styling class, which makes them far less likely to break on the next redesign.

What is HTML scraping?

HTML scraping is the form of web scraping that reads a page’s HTML markup and extracts data by locating elements with CSS selectors or XPath. It is the oldest and simplest technique: fetch the page, parse the markup into a tree, select the nodes that hold the fields you want, and read their text or attributes. Libraries such as BeautifulSoup and lxml in Python, Cheerio in Node.js, and Nokogiri in Ruby exist for exactly this step. HTML scraping works when the content is present in the server’s response. It fails on client-rendered pages, where the HTML is a shell and the data arrives later through JavaScript; those need rendering or a call to the underlying API instead. Because selectors depend on page structure, HTML scrapers are also the first to break after a redesign, which is why production systems validate output against a schema and alert on drift.

Types of web scraping

“Web scraping” covers several techniques with very different costs. Choosing the lightest one that works is most of the skill.

A note on “client-side web scraping with JavaScript”: the phrase is used for two different things. Sometimes it means JavaScript running in your browser (an extension, or code pasted into DevTools) that reads the page you are viewing. Browser security rules — the same-origin policy and CORS — prevent ordinary page scripts from fetching and reading other sites, so this approach is limited to pages you open yourself. More often it means scraping sites that render on the client side, which requires a headless browser or a JavaScript rendering API. The tool-level trade-offs are compared in Puppeteer vs. Selenium vs. scraping API.

Web scraping vs. web crawling vs. official APIs

These three are often confused. They answer different questions: crawling asks which pages exist, scraping asks what data is on this page, and an official API gives you the data without asking the page at all.

In practice, most production systems combine crawling and scraping: a crawler walks category and pagination pages to find product URLs, and a scraper extracts price and stock status from each one. If an official API exists and covers the fields you need, use it — it is more stable and removes most legal ambiguity. Scraping fills the gap when there is no API, when the API omits the data that matters, or when you need the same data from hundreds of sites that will never share one format.

What is web scraping used for?

• Price monitoring — retailers and brands track competitor prices, promotions, and stock across marketplaces, often several times a day. See price monitoring and the technical guide to ecommerce price scraping.
• Lead enrichment — filling CRM records with firmographics from public company pages and directories. See lead enrichment.
• Machine-learning datasets — building training and evaluation corpora from public text, listings, and structured pages. See ML training data.
• Market and competitive research — aggregating listings, reviews, and job postings to measure demand, sentiment, and hiring trends.
• Aggregation — travel, real-estate, and job sites that combine listings from many sources into one searchable index.
• Monitoring and alerting — watching regulatory filings, news pages, policy documents, or the availability of scarce inventory.
• Research and journalism — academics and reporters collecting public records at a scale manual copying cannot reach.

Why web scraping gets hard

The first scraper is easy. Keeping scrapers correct for months is not. Four problems account for most of the effort:

1. JavaScript rendering

Many modern sites send a nearly empty HTML shell and build the page in the browser. A plain HTTP request sees none of the data. The fixes — finding the underlying JSON endpoint, running a headless browser, or using a rendering service — are covered in how to scrape JavaScript-rendered websites and what a JavaScript rendering API is.

2. Anti-bot protection

Retail, travel, ticketing, and social sites deploy bot management that inspects IP reputation, request rate, TLS and browser fingerprints, and behaviour, and responds with blocks, CAPTCHAs, or quietly degraded content. Handling this means proxy strategy, realistic browser configuration, and constant adjustment; see advanced web scraping techniques.

3. Maintenance and silent breakage

Sites change markup without warning. A renamed class does not throw an error — the scraper keeps running and starts returning empty or wrong fields. Without schema validation and per-field monitoring, bad data flows downstream for days before anyone notices.

4. Scale and politeness

Going from a hundred pages to a million adds scheduling, deduplication, retries, storage, and rate limiting per host so you do not overload the sites you depend on.

Is web scraping legal?

Scraping publicly accessible data is generally lawful in the US and EU, but it is not a free-for-all. Terms of service, copyright and database rights, anti-hacking statutes, and privacy laws such as the GDPR each draw boundaries, and data behind a login or data about identifiable people changes the analysis substantially. The main court decisions and a practical checklist are in Is web scraping legal? That page is an overview, not legal advice; get counsel for anything high-stakes.

Web scraping best practices in brief

The full list, with code, is in Web scraping best practices: 12 rules.

Should you build a scraper or use a scraping API?

A weekend script handles one stable site fine. Production scraping is a different problem: JavaScript rendering, proxy rotation, CAPTCHA and anti-bot challenges, retry logic, and selector maintenance every time a target ships a redesign. Teams that depend on web data usually reach a build-vs-buy decision within months. The cost breakdown is in Build vs. buy: the real cost of in-house web scraping.

A managed web scraping API such as Scrapeshop moves that operational burden server-side: you send a URL and a schema, and receive validated, typed JSON or CSV back — no browser fleet, no proxy pool, no parser upkeep. Scrapeshop is available now; get a quote to start.