Design a Web Crawler

You're asked to design a web crawler that starts with a set of URLs, fetches their pages, extracts new links, and continues across a large part of the web. No user waits for an individual fetch, the work never ends, and the system must run billions of small tasks reliably without overwhelming other people's servers or fetching the same content repeatedly. Let's focus on the crawl frontier, politeness, and deduplication.

Clarify the requirements

A crawler takes seed URLs, fetches their pages, extracts links, and follows those links outward. A search engine's indexing pipeline is the usual consumer. Clarify the following before you design it:

• What happens to fetched pages? They are usually stored for downstream indexing and ranking, so the design needs storage as well as fetching.
• Is this a one-time or continuous crawl? A continuous crawler schedules pages for future visits as they change. That requirement turns a batch script into a scheduling system.
• Are all sites equal? No. A news homepage changes hourly; an old blog post changes never. The crawler should prioritize accordingly.
• How should duplicates be handled? The web contains URLs you have already seen and content stored under more than one URL. Both forms of duplication waste crawl capacity and pollute the index.
• How often can the crawler request a single site? A polite crawler limits requests to each website and follows the rules in its robots.txt file. Sending traffic too aggressively can overwhelm the site and get the crawler blocked.

The system should scale to a large fraction of the web, remain highly available, and avoid fetching pages it has already seen.

Back-of-envelope numbers

Assume 1 billion pages at roughly 100 KB per page, recrawled on average every 7 days.

• Storage per full pass: $10^9 \text{ pages} \times 100\text{ KB} \approx 100\text{ TB}$. With extracted text, metadata, and retained versions, plan on a petabyte-class store over time.
• Throughput: $10^9 \text{ pages} / 7\text{ days} \approx 1{,}650\text{ pages/sec}$ sustained
• Bandwidth: $1{,}650\text{ pages/sec} \times 100\text{ KB} \approx 165\text{ MB/s}$ of fetch traffic

These estimates lead to two design choices. Store page content in S3-style object storage rather than a database. Then use a large worker fleet coordinated through a shared frontier, because one machine cannot fetch 1,600 pages per second while respecting per-site limits. That fleet follows the async jobs and workers architecture.

High-level architecture

Components

1. Scheduler. Finds URLs whose next_crawl_at has passed and materializes them into the frontier. Never fetches anything itself.
2. Crawl frontier. The dispatch queue, partitioned by domain for politeness, with priority classes so urgent recrawls skip the backlog.
3. Fetcher workers. Form a stateless, pull-based pool. Each worker leases a URL, fetches it, and reports the result.
4. Extractor. Parses the page, stores the content, and hands discovered links to the dedupe path.
5. Object storage. Fetched pages, written once per unique content checksum.
6. URL metadata store. Holds one row per URL with its priority, next_crawl_at, attempt count, and last status. This is the job store and source of truth.

The system runs as a loop. The scheduler finds due URLs, the frontier dispatches them, workers fetch and parse pages, and the extractor stores content. Newly discovered URLs go back into the metadata store to keep the crawl moving.

Map the components to familiar job-system roles as you explain the design. The URL metadata store is the job store, with one row per URL for priority, next_crawl_at, attempt counts, and last status. The frontier handles dispatch only: it answers "what should a worker fetch next?" while the metadata store holds the rest of the state. The scheduler moves due work into the frontier but never fetches pages. Fetcher workers form a stateless, pull-based pool. Crawling one URL is a job, while "recrawl news sites hourly" is a recurring job definition.

Deep dive 1: scheduling and prioritization

Not all pages need the same attention. Store a priority and next_crawl_at for each URL. Give news and other fast-changing sites short intervals, while static reference pages can wait much longer. The scheduler finds due URLs (next_crawl_at <= now()) in small, fixed-size batches and pushes them into the frontier. Separate queues for each priority class keep urgent recrawls from waiting behind low-priority work.

Add two refinements to make the scheduler more robust. First, adapt the interval per page. If the checksum remains unchanged across several crawls, reduce the frequency; if the page changes on every visit, crawl it more often. This lets the system learn each page's change rate instead of relying on one fixed interval. Second, keep schedule state durably in the metadata store so a restarted scheduler can resume from each next_crawl_at value. A uniqueness constraint on (URL, scheduled time) makes duplicate materialization by two schedulers harmless, using the idempotent-materialization technique from async jobs and workers. If the scheduler misses several windows, crawl the page once now instead of replaying every missed run.

Strict priority can starve the long tail if news URLs saturate the fleet. Reserve some capacity for low-priority work or increase a URL's priority as it waits.

Deep dive 2: deduplication, twice

Handle URL duplication and content duplication separately because they occur at different points in the crawl.

URL-level dedupe: has the crawler seen this URL before? With billions of known URLs, querying a database for every discovered link creates a heavy read path. A hash-set lookup is O(1) in theory, but the table is enormous, part of it lives on disk, and every link on every page must check it.

Use a Bloom filter as a compact, probabilistic membership check. "Definitely not seen" is always correct, while "probably seen" has a small false-positive rate. A few hash functions over an in-memory bit array keep checks at O(1), even with billions of entries. The occasional skipped URL may be acceptable at web scale. If it is not, use the filter as a first pass and confirm positive matches against the authoritative store. Normalize case, trailing slashes, tracking parameters, and fragments before checking so trivial URL variants do not slip through.

Content-level dedupe: has the crawler stored this page under another URL? Mirrors, session-ID URLs, and print views can serve identical content at different addresses. Compute a checksum or fingerprint of the fetched content before writing it. MD5-style hashes detect exact matches, while fingerprints such as SimHash can also find near-duplicates. If the checksum already exists, record the URL-to-content mapping and skip another object-storage write.

Both checks are forms of idempotency, the property async jobs and workers demands of any at-least-once system: fetching or processing the same thing twice must not corrupt the result or double the cost.

Deep dive 3: politeness, traps, and fairness

Politeness makes a crawler unusual because it must protect capacity owned by other people. The system sends requests to servers whose operators did not agree to host its traffic. Without rate limits, the crawler can look like an attack and will likely be blocked.

Respect robots.txt. Fetch and cache it per domain, honor disallow rules and Crawl-delay, and re-fetch it periodically.

Rate-limit per domain. Cap concurrent connections, often at one, and enforce a delay between requests to the same domain. Partition the frontier by domain, with a separate queue and token bucket for each. This adapts the per-tenant token bucket from async jobs and workers: one slow or strict domain throttles only its own queue while the fleet stays busy elsewhere.

Cap pages per domain. Some sites have millions of auto-generated pages, and crawler traps (calendars that link forward forever, infinitely paginated listings) generate URLs without end. Set a per-domain page budget, cap URL depth, and watch for URL patterns that keep growing without new content checksums. Without these limits, one pathological site can eat a meaningful fraction of your fleet.

Treat DNS as a separate dependency at this scale. Millions of lookups per hour can trigger rate limits at public resolvers and add latency to every fetch. Run a caching resolver near the workers. At larger scales, a dedicated resolution layer gives you control over caching and timeouts.

Deep dive 4: failure handling and worker scaling

Fetches fail for many reasons, including timeouts, 5xx responses, connection resets, and rate limits. Classify each failure before retrying it. Retry timeouts, 429s, and 5xx responses with exponential backoff and jitter under a small per-URL budget. A 429 or Retry-After response should also slow the domain's entire bucket. Do not retry permanent failures such as 404, 410, or a missing DNS record. Move URLs that exhaust their retry budget to a dead-letter state with enough context to diagnose system-wide failures, such as the crawler's IP range being blocked.

Workers may also crash during a fetch. Give each frontier claim a lease, or visibility timeout, so the URL reappears for another worker after the lease expires. Content-checksum deduplication makes any resulting duplicate fetch harmless.

Scale the fleet using oldest-due-URL age. Add fetchers when high-priority URLs remain overdue for too long. Raw frontier depth is misleading: a large backlog of weekly pages may be fine, while news pages that are an hour late are not. Use separate pools because fetching is network bound while parsing and extraction are CPU bound, and each pool needs different scaling signals. Keep fetchers stateless and put coordination state in the frontier and metadata store so any worker can process any eligible URL.

Follow-up questions to expect

• How and when do you recrawl? Per-URL next_crawl_at driven by observed change rate; the extractor feeding discovered URLs back into the metadata store keeps the loop alive.
• Can someone attack or exhaust the crawler? Crawler traps and giant auto-generated sites; defend with per-domain budgets, depth caps, and trap detection.
• Why a Bloom filter instead of a hash table? Constant-time membership at a fraction of the memory, with the false-positive direction being the safe one for this use case.
• Two different URLs return identical pages. What happens? Content checksum hits the dedupe store; store the mapping, skip the write.
• A domain starts returning 429s. What changes? That domain's token bucket slows, its retries back off with jitter, and the rest of the fleet is unaffected.
• A fetcher dies holding 50 URLs. What happens? Leases expire, URLs reappear in the frontier, dedupe absorbs any double-processing.

Leveling signals

Related lessons

Design a Job SchedulerHard

The recurring-work sibling: definitions materialized into runs, dispatched to a worker fleet.

Design Webhook DeliveryHard

Retry budgets and per-target politeness, applied to endpoints you deliver to rather than fetch from.