Web Crawler

URL frontier, politeness, dedup, robots.txt and a path to search.

• Advanced
• 36 min read
• 3 objectives

Before this lessonLesson 12: File Storage (Dropbox / Drive)

Design a web crawler that starts from seed URLs, downloads pages, respects robots.txt, and feeds an indexer. This is a production crawler, not a 50-line script: billions of URLs, politeness, and traps.

Step 1 — Clarify

• One-shot snapshot, or continuous recrawl?
• HTML only, or PDFs/images? Render JavaScript?
• Must we obey robots.txt and crawl-delay?
• Output: raw HTML, extracted text, or both?

v1: continuous crawl of HTML, politeness, URL and content dedup, store pages for an indexer. Headless rendering only for a high-value allow-list of hosts — it is expensive.

Step 2 — Pipeline

flowchart LR Seeds --> F[URL frontier] F --> DNS[DNS cache] DNS --> DL[Downloaders] DL --> P[Parse] P --> DU[URL seen-filter] DU --> F P --> Store[(Page store)] Store --> Index[Indexer]

The frontier is the scheduler. Parsers emit new URLs that pass the seen-filter before going back on the frontier.

Step 3 — URL frontier

The frontier is a prioritized queue of URLs still to visit. Priorities: sitemap, a cheap importance score, recency SLA (news hourly, archives monthly). Implement many per-host queues so you can pop 'next URL for a host we have not hit in T ms' — that is politeness. A map host → next_allowed_timestamp plus crawl-delay from robots.txt keeps you from stampeding a small site. Shard downloaders by host hash so two workers never hit the same host at once.

from urllib.parse import urlparse, urljoin, urldefrag

def normalize(base, href):
    url, _frag = urldefrag(urljoin(base, href))
    p = urlparse(url)
    host = (p.hostname or "").lower()
    path = p.path or "/"
    return f"{p.scheme}://{host}{path}", host

print(normalize("https://News.Example.com/a", "../b#section"))

('https://news.example.com/b', 'news.example.com')

Step 4 — Dedup and download

• URL seen: Bloom filter plus a durable KV of visited URLs. Bloom false positives skip a URL (acceptable); false negatives recrawl (wasteful, safe).
• Content: hash extracted text; SimHash for near-duplicates so mirrors do not flood the index.
• Downloaders: cached DNS, timeouts, byte cap, limited redirect hops. Store raw bytes in object storage keyed by URL hash; metadata (status, fetched_at, outlinks) in a log.
• robots.txt: cached per host with a TTL; skip disallowed paths.

Step 5 — Traps and freshness

• Blacklist exploding query strings and calendar traps.
• Retry 5xx with backoff; honor 429 / Retry-After; drop 404 after a few recrawls.
• Not every page is equal: a separate scheduler recrawls popular URLs more often. That freshness loop is half of a real crawler.

// Write your solution here