litmus

Open this on a laptop

The canvas needs room: requirements on one side, live feedback on the other. Any screen 1024px or wider works.

Design a Web Crawler

medium

Download web pages from a list of addresses, save them, and follow their links, without fetching any page twice.

Browse all problems

The problem

Build the crawler behind a search engine. It starts from a list of seed addresses, downloads each page, saves it for the search index, pulls the links out of it, and sends the new ones back to its to-do list (the frontier). It's a loop.

Nobody waits on a crawler, so the speed of one page doesn't matter. A billion pages a month is only about 400 a second: fetching is the easy part.

The hard parts are politeness, never knocking on one website too often, and repeats: most links point at pages already seen.

What it must do

  • Fetch. Take the next address and download the page. Most of the time is spent waiting on other people's websites, so one fetcher keeps many downloads going at once.
  • Store. Keep each page (about 500 KB) in object storage for the search index, and a row per address in URL records: when it was fetched, what came back, and a fingerprint of the page.
  • Follow links. A parser reads each saved page, pulls out its links (about 30), skips pages that are copies of one already saved, and sends only addresses never seen before back to the frontier.
  • Be polite. Obey each site's robots.txt (fetched once and kept for a day) and leave a few seconds between visits to one site. The frontier keeps a queue per website and hands each site to one fetcher at a time.

What you'll learn

  1. Start with the simplest crawler that works
  2. Pass work between stages through a queue
  3. Get more done by adding workers
  4. Keep each website to one visit at a time
  5. Answer the most asked question from memory
  6. Keep a copy of the crawl record
  7. Split everything by website when it grows