bothy.systems Email us

Blog

Keep everything you fetch

Luke Silver 19 October 2026

Almost everything our software shows comes from someone else's website. We build for sports' governing bodies, and a sport's history is scattered across results PDFs, a results service, a volunteer's spreadsheet, and the Internet Archive's copy of a site that was relaunched years ago.

We read all of it and work out rankings, selections and records. A ranking decides who goes to a championship, so sooner or later someone asks us to prove one. To prove it we have to say what every source said, and when. This post is about the rules that let us do that, and why each one is there.

When we started, I asked what teams with petabytes do, and read the books and essays they learnt from. Almost every rule below has a name in that world. The practices carried over to our few gigabytes. The tools didn't.

How it went wrong

For the first few weeks we did what everyone does. Each site had a script that fetched what it needed, kept a cache, and threw the rest away. Three things broke it.

One site put its pages behind a bot check. Our import read the check's 403 as "no more pages" and brought in nothing, without complaint. Another body's relaunch had dropped a decade of results, and the only copies left were in the Internet Archive. A third put its results up before scoring them, and our cache kept the unscored page for good.

Three different bugs, one cause. We kept what we had worked out and threw away what we had been told. When something looked wrong, we couldn't tell a source that changed from code that broke.

Martin Kleppmann draws this exact line in Designing Data-Intensive Applications. The system of record holds the authoritative version of your data, and everything else is derived data, built from it and rebuildable. Where they disagree, the system of record is right by definition. We had built a system with no system of record. What a source had told us existed only as the conclusions we'd drawn from it.

What we were told, kept for good, and what we worked out, which we can throw away and rebuild.

One fetcher, default deny

Every request to someone else's site goes through one fetcher, and the fetcher reads one list. For every source it says whether we may read it, on what terms, how many requests may be open at once, and how long each must wait after the last. The fetcher refuses any host that isn't on the list, and any host whose terms say no. Two international results sources forbid automated collection in their terms, and one governing body asks permission to reproduce its site. We wrote to all three, all three said yes in writing, and the list says so. Changing a source's status is a one-line change, and it has to pass the same checks as any other.

Politeness belongs in the fetcher, not in each script. A script that sleeps between requests is correct until someone adds a second script, or runs the first one twice. Our fetcher counts open requests per source, and a finished request keeps its slot for the source's gap before it hands it on. Ten seconds between requests then holds however many callers there are, and there isn't a single sleep in our code.

The fetcher holds a source's slot for its gap after each request, so the gap holds however many callers there are, with no sleep in any of them. Illustrative.

Retries live there too. After a dropped connection, a 429 or a 5xx, the fetcher tries again after 1, 4, 9 and 16 seconds. A form someone is waiting on gets one retry, not four.

Keep every answer, once

We store each response body compressed and named by its SHA-256. A body fetched a thousand times takes the space of one, and its name doubles as a checksum. Next to the bodies sits a manifest, an append-only log with a line for every fetch. Each line says what we asked, when, the status, the ETag, and which body came back.

The manifest logs every fetch and names its body by hash. A body fetched twice is stored once, and every line names a body in storage. Illustrative.

None of this is new. Web archives store their crawls as WARC files, an ISO standard that holds requests, responses and their metadata, and ours is a much simpler cousin. Jay Kreps' essay The Log makes the case for an append-only, totally ordered record as the truest form of a dataset. Databricks calls this layer bronze, the raw data exactly as it arrived, from which every later layer can be rebuilt.

The manifest is what makes caching honest. For a page we've read before, the fetcher only asks whether it changed, with If-None-Match and If-Modified-Since. It answers a 304 from the stored body and logs it like any other fetch. For a file that can't change, such as an archived PDF, it doesn't ask at all. Such a file is never fetched twice, so a long job can stop at any point and start again for free.

We copy everything to object storage in the sport's own cloud account. Every write there is a create-if-absent, so sending the same object twice does no harm. Our servers can add and read, but they have no permission to delete. No bug, and no attacker on a server, can empty the history.

Derive, don't mutate

Nothing a site shows comes straight from the network. Each import works from the stored responses, so the same responses give the same tables every time. When we fix a bug, we can run the fix over every response we've ever stored without asking any source again.

Nathan Marz built the Lambda architecture on this idea. In his book Big Data, the master dataset is immutable and append-only, and every view is recomputed from it, so a human mistake can corrupt a view but never the data. Maxime Beauchemin, who created Airflow, calls the same discipline functional data engineering, with immutable inputs and tasks that give the same output however many times they run.

Most of our sites build their data into a versioned snapshot and load it in one transaction. The loader first compares the snapshot with what the last load put in. If any table would lose more than a fifth of its rows, it refuses, and the site keeps what it has. A sport doesn't lose a fifth of its results overnight. A source can. And because we can load any earlier snapshot by name, rolling back is one command.

Before a load, every table is compared with what the site has now. One that would lose more than a fifth of its rows stops the load, and the site keeps what it has. Illustrative.

Netflix calls this pattern write-audit-publish. New data goes somewhere nobody reads it, an audit checks it, and only data that passes is published. Theirs runs on Iceberg snapshots across a data lake. Ours is one comparison of row counts before one transaction.

Checkpoint before you join

This was the subtle one. Our parsers read one document at a time, a results sheet or a ranking list. Then the pipeline joins documents together. It matches the same athlete across events, spots one name spelt three ways, and carries an age learnt at one event to another. Joins hide failures. If a sheet stops parsing, its rows don't vanish with an error. They turn into a few fewer matches somewhere downstream, and nobody notices.

So every parser writes down what it read before anything gets joined. That's the documents it read, the ones it couldn't, and each row with the document it came from. We compare each run with the one before. When a source changes its layout, the next morning shows fewer documents read, long before anyone spots a name missing from a ranking.

Check each document before anything joins it to another. Afterwards, a sheet that read nothing is only a smaller number. Illustrative.

In Databricks' terms this is the silver layer, validated data, and the usual advice is to check a table's shape there. Our point is narrower, and it's the one I'd most like other people to take away. Check each document on its own, before anything joins it to another, because after the join a failure looks like a smaller number.

Checks that can't see must fail

Every morning a job reads every sport's stored data. It looks for a source failing most of its fetches, a scheduled source gone quiet, a snapshot that shrank, and a parser reading less than it did. If it finds any of them, it emails us. Each sport has its own role for this, which can read that sport's data and nothing else.

Monte Carlo, who sell data observability, describe five pillars of data health: freshness, volume, schema, distribution and lineage. Our check covers the first three. A quiet source is freshness, a shrinking table is volume, and a parser reading less is schema.

The first version had a bug I'm glad we caught early. When its sign-in expired, it skipped every sport and passed. A health check that goes green because it can't see anything is worse than none at all, because now you trust it. It now fails when it can't sign in, and it only skips a sport that has nothing to check.

Never delete

We started with a rule that expired stored responses after a year. It looked tidy, and it was wrong. Archive copies disappear, results services prune old seasons, pages get rewritten, and the manifest that says what we fetched would have expired along with the bodies. The server's disk keeps a copy too, but the bucket exists for the day the disk is gone. Now nothing we fetch expires. Taking one person's details out of every layer is a separate problem, and I'd rather solve it properly than with a timer that deletes everyone's.

What went wrong anyway

We fetched 4,600 archived PDFs for one sport. The Internet Archive refuses an address that asks too fast, so we left four seconds between requests. After twenty files it stopped answering us at all. We think each archived address redirects before it serves the file, so each file cost two requests and we went twice as fast as we meant to. We moved to one request every ten seconds and ran it on a server that restarts the job an hour after any refusal. It took about fourteen hours. Nothing fetched is ever asked for again, so every restart picked up exactly where the last one stopped.

The first attempt crashed, and the crash exposed a real bug. Each upload sent the bodies its own process had fetched, a list it kept in memory, plus any manifest lines the bucket didn't have yet. A process that died lost its list. The next run sent the dead run's manifest lines but not the files they named, and twenty-nine lines ended up pointing at files that weren't in the bucket. Now every upload also sends the body behind every line it sends. Every line in storage names a body in storage. We should have written that invariant down on day one.

Then came the payoff. We rebuilt the text of all 4,600 PDFs from the stored copies and compared it with the text we'd made a week earlier. 46 documents differed. The new text had dates the old extraction lost, and some lines wrapped differently. Both parsers read both versions and produced the same rows, down to the same hash. Without the checkpoint we'd have been taking that on trust.

It isn't big data

All of it, for every sport we work with, comes to a few gigabytes. The largest table is under two million rows. We query everything as compressed JSON lines with DuckDB on a laptop, every sport in one session. Spark, a warehouse or a stream would be infrastructure looking for a problem.

Jordan Tigani, a founding engineer on Google BigQuery, made the general case in his essay Big Data is Dead. Most data fits on one machine, and size was never what stopped people getting answers from it. We're a small example of that. What we took from the big data world was its habits, not its machinery.

The hard part was never scale. It was discipline. Write down where every byte came from. Never change it. Build everything from it. Check it every morning. None of it is clever, and that's why I trust it.

Further reading

LinkedIn has since taken The Log down, so its card links the Internet Archive's copy, which rather proves the point.