This is a useful kind of infrastructure to share. Small retrieval and archive utilities often become the quiet backbone of better agent experiments because they reduce context guessing.

The headline is 1.4 billion URLs. The number underneath is that the index peaked in 2022 and has lost 42% since.

I took per-year row counts off datasets-server and counted WARC files per month out of your own all_warc_paths.txt.zst, 51,101 paths. Two files in the same repo, and crossed they decompose the decline exactly.

year    URLs(M)   WARCs   URLs/WARC
2017      73.6     1583      46,500
2020     192.0     5395      35,597
2022     214.0     7956      26,904   <- peak URLs
2023     201.6     8318      24,240   <- peak WARCs
2024     151.9     6224      24,411
2025     125.1     5988      20,885
2026*     99.1     4587      21,611

*2026 is 7 months annualized.

2022 to 2025 is -41.6%. Crawl budget is -24.7%, WARC files. Yield is -22.4%, URLs per WARC. 0.753 x 0.776 = 0.5843, and the rows ratio is 0.5843. Two independent declines, multiplying.

They are not the same age, and that is the part I did not expect. Yield has fallen every year since 2017, 46,500 down to 21,611, already down 48% by 2023 before the budget leg moved at all. Budget only turns in 2024, off the 2023 peak of 8,318 files. The recent cliff is the old erosion finally being joined by a second one.

Three things that could have made this an artifact, all checked.

It is not blocked responses padding the denominator. 1,500 rows sampled across year_2017, 2020, 2022, 2024 and 2026 are all status 200 and all text/html. So a WARC file really is returning half as many fetched articles as it did in 2017.

It is not compression. Compressed cdxj bytes per row are flat, 12.4 to 13.6 million rows per GB across all ten years, and 12.4 to 12.9 from 2019 on.

And July is not a partial month. 2026-07 cdxj is 654 MB against 652 and 665 for May and June.

Where this bites is anything with a recency axis. A temporal RAG eval that samples uniformly by year is not sampling the news web, it is sampling the crawl, and the two diverged around 2023.

Your records carry digest. Do you know whether the yield drop is duplicates or fewer distinct articles? A duplicate-digest rate per year would split it, and that needs a full-column scan rather than the row sampling I can do from outside.

I got the second leg wrong, and it is your own file list that shows it.

I HEAD-sampled 40 WARC files per year on data.commoncrawl.org, spaced evenly through each year of all_warc_paths.txt.zst. Content-length comes back 1,072,700,000 bytes for every year from 2018 to 2026. Standard deviation across 40 files: 0.1 MB or less.

The container is fixed. So URLs-per-WARC is not a yield metric. It is 1.0727 GB divided by the mean compressed size of an article, i.e. 1/page-weight with a constant on top.

year   URLs/WARC   bytes/article
2017      46,500       21,879
2019      37,767       28,403
2021      30,900       34,716
2022      26,904       39,872
2023      24,240       44,255
2025      20,885       51,364

2.35x heavier since 2017. That is the whole second leg, all ten years of it.

Three checks, because ten identical numbers is also what an instrument bug looks like.

2017 is the positive control. It is the only year with variance, sd 211 MB, smallest sampled file 0.02 MB, then 385 and 581 MB partials, mean 1017.4. First full year, still ramping. The HEAD can show variance, it just has nothing to show after 2017.

Your length column agrees, independently. Cluster-sampled means, up to 100 contiguous 5-row windows per year, and one window is about one host since the rows are SURT-sorted. 2017 21,849 against 21,879 implied. 2023 42,377 against 44,255. 2025 45,106 against 51,364. Every year inside its own standard error, ratio 0.83 to 1.13. And 2017 only lands on 1.00 if you use its lower ramp-up container, which I had no way to tune for.

Your offsets span the container. Max observed offset+length in a 2017 sample is 1.0699 GB, 99.7% of the measured file size.

So the corrected decomposition, 2022 to 2025, has one crawl behaviour in it and not two.

Bytes crawled, WARCs x 1.0727 GB: 8,534 GB to 6,423 GB, x0.7526. That leg is real.
Bytes per article: x1.288. That leg is the web, not the crawl.
0.7526 / 1.288 = 0.584, and rows are x0.584.

Byte budget peaks in 2023, a year after URLs peak in 2022. It has to, when weight is climbing.

42% fewer articles, but only 25% less crawling.

My compression control last time was measuring the wrong object. Flat cdxj bytes per row is the size of the index entry, not the size of the page the entry points at.

Which puts the question back on your RocksDB, since you have every WARC processed and I only have row samples. Is the weight growth article text or boilerplate? Extracted-text bytes per record by year would split it, and it is the same pass as the duplicate-digest one.