Why was this already marked as resolved? New dumps haven't even been published yet, so it's impossible to verify.

@jberkel Thank you for the updated info. The recent changes have not made it into our production infra yet, they are still in staging. I will keep you posted on when they do.

Ok, I hope this can be rolled out quickly, it can't get much worse than the current state

investigation timeboxed for 5 days and made by @LDlulisa-WMF was completed. With the results of the investigation, We´ll triage accordingly during the next sprint planning.

BTW, there is a related issue T300124 that some categories and templates are not always extracted.

file sizes from the most recent enwikt HTML dumps (NS0):

20230701: 13 GB
20230720: 7.1 GB
20230801: 1.1 GB
20230820: 4.6 GB
20230901: 7.2 GB
20230920: 3 GB
20231001: 5 GB

something's going really wrong there.

Every MediaWiki site publishes a count of its own "content" pages at the API endpoint https://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics . The definition of "content" can be confirmed through the https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces API, and for most wikis this will be strictly NS0.

Please include some kind of basic health check after each dump run, to verify that the article count matches these numbers.

Sadly, the stats are not easily available for other namespaces (while dumps are made for them as well). I think currently the best way to get those counts is to use ElasticSearch endpoint, as described in T312200.

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

In T305407#9223782, @jberkel wrote:

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

When you say "base off them", are you suggesting that the HTML dumps be produced by iterating over the XML dumps and then fetching the HTML content for each row? This would be a cumbersome approach since it introduces an unnecessary extra dependency. The HTML content of the new dumps is not directly derived from the XML dumps, so I don't see much advantage to this approach. I agree that it would be nice to snapshot the wiki content before dumping but this isn't feasible given the way that HTML rendering requires random access to all articles and templates.

In T305407#9223796, @awight wrote:

In T305407#9223782, @jberkel wrote:

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

When you say "base off them", are you suggesting that the HTML dumps be produced by iterating over the XML dumps and then fetching the HTML content for each row? This would be a cumbersome approach since it introduces an unnecessary extra dependency. The HTML content of the new dumps is not directly derived from the XML dumps, so I don't see much advantage to this approach. I agree that it would be nice to snapshot the wiki content before dumping but this isn't feasible given the way that HTML rendering requires random access to all articles and templates.

I don't know how the HTML dumps are currently produced (the project is not very transparent), but you probably don't have to transform the XML itself, you could take the page ids and revision numbers from the XML dump as a starting point, and then fetch the HTML of those revisions. Of course this is more complicated with templates dependencies, but it would already be an improvement, and ensure the data you get is complete.

so why not simply base the HTML dumps off them?

This does not work. You cannot convert wikitext to HTML in offline mode. You need access to running Mediawiki instance to render macros, templates, etc. So rendering HTML is by definition a dynamic process and something which has not been available otherwise.

In T305407#9223814, @Mitar wrote:

so why not simply base the HTML dumps off them?

This does not work. You cannot convert wikitext to HTML in offline mode. You need access to running Mediawiki instance to render macros, templates, etc. So rendering HTML is by definition a dynamic process and something which has not been available otherwise.

I'm not suggesting this, see my reply above. The XML dumps would serve as a baseline for which page ids / revisions to include.

I ran into another surprising quirk in these dumps: a page may appear multiple times, sometimes with a duplicate revision, or with several different revisions. Please deduplicate by page ID.

For example, the dewiki-20230820 NS0 dump contains these revisions of one article:

["Haus Lohe",192272812]
["Haus Lohe",236444682]
["Haus Lohe",236444682]
["Haus Lohe",236444682]
["Haus Lohe",236444682]

adding some more stale data examples I came across while working with a snapshot of the HTML dumps, specifically /public/dumps/public/other/enterprise_html/runs/20231201/enwiki-NS0-20231201-ENTERPRISE-HTML.json.tar.gz on PAWS:

• A redirect that was briefly vandalized in July 2023 to not be a redirect: https://en.wikipedia.org/w/index.php?title=Adhirath&oldid=1167382940
• This also very old briefly-an-article-now-a-redirect: https://en.wikipedia.org/w/index.php?oldid=1025016513

The situation has improved somewhat, but there are still caching issues: if a page A transcludes template B, and B is updated (but not A), then the HTML dumps will still show the old content of B in A until A itself is updated.

Caching only makes sense if the invalidation works, which doesn't seem to be the case.

Hi all! We’re following up on this task with the results of a 2026 audit of the English Wiktionary Enterprise dumps to confirm if the systemic synchronization issues reported here have been resolved. We performed an audit of the current 18.3GB enwiktionary snapshot.
Findings:

• Resolved Missing/Stale Data: We specifically targeted the original "ghost" records ("paniaguarse" and "apreciable") that launched this investigation. Both are now fully indexed, present in the NDJSON, and up-to-date.
• Accuracy vs. Historical Lags: In March 2023, this community identified records with a 12-year lag and over 2,200 missing entries. In our random cross-reference of 500 articles against the live API, we found no stale records.
• System Integrity: The 18.3GB volume indicates that the "2.5 million missing pages" issue from July 2023 has been fully remediated and the output has stabilized.

Please let us know if issues persist by reopening this task.
Thank you all!

thanks so much for taking the time to review @jberkel! we'll continue working on it and will update accordingly when we have more information.

2026-04-21 snapshots missing many chunks, "enwiktionary_namespace_0" now has 48 chunks but it has 81 chunks in 2026-04-02. Other Wiktionary like es, fr, ru, de also missing many chunks.

I also notice the duplicated pages are only inside the last few chunks. In 2026-04-02 enwiktionary snapshots, duplicated pages are in the last 3 chunks(78, 79, 80). In 2026-04-02 frwiktionary snapshots, duplicated pages are in the last 2 chunks(37, 38).

In T305407#11858637, @null wrote:

2026-04-21 snapshots missing many chunks, "enwiktionary_namespace_0" now has 48 chunks but it has 81 chunks in 2026-04-02. Other Wiktionary like es, fr, ru, de also missing many chunks.

I also notice the duplicated pages are only inside the last few chunks. In 2026-04-02 enwiktionary snapshots, duplicated pages are in the last 3 chunks(78, 79, 80). In 2026-04-02 frwiktionary snapshots, duplicated pages are in the last 2 chunks(37, 38).

The 2026-04-30 snapshot has 46 chunks, weighing around 7 GB less than the 04-01 snapshot. I'm going to crosscheck against the current db.

The 2026-04-30 snapshot has only 2427207 entries (compared to 8741772 in 2026-03-01). No point doing any further analysis.

How can I download 2026-04-01 or 2026-03-01 snapshot files?

@null
It looks like you can't. When querying the API from within the Wikimedia cloud, I get:

available snapshots: ['Snapshot[20260506, 46 chunks, 11259.051 MB]']

From outside:

available.snapshots: ['Snapshot[20260502, 46 chunks, 11238.576 MB]']

Both are obviously missing data.

Unfortunately older snapshots are no longer available via the web, the last one is from 20250320: https://dumps.wikimedia.org/other/enterprise_html/runs/
I think the idea was that you should be able to get them via the API ¯\_(ツ)_/¯

I'm currently working on a project to generate community HTML snapshots for Wiktionary on WMCS. The enterprise snapshots have been around for years now and the data quality has always been a problem, plus it's missing some namespaces for analysis work (T303652).

Could you share more about the community HTML snapshots? I'm very interested. I tried to read the enterprise code(wikimedia-enterprise) to find what's wrong with it but it feels very complicated. I'm not sure if it's calling "en.wiktionary.org/w/rest.php/v1/page/page/html" API many times or doing something else.

The idea is one I've outlined earlier in this ticket (T305407#9223782): we know that the XML dumps are reliable and complete, so they can be used as a starting point to generate the HTML output. A sqlite db is produced from the XML dump and dropped into a local (containerized) MediaWiki instance. Another script then makes API requests against this instance to get the HTML for each page. The advantage is that the instance is not live and will produce consistent data (and there won't be API limit issues).

This already works, but requests are quite slow (and there are millions needed) so I'm working to optimize this. Perhaps there is a way to render the pages outside of a HTTP request cycle, and avoid the startup cost.
If you're curious the project is here: https://gitlab.wikimedia.org/toolforge-repos/digero/-/tree/main/mediawiki?ref_type=heads

Maybe try enable SQLite WAL mode(PRAGMA journal_mode = WAL;) after adding pages? Or perhaps it's normal that generate a HTML page at first time is slow?

This is a lot harder to detect, and the pages will stay outdated forever unless the page itself is updated

FWIW, this is a tricky problem to solve, even inside of WMF.

Here is one practical solution:
https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#Saneitizer_(background_repair_process)

The solution there isn't very detailed, but in summary:

slowly (over several weeks?) iterate from 0 to max page_id and re-fetch the HTML for each page.

This ensures that you will eventually some recentish HTML for every page, even if a page is missing from your dataset, or if a page hasn't received edits in a long time.

In T305407#11895920, @Ottomata wrote:

This is a lot harder to detect, and the pages will stay outdated forever unless the page itself is updated

FWIW, this is a tricky problem to solve, even inside of WMF.

Here is one practical solution:
https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#Saneitizer_(background_repair_process)

The solution there isn't very detailed, but in summary:

slowly (over several weeks?) iterate from 0 to max page_id and re-fetch the HTML for each page.

This ensures that you will eventually some recentish HTML for every page, even if a page is missing from your dataset, or if a page hasn't received edits in a long time.

poor man's cache invalidation :) as for page rendering performance, this seems to be particularly a problem with the large and heavy-templated Wiktionary pages. some pages take over a second to parse. by the time the snapshot is produced it will be outdated.

Why can't you call Wiktionary's Parsoid API, isn't API limit exempted for Wikimedia Cloud Services? Another potential alternative is kiwix's zim file but unfortunately they removed "data-mw" attribute which is required data for my use case...

In T305407#11897430, @null wrote:

Why can't you call Wiktionary's Parsoid API, isn't API limit exempted for Wikimedia Cloud Services? Another potential alternative is kiwix's zim file but unfortunately they removed "data-mw" attribute which is required data for my use case...

that's an option, but I like the idea of producing a real snapshot. using the live Parsoid API you might get different results based on the time the request is made (you could request a specific page revision, but all dependent templates will be the most recent).
if there's really no rate limit it might be the more practical solution, though.

I think this is something which impacts also all Enterprise users. So while I love community workarounds I think that might be something Enterprise team should look into and just fix it inside their codebase?

At least the Wikimedia API document says WMCS is exempted from rate limiting, maybe worth a try.

The snapshot is being prepared (querying directly the API from WMCS). At the current rate it'll take 1.5–2 days to fetch all the content, which is still acceptable.

That's great news! Could you also add other Wiktionaries and namespaces later on?

And two small issues:

• I notice the snakemake code downloads the deprecated xml file, they recommend MediaWiki Content File Exports now
• zstd has been added to Python 3.14, could use the standard library if you are able to use 3.14. And if dictionary file is used in compress it's also needed in decompress.

• zstd has been added to Python 3.14, could use the standard library if you are able to use 3.14. And if dictionary file is used in compress it's also needed in decompress.

I know, but the Python images on WMCS are still on 3.13. The code can easily be migrated once the 3.14 images become available.
Regarding the zstd dictionaries, I did some testing and it does improve the compression rate up to factor 3 for some files. However it might be the wrong choice for archival purposes, because you'll need the dictionary to decompress as you point out. At the moment there's code to train a dictionary but it's unused.

I'm unsure about the final format of the snapshots. Maybe the SQLite db for the metadata and (chunked) tar archive(s) for the zsdtd-compressed pages? Or reproduce the enterprise snapshot format (JSON) so that they can be used interchangeably? If several pages are grouped together they should compress much better.

I think zim is a good option if don't need to be compatible with enterprise api, it uses zstd and pages can be searched. They have a python package: python-libzim

I'm not familiar with WMCS, but you may not need to wait for their update if uv could be used in WMCS.

• I notice the snakemake code downloads the deprecated xml file, they recommend MediaWiki Content File Exports now

It uses the old format because processing (reading) can then be parallelized (with the help of the multistream index file).

I think zim is a good option if don't need to be compatible with enterprise api, it uses zstd and pages can be searched. They have a python package: python-libzim

Yes, and it should compress a lot better because the data is clustered there. I'll give it a try.

I'm a little worried about creating too many files might be problematic for the file system consider more Wiktionaries and namespaces will be added in the future and also delete the root html folder could take some time. Do you think it's feasible to save pages to zim first then convert the data from zim to other formats as needed?

In T305407#11920355, @null wrote:

I'm a little worried about creating too many files might be problematic for the file system consider more Wiktionaries and namespaces will be added in the future and also delete the root html folder could take some time. Do you think it's feasible to save pages to zim first then convert the data from zim to other formats as needed?

The files are stored in multiple directories keyed by id, it should be fine. Having one file per page is more fault-tolerant and scalable than updating a single zim file. I'm also not sure if zim handles this case, I checked the mwoffliner project, they incrementally add content to the zim file, but the zim isn't created (finalized) until all the pages have been fetched. maybe it's not designed for this use case.

Here's a first snapshot of the English Wiktionary (~ 5.5 GB, containing Main, Appendix:, Thesaurus: and Reconstruction:):
https://tools-static.wmflabs.org/digero/20260501/community_snapshot/

It uses (more or less) the same JSON format as the enterprise snapshots, (no wikitext, just html, and not all fields are set). Additionally, there's a pages.sqlite index db which allows querying (but has no HTML content). It is derived from the XML dumps (https://dumps.wikimedia.org/enwiktionary/20260501/). The revisions contained in the HTML snapshot should match the ones from the XML dump (minus some pages which have been deleted on the wiki in the meantime).

The whole process needs optimizing a bit before it can be applied to other wikis though, some parts are still too slow.

Thank you so much! Could you provide a JSON file for example /latest/en.json:

[{
  "date": "20260501",
  "chunks": ["pages-NS0-00-16-36422.ndjson.zst"],
  "namespace": 0,
}]

otherwise I'll have to scrape the page to know which file to download.

And please include these namespaces in other Wiktionary editions when you have time:

• fr: Conjugaison(id 116)
• de: Flexion(id 108)
• fi: Liite(id 100)
• ku: Tewandin(id 106)

some other editions I'm extracting and hopefully could be added too: es, ru, zh, ja, pl, el, nl, th, sv, simple

I pasted the code to a few llm and here are the same two suggestions made by these models:

1. buffer at enterprise.py line 55 is not cleared, it grows indefinitely. The fix is clear the buffer at the end of each iteration, or remove the buffer.

for page in pages:
    f.write(encoder.encode(page)  + b"\n")

2. remove asyncio.sleep(0.5) at spiders.py line 104, better to let Scrapy decide the pace itself by giving it an iterator. If must batch from database, increase chunk size dramatically (e.g., 5,000–10,000 items) so spend less time query-blocking the event loop.

In T305407#11928105, @null wrote:

I pasted the code to a few llm and here are the same two suggestions made by these models:

1. buffer at enterprise.py line 55 is not cleared, it grows indefinitely. The fix is clear the buffer at the end of each iteration, or remove the buffer.

for page in pages:
    f.write(encoder.encode(page)  + b"\n")

The buffer is reused in the loop, whereas the suggested code allocates new memory for the copy in a loop. See https://jcristharif.com/msgspec/perf-tips.html#reduce-allocations.
The buffer doesn't grow unbounded though, it'll never get bigger than the largest page in that loop.

2. remove asyncio.sleep(0.5) at spiders.py line 104, better to let Scrapy decide the pace itself by giving it an iterator. If must batch from database, increase chunk size dramatically (e.g., 5,000–10,000 items) so spend less time query-blocking the event loop.

Unfortunately that's not how scrapy works. It'll request new items even when it doesn't need them, leading to huge allocations for the query. The query itself is really fast (it's SQLite), and it's already batched.

The bottleneck is actually not scrapy, but the Parsoid API. For the post-processing the bottleneck is the construction of compressed pages files, it needs to decompress every page and then recompress them in batches, that code is not parallelized. It probably makes sense to move this from Python to something like go, which handles concurrency much better.

I should read the document first and verify the model output more thoroughly....

It seems a bit wasteful to compress and decompress a single page?

In T305407#11928176, @null wrote:

I should read the document first and verify the model output more thoroughly....

It seems a bit wasteful to compress and decompress a single page?

yes, technically not necessary, but to play nice and avoid dumping hundreds of GB on the WMCS file server

2026-04-21 snapshots missing many chunks, "enwiktionary_namespace_0" now has 48 chunks but it has 81 chunks in 2026-04-02. Other Wiktionary like es, fr, ru, de also missing many chunks
I also notice the duplicated pages are only inside the last few chunks. In 2026-04-02 enwiktionary snapshots, duplicated pages are in the last 3 chunks(78, 79, 80). In 2026-04-02 frwiktionary snapshots, duplicated pages are in the last 2 chunks(37, 38).

Thank you all for reporting the mismatch. We've fixed the issue, please let us know if the problem still persists. Thanks!

Thank you all for reporting the mismatch. We've fixed the issue, please let us know if the problem still persists. Thanks!

I only check the chunk number and file size, looks like the problem is fixed in 2026-05-21 snapshots. Thanks for the fix!

@null I've added an index file to the snapshots: https://tools-static.wmflabs.org/digero/community_snapshot/index.json

Thank you!