Wikidata-tech February 2025

2 participants
3 discussions

Several database changes happening in the coming months
by Amir Sarabadani 24 Apr '25

24 Apr '25

Hello, in the next coming months, these changes will happen in databases and the infrastructure. And it might affect you if you rely on them in your tools or queries. This list is ordered based on how soon the change will happen. We understand that updating your tools and systems can be time consuming, hence we are giving an advanced notice. I truly apologize for the inconvenience but many of these changes are needed to keep the site running smoothly. Image table redesign Around fourteen years after the creation of T28741 <https://phabricator.wikimedia.org/T28741>, we are implementing the changes described therein. Currently, every current version of an image has a row in the image table and if there are older versions of that file, those rows could be found in the oldimage table. These two tables (image and oldimage) will be dropped in around two months. The replacement will be two main tables: file and filerevision. Every file will have a row in the file table describing the name and the type. Every version of the file (current and old) will have a row in filerevision describing the file-specific information such as its size or the hash of the file, similar to the existing distinction between pages and revisions. Another improvement is that every file and file revision will get a unique auto increment id simplifying many operations and queries. You can check T28741 <https://phabricator.wikimedia.org/T28741> for more information. The new tables are already accessible in wikireplicas but the data hasn’t been fully migrated yet. Term store split out of wikidata’s database Wikidata’s database has been growing too fast and we need to move the term store (tables starting with wbt_) to a dedicated cluster to allow growth and improve wikidata’s performance by utilizing cache locality. The new section will be called x3 and you will be able to access it in wikireplicas but this also means you won’t be able to join these tables with the rest of wikidata’s database (such as page table) since they will be residing in two physically separate servers that also means most of your queries to wikidata’s database (and term store) will become faster. We are aiming for the switch to happen in three months’ time. You can follow the work in T351820 <https://phabricator.wikimedia.org/T351820>. Additionally, wb_type table will be dropped and the mapping will be hard-coded in the code instead. See gerrit:1110810 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110810> for more details. This helped us simplify a lot of Wikibase code (example <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110720>). Categorylinks normalization Categorylinks is the next table in the series of links tables being normalized via the linktarget table (parent ticket <https://phabricator.wikimedia.org/T300222>, RFC <https://phabricator.wikimedia.org/T222224>). Similar to templatelinks and pagelinks tables, cl_to will be dropped and instead the new field cl_target_id will point to lt_id in the linktarget table. We will also drop the cl_collation field and replace it with cl_collation_id which will point to the collation_id field on the new table we are introducing called collation. We are aiming to get this fully done by the end of the next quarter (end of June 2025) but it depends on how fast the migration script can operate and that’s outside of our control. You can follow the work in T299951 <https://phabricator.wikimedia.org/T299951>.It’s worth noting that after this migration is done, we will start working on the imagelinks table. Thank you -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

Versatile Data Sets
by Jamie Willoughby 18 Feb '25

18 Feb '25

Dear Wiki Data Community, I am a data science student at Wesleyan University. This spring, my professor asked me to choose a data set to visualize. She suggested one that contains a combination of dates, geographic information, and quantitative and qualitative variables. My mind went to Wikipedia. I've spent a long time searching, but the volume of available data is overwhelming. Can you recommend a specific Wiki dataset? Thank you, Jamie Willoughby

1 0

[SIGNIFICANT CHANGE ANNOUNCEMENT] Updates to Wikidata RDF Prefixes
by Mohammed Sadat Abdulai 18 Feb '25

18 Feb '25

*(apologies for cross posting)* Hello, This significant change announcement is potentially relevant to those working with Wikidata RDF <https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format> output from the Linked Data Interface <https://www.wikidata.org/wiki/Wikidata:Data_access#Linked_Data_Interface_(U…> (Special:EntityData) or the RDF dumps <https://www.wikidata.org/wiki/Wikidata:Database_download#RDF_dumps>. (Note: The Query Service is not affected.) What’s Changing? Currently, Wikidata and WDQS use different prefixes for RDF URIs related to statements <http://www.wikidata.org/entity/statement/>, references <http://www.wikidata.org/reference/>, and value nodes <http://www.wikidata.org/value/>, leading to confusion when writing queries. - Wikidata uses: s:, ref: and v: - WDQS uses: wds:, wdref: and wdv: To ensure consistency and simplify query writing, we are aligning Wikidata RDF output with WDQS by updating the prefixes: - s: → wds: - ref: → wdref: - v: → wdv: When is this happening? These changes will be deployed on 3 March. They are already live on Test Wikidata <https://test.wikidata.org/> for you to try out. Example Change Part of the .ttl output for Q42 <https://www.wikidata.org/wiki/Special:EntityData/Q42.ttl> will change from: -- @prefix s: <http://www.wikidata.org/entity/statement/> . @prefix ref: <http://www.wikidata.org/reference/> . @prefix v: <http://www.wikidata.org/value/> . s:Q42-D6E21D67-05D6-4A0B-8458-0744FCEED13D pqv:P580 v:811f31244f158332fb4f1e13ae9f4d50 ; prov:wasDerivedFrom ref:ebf92afc2ec3ffcf1aec1d55f0b066d190281576 . -- to: -- @prefix wds: <http://www.wikidata.org/entity/statement/> . @prefix wdref: <http://www.wikidata.org/reference/> . @prefix wdv: <http://www.wikidata.org/value/> . wds:Q42-D6E21D67-05D6-4A0B-8458-0744FCEED13D pqv:P580 wdv:811f31244f158332fb4f1e13ae9f4d50 ; prov:wasDerivedFrom wdref:ebf92afc2ec3ffcf1aec1d55f0b066d190281576 . -- Will this affect you? If you parse the Wikidata RDF output with a proper RDF parser, this change should have no effect on you: the RDF parser should resolve the prefix, and the resulting URIs will not change. If you parse the RDF output differently, you may have to make adjustments. Temporary Impact on Query Service As part of the deployment of this change, the Wikidata Query Service may also temporarily exhibit some inconsistent results around “unknown value” statements (as the internal hashes used to represent them in RDF will change). This will be resolved with a full data reload (see T386098 <https://phabricator.wikimedia.org/T386098>). If you have any questions, feel free to leave a comment on T384344 <https://phabricator.wikimedia.org/T384344>. Cheers, -- Mohammed S. Abdulai *Community Communications Manager, Wikidata* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0) 30 577 116 2466 https://wikimedia.de Grab a spot in my calendar for a chat: cal.com/masssly. A lot is happening around Wikidata - Keep up to date! <https://www.wikidata.org/wiki/Wikidata:Status_updates> Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/> . Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine, Dr. Christian Humborg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech February 2025