Data-ServicesComponent
ActivePublic

Milestones

Members

  • This project does not have any members.
  • View All

Details

Description

Data services available for use on Cloud-Services.

  • Wiki Replicas - redacted copies of the Wikimedia wiki's production databases
  • Wikimedia Dumps - public hosting of full text dumps from Wikimedia wikis and other datasets in Cloud (see also: Dumps-Generation for generation of dumps themselves)
  • Shared Storage - NFS storage for cross-VM and cross-project use

Recent Activity

Mon, Dec 1

Maintenance_bot removed a project from T306550: Move dumps.wikimedia.org HTTP service behind CDN edge: Patch-For-Review.
Mon, Dec 1, 4:31 PM · cloud-services-team, Data-Services
gerritbot added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Change #1213461 merged by Majavah:

[operations/dns@master] Add dumps-rsync

https://gerrit.wikimedia.org/r/1213461

Mon, Dec 1, 3:59 PM · cloud-services-team, Data-Services
gerritbot added a project to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge: Patch-For-Review.
Mon, Dec 1, 1:21 PM · cloud-services-team, Data-Services
gerritbot added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Change #1213461 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] Add dumps-rsync

https://gerrit.wikimedia.org/r/1213461

Mon, Dec 1, 1:21 PM · cloud-services-team, Data-Services
taavi triaged T411192: wmcs-wikireplica-dns is horribly inefficient as Medium priority.
Mon, Dec 1, 12:56 PM · cloud-services-team, Data-Services

Fri, Nov 28

fnegri added a parent task for T411248: Plan to make clouddumps more resilient and easier to operate: T391369: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge.
Fri, Nov 28, 1:52 PM · Data-Services, Cloud-VPS, cloud-services-team
taavi moved T411248: Plan to make clouddumps more resilient and easier to operate from Backlog to Dumps on the Data-Services board.
Fri, Nov 28, 11:25 AM · Data-Services, Cloud-VPS, cloud-services-team
taavi added a project to T411248: Plan to make clouddumps more resilient and easier to operate: Data-Services.
Fri, Nov 28, 11:25 AM · Data-Services, Cloud-VPS, cloud-services-team
Maintenance_bot removed a project from T409557: Productionize new clouddb* hosts (clouddb1022-1033): Patch-For-Review.
Fri, Nov 28, 4:30 AM · Data-Services, cloud-services-team, DBA
Marostegui updated the task description for T409557: Productionize new clouddb* hosts (clouddb1022-1033).
Fri, Nov 28, 4:30 AM · Data-Services, cloud-services-team, DBA
Marostegui added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

@taavi @fnegri clouddb1022 and clouddb1023 (both x3 and s3) are ready to be added to the LB.

Fri, Nov 28, 4:29 AM · Data-Services, cloud-services-team, DBA
gerritbot added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Change #1212343 merged by Marostegui:

[operations/puppet@production] clouddb102[23]: Enable notifications

https://gerrit.wikimedia.org/r/1212343

Fri, Nov 28, 4:29 AM · Data-Services, cloud-services-team, DBA
gerritbot added a project to T409557: Productionize new clouddb* hosts (clouddb1022-1033): Patch-For-Review.
Fri, Nov 28, 4:28 AM · Data-Services, cloud-services-team, DBA
gerritbot added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Change #1212343 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb102[23]: Enable notifications

https://gerrit.wikimedia.org/r/1212343

Fri, Nov 28, 4:28 AM · Data-Services, cloud-services-team, DBA

Thu, Nov 27

fnegri reassigned T404570: [wikireplicas] Create views for new wiki tokwiki from fnegri to taavi.
Thu, Nov 27, 4:07 PM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
taavi updated the task description for T411192: wmcs-wikireplica-dns is horribly inefficient.
Thu, Nov 27, 2:37 PM · cloud-services-team, Data-Services
taavi created T411192: wmcs-wikireplica-dns is horribly inefficient.
Thu, Nov 27, 2:34 PM · cloud-services-team, Data-Services
taavi closed T404570: [wikireplicas] Create views for new wiki tokwiki as Resolved.
Thu, Nov 27, 2:31 PM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Stashbot added a comment to T404570: [wikireplicas] Create views for new wiki tokwiki.

Mentioned in SAL (#wikimedia-operations) [2025-11-27T14:31:19Z] <taavi@cumin1003> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database tokwiki (T404570)

Thu, Nov 27, 2:31 PM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Stashbot added a comment to T404570: [wikireplicas] Create views for new wiki tokwiki.

Mentioned in SAL (#wikimedia-operations) [2025-11-27T13:34:20Z] <taavi@cumin1003> START - Cookbook sre.wikireplicas.add-wiki for database tokwiki (T404570)

Thu, Nov 27, 1:34 PM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Maintenance_bot removed a project from T409557: Productionize new clouddb* hosts (clouddb1022-1033): Patch-For-Review.
Thu, Nov 27, 11:31 AM · Data-Services, cloud-services-team, DBA
gerritbot added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

Change #1203042 merged by Majavah:

[operations/puppet@production] hieradata: cloudlb: Add x4 section to wiki replicas

https://gerrit.wikimedia.org/r/1203042

Thu, Nov 27, 10:37 AM · Patch-For-Review, Data-Services, cloud-services-team
gerritbot added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Change #1211083 merged by Majavah:

[operations/puppet@production] conftool-data: Add clouddb1024/5 as x4

https://gerrit.wikimedia.org/r/1211083

Thu, Nov 27, 10:36 AM · Data-Services, cloud-services-team, DBA
fnegri moved T404570: [wikireplicas] Create views for new wiki tokwiki from Blocked to Backlog on the cloud-services-team (FY2025/26-Q1-Q2) board.
Thu, Nov 27, 9:41 AM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Marostegui changed the status of T404570: [wikireplicas] Create views for new wiki tokwiki from Stalled to Open.

This can happen now.

Thu, Nov 27, 6:26 AM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Marostegui closed T404566: Prepare and check storage layer for tokwiki, a subtask of T404570: [wikireplicas] Create views for new wiki tokwiki, as Resolved.
Thu, Nov 27, 6:22 AM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services

Tue, Nov 25

Stashbot added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Mentioned in SAL (#wikimedia-operations) [2025-11-25T15:29:48Z] <marostegui> Add clouddb1023 (s3,x3) to zarcillo T409557

Tue, Nov 25, 3:30 PM · Data-Services, cloud-services-team, DBA
taavi moved T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer from Backlog to Wiki replicas on the Data-Services board.
Tue, Nov 25, 12:05 PM · Patch-For-Review, Data-Services, cloud-services-team
taavi moved T409557: Productionize new clouddb* hosts (clouddb1022-1033) from Backlog to Wiki replicas on the Data-Services board.
Tue, Nov 25, 12:05 PM · Data-Services, cloud-services-team, DBA
gerritbot added a project to T409557: Productionize new clouddb* hosts (clouddb1022-1033): Patch-For-Review.
Tue, Nov 25, 11:55 AM · Data-Services, cloud-services-team, DBA
gerritbot added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Change #1211083 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] conftool-data: Add clouddb1024/5 as x4

https://gerrit.wikimedia.org/r/1211083

Tue, Nov 25, 11:55 AM · Data-Services, cloud-services-team, DBA
Maintenance_bot removed a project from T409557: Productionize new clouddb* hosts (clouddb1022-1033): Patch-For-Review.
Tue, Nov 25, 11:31 AM · Data-Services, cloud-services-team, DBA
taavi moved T409557: Productionize new clouddb* hosts (clouddb1022-1033) from Inbox to Watching on the cloud-services-team board.
Tue, Nov 25, 11:01 AM · Data-Services, cloud-services-team, DBA
taavi added a project to T409557: Productionize new clouddb* hosts (clouddb1022-1033): Data-Services.
Tue, Nov 25, 10:59 AM · Data-Services, cloud-services-team, DBA

Fri, Nov 21

ayounsi added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Thanks, then as it's only a few, option 1 seems best to me. Much less complex to setup and maintain.
Setup the new FQDN, ask people to migrate, give them X months, check activity on the old one, send reminder email, move to the CDN.

Fri, Nov 21, 9:07 AM · cloud-services-team, Data-Services

Mon, Nov 17

bd808 added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

do we know how many systems pull from our rsync ? Maybe it's not worth the hassle of the tcp-proxy if the number is low enough.

Mon, Nov 17, 4:27 PM · cloud-services-team, Data-Services
ayounsi added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Oops, I'm still catching up. Sounds great to minimize user impact. do we know how many systems pull from our rsync ? Maybe it's not worth the hassle of the tcp-proxy if the number is low enough.

Mon, Nov 17, 12:50 PM · cloud-services-team, Data-Services
taavi added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

@taavi Could option 3 be something like what's currently being done for Gerrit ? {T365259}

Mon, Nov 17, 12:37 PM · cloud-services-team, Data-Services
ayounsi added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Could option 3 be something like what's currently being done for Gerrit ? https://phabricator.wikimedia.org/T365259

Mon, Nov 17, 12:35 PM · cloud-services-team, Data-Services
Stashbot added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

Mentioned in SAL (#wikimedia-operations) [2025-11-17T12:13:42Z] <taavi> update CR firewall policy to add the x4 port to wiki replicas related rules T409560

Mon, Nov 17, 12:13 PM · Patch-For-Review, Data-Services, cloud-services-team
gerritbot added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

Change #1205090 merged by Majavah:

[operations/homer/public@master] definitions: Add port for x4 on the wiki replicas

https://gerrit.wikimedia.org/r/1205090

Mon, Nov 17, 12:03 PM · Patch-For-Review, Data-Services, cloud-services-team
Maintenance_bot removed a project from T382607: Decision request - Who runs wikireplicas cookbooks: Patch-For-Review.
Mon, Nov 17, 11:31 AM · Data-Platform-SRE (2025.11.07 - 2025.11.28), Essential-Work, cloud-services-team (FY2025/26-Q1-Q2), Data-Services, Data-Persistence, Cloud Services Proposals
gerritbot added a comment to T382607: Decision request - Who runs wikireplicas cookbooks.

Change #1206196 merged by Btullis:

[operations/puppet@production] Remove the Data Platform SRE team from the contactgroup for wikireplicas

https://gerrit.wikimedia.org/r/1206196

Mon, Nov 17, 10:53 AM · Data-Platform-SRE (2025.11.07 - 2025.11.28), Essential-Work, cloud-services-team (FY2025/26-Q1-Q2), Data-Services, Data-Persistence, Cloud Services Proposals
gerritbot added a project to T382607: Decision request - Who runs wikireplicas cookbooks: Patch-For-Review.
Mon, Nov 17, 9:54 AM · Data-Platform-SRE (2025.11.07 - 2025.11.28), Essential-Work, cloud-services-team (FY2025/26-Q1-Q2), Data-Services, Data-Persistence, Cloud Services Proposals
gerritbot added a comment to T382607: Decision request - Who runs wikireplicas cookbooks.

Change #1206196 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove the Data Platform SRE team from the contactgroup for wikireplicas

https://gerrit.wikimedia.org/r/1206196

Mon, Nov 17, 9:54 AM · Data-Platform-SRE (2025.11.07 - 2025.11.28), Essential-Work, cloud-services-team (FY2025/26-Q1-Q2), Data-Services, Data-Persistence, Cloud Services Proposals

Fri, Nov 14

fnegri moved T404570: [wikireplicas] Create views for new wiki tokwiki from In progress to Blocked on the cloud-services-team (FY2025/26-Q1-Q2) board.
Fri, Nov 14, 5:15 PM · cloud-services-team (FY2025/26-Q1-Q2), Data-Services
Marostegui added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

x3 is already live on the replicas and has been for a while (T390954).

Fri, Nov 14, 10:34 AM · Patch-For-Review, Data-Services, cloud-services-team
taavi added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

x3 is already live on the replicas and has been for a while (T390954).

Fri, Nov 14, 10:07 AM · Patch-For-Review, Data-Services, cloud-services-team
gerritbot added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

Change #1205093 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: cloudlb: Add x1 section to wiki replicas

https://gerrit.wikimedia.org/r/1205093

Fri, Nov 14, 10:05 AM · Patch-For-Review, Data-Services, cloud-services-team
gerritbot added a comment to T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.

Change #1205090 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] definitions: Add port for x4 on the wiki replicasa

https://gerrit.wikimedia.org/r/1205090

Fri, Nov 14, 9:58 AM · Patch-For-Review, Data-Services, cloud-services-team