I have granted the quota increase. Do please create a subticket about reverting, or follow up here when you're done with the migration.

+1 lgtm

ok, I'm caught up now! I hadn't realized that we had reproducible uuids for new hosts, I was still just taking nova's word for it :(

The stated quota increase is to make room for an identical VM to the main one we use now, which is of special flavour g4.cores24.ram122.disk20

Our tool cannot run in Toolforge

I made another pass at rebalancing but 1004 is still taking a bit longer than 24 hours (24 hours and 6 minutes) to complete so this needs yet one more pass.

There are lots of reasons for a host to become unreachable, but falling off the network is one of them. This dashboard is a good place to start:

I think I'm behind on something. When you say...

This is now done. I did wind up doing a staggered reboot of all servers because NFS clients are not great at handling change.

Typically I would restart nfs clients after an nfs migration to make sure they're mounting the new server properly. So the most straightforward thing is for me to just shut down all the -app hosts in the cluster, flip the nfs switch, and then start them up again. That's also the best way to avoid any weird in-between states; does that sound ok? I'm assuming the above jobs run on the app hosts and not the apache host, please lmk if that's wrong.

Thank you @jsn.sherman

Thanks! I have put this host back in service and we'll keep an eye on it.

I have a start on this

this needs more testing before I ship it to eqiad1, but the specific blubber update is done.

this is done. The old server dumps-nfs-1 is shutdown and can be deleted after a few days of good health.

Oh, also, to be clear: This is entirely a storage-based solution so far. Ideally we would be blocking and deleting users that cross the line but at the moment we're not doing anything to prevent users from re-creating their notebooks every 24 hours.

The current running code regards 'freeroot' as a strike against a notebook but not an automatic block. If that gets file usage under control then that may be good enough but I'll keep an eye on it.

Let's do a memory scan on this server too, while it's out of service. Thanks!

Oh -- it's out of service now, so y'all can switch it off whenever.

For now: this host is in the maintenance aggregate and drained of all user VMs.

And, just like T410470, a totally useless syslog during the downtime:

find . -name README.md -exec grep "Foxytoux" {} \; | wc

3401   32322  244072

In T431340#12091788, @bd808 wrote:

Is the freeroot you are seeing @Andrew this https://github.com/foxytouxxx/freeroot project?

A bit of documentation added here: https://wikitech.wikimedia.org/w/index.php?title=PAWS%2FGetting_started_with_PAWS&diff=2433391&oldid=2254420

In T429578#12082915, @BLiviero-WMF wrote:

how does it decide what to remove? (size, age, last use, patterns?)

Here is a snippet of the kind of thing the attached patch would do. Might be too aggressive...

lgtm

I'm pretty sure that the cleanup script that copies things into /srv/paws/files-to-remove/ was lost when I rebuilt the NFS server. Going to wait 24 hours to confirm that that's right -- I certainly can't find a job scheduled anyplace.

I'll go ahead and run this update next week if I don't hear anything here in the meantime.

This is almost certainly the result of a recent upstream change which limits app credential delegation. In theory we've already activated all the workarounds but I'll see if there's another one for this exact case. In the meantime: painful as this is you can likely do this via Horizon which allows access as a human user.

Belinda -- assigning back to you to close or follow up as needed.

These numbers are never full stable due to ongoing decoms and new hardware, but here's a snapshot of cloud resources:

Here are the two nfs mounts we will be replacing:

we're pitching NFS overboard as part of T402054: twl: Replace deprecated Bullseye VMs in Cloud VPS this sprint.

Hello @jsn.sherman! I have not yet started work on this, but twl is one of the projects that needs an NFS update sometime soon (see T429793). Do you have any idea what went wrong with the NFS mount? And, is this a project that still requires nfs for sharing files between hosts?

I'm not sure why this is happening. Most likely it's due to a different cookbook (maybe wmcs.openstack.cloudvirt.safe_reboot) failing and leaving things in an inconsistent state.

+1

I built fresh base images for Bookworm and Trixie which should resolve this.

In T428196#12041187, @Don-vip wrote:

In T428196#12040703, @Andrew wrote:

You're right! I will take this on -- I just opened T429793. Do you have any concerns about when the downtime is? The most likely scenario is: a few minute outage followed by a reboot of all project VMs.

Thank you ❤️ No concern for the downtime, I guess a small message on V2C talk page is enough to warn people about the maintenance :)

• Add a runbook to the alert -- if it's generic, make an index page for all nfs full alerts
• Add a daily timer to delete all files older than 2 weeks from /srv/paws/files-to-remove/ on the paws NFS server

There are some very brief docs about this here: https://wikitech.wikimedia.org/wiki/PAWS/Admin#Deleting_user_data_in_case_of_spam_or_credential_leaks

In T402738#12040229, @fgiunchedi wrote:

In T402738#12033703, @Andrew wrote:

In T402738#12033646, @fgiunchedi wrote:

wrt metrics, I'm wondering how easy it would be to bundle node-exporter in the trove images. Meaning we would get vm observability (most importantly disk space) like standard VMs, what do you think @Andrew ? Along the same lines, we can also ship mysqld-exporter and get standard observability

We don't currently build our own Trove images. We certainly could although it was add some pain to the upgrade process. If those are standard debian packages we could probably install them at boot time using some sort of fancy cloud-init conditionals, similar to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302236/16/modules/openstack/templates/nova/vendordata.txt.erb

Thank you, indeed those are standard Debian packages so we could install them at cloud-init time, possibly with minimal configuration as well

In T429544#12039331, @fgiunchedi wrote:

I'm not sure if we have an end to end guide for users on how to self-service migrate their nfs servers. Maybe @Andrew knows

In T428196#12035176, @Don-vip wrote:

Hi @Andrew , I received this e-mail as maintainer of the video CloudVPS project. The only remaining instance to migrate is the NFS one, I think we agreed long time ago this one would be for you, am I correct? If I'm wrong I can look at it, no problem.

Indeed, the alert seems to have cleared. Thank you!

In T402738#12033646, @fgiunchedi wrote:

wrt metrics, I'm wondering how easy it would be to bundle node-exporter in the trove images. Meaning we would get vm observability (most importantly disk space) like standard VMs, what do you think @Andrew ? Along the same lines, we can also ship mysqld-exporter and get standard observability