I deleted all the instances in the project with openstack database instance delete {id}, and reset the quotas with update quota_usages set reserved=0, in_use=0 where tenant_id='tofuinfratest';

This is still happening:

Accessible again, closing.

In T410083#11373608, @taavi wrote:

If the initial set of Puppet runs fail then the instance will get stuck in a weird state. I have manually kicked off a run which might or might not get it working again.

If the initial set of Puppet runs fail then the instance will get stuck in a weird state. I have manually kicked off a run which might or might not get it working again.

@taavi, I removed all puppet roles, and it still doesn't allow me to SSH in.

The instance has Puppet roles applied that do not compile:

root@gitlab-docker-runner-v2:~# sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Docker::Configuration]: expects a value for parameter 'settings' (file: /srv/puppet_code/environments/production/modules/docker/manifests/init.pp, line: 22, column: 5) on node gitlab-docker-runner-v2.analytics.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

And as the Puppet compilation is failing, the instance won't have the settings needed for logging in applied on it.

This is intentional. Please see note at top of page "NOTE: Due to a future release we are temporarily limiting updates to tier 3 wikis. If you are a tier 1 wiki and using this page please file a phabricator ticket and tag @Jdlrobson-WMF to request it"

We now have ctt-prv-04 set up and configured and a first round of visual diff runs have successfully completed and we have a proxy set up for its services at http://prv-tests.wmcloud.org.

Thank you for the explanation, this is way clearer. I was confused believing we've made something in recent days which caused us to go over-quota.

@dcaro can you develop a bit why we suddenly get this message, and where we can find our current object storage usage and total entitled storage? I struggle to find the info in Horizon console.

Honestly I feel like we can just ignore this task, or delete these objects, or both.

Make sure you have mwoffliner selected as your project:

@dcaro can you develop a bit why we suddenly get this message, and where we can find our current object storage usage and total entitled storage? I struggle to find the info in Horizon console.

You're right, it's for database backups originating from Horizon. You're also right about the date it was enabled. Of course, since we have our own database backups, these are useless. The actual mystery to me is why we suddenly got this message. Either way, it's not something we are using so it can be deleted or just ignored.

@Audiodude do you have any idea what this object storage is used for? Looks like DB backups from Oct 1 2024, but these are not supposed to be there and they never got updated. I'm a bit confused.

We probably don't need a new large disk. I think there is a spare unmounted volume allocated for ctt-qa-03 which could be used for this one.

Optimistically closing this since I updated senders permissions on the group email.

In T386416#11027704, @taavi wrote:

This doesn't seem to have ever worked; the notification emails are being bounced back due to a permission issue on the Google group end. Please fix the settings to allow [email protected] and [email protected] to post to the list.

This doesn't seem to have ever worked; the notification emails are being bounced back due to a permission issue on the Google group end. Please fix the settings to allow [email protected] and [email protected] to post to the list.

Change #1165962 merged by Majavah:

[operations/puppet@production] cloudnfs: Add dumpstorrents project to dumps mounts

https://gerrit.wikimedia.org/r/1165962

Change #1165962 had a related patch set uploaded (by Samtar; author: Samtar):

[operations/puppet@production] labstore: Add dumpstorrents project to dumps mounts

https://gerrit.wikimedia.org/r/1165962

@TheDJ: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome!
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!