I think that would be I/F now.

In T433924#12307540, @JMeybohm wrote:

In T433924#12303244, @Clement_Goubert wrote:

Agreed but I think an exception should be carved out for staging, as it would simplify the testing loop for developers by cutting out patching deployment-charts to bump the image version.

I can see the benefit, but I am not a huge fan and would advice against. Partly because it's a derivation from how production works, partly because it makes it harder to track which version is currently running and because it makes rollbacks pretty hard. There is also the "quirk" that changing an image tag to :latest does not update the default imagePullPolicy. E.g. a new deployment created with a :latest tag will have imagePullPolicy: Always set by default. Changing the tag for an image of an existing deployment to :latest does not update the imagePullPolicy, it will still be IfNotPresent.

In T437209#12310454, @Tgr wrote:

What exactly do you want to migrate? There's quite a bit of configuration.

This is a stopgap measure until T436223: Add backend information to webrequests in data lake can be implemented to avoid crumbling Druid through Turnilo when we need to separate requests that go to mw-web or rest-gateway

As far as complex queries go, T436223: Add backend information to webrequests in data lake would help, I also asked @JAllemandou to create a derived column for is_api_call in the meantime for

URI Host != 'thumb.wikimedia.org'
URI Path !~ (?i)^/(|w/api.php|w/rest.php|api/|core|service|feed)
Webrequest source = text

I'm gonna be bold and close this task. Invalid is the closest status I can think of, but it is not a reflection on the information or discussion that happened in this task.

Agreed but I think an exception should be carved out for staging, as it would simplify the testing loop for developers by cutting out patching deployment-charts to bump the image version.

I ran debmonitor manually and it's gone.

Updated, this should now be cleared but I'll check after the next debmonitor run

On it, this is mw-debug in staging which hasn't been updated since April because it's mostly sitting unused.

In T377497#12295133, @Urbanecm_WMF wrote:

In T377497#12295126, @MLechvien-WMF wrote:

this task came up as the last reason why we have PHP on deploy hosts.

Is this actually the last reason? That doesn't match what I remember. For example, sql enwiki executed on the deployment host also shells out to PHP internally. I'm not sure if that has a task though.

commonswiki Error: 2006 MySQL server has gone away

In T437033#12291001, @Mvolz wrote:

In T437033#12290640, @Scott_French wrote:

[...]
We'd also have to put our public key in some open repository somewhere, that might also be something service ops would do?

Deployed, but I can't seem to find the metric being exported by mw-debug's statsd exporters after triggering errors with curl -k -H 'Host: en.wikipedia.org' 'https://mwdebug.discovery.wmnet:4444/w/fatal-error.php?action=oom&password=XXXXXXX'

I'll try to sneak it in after T436781: Live-Test - September 2nd 2026

I'll be your scroll bearer, you can set up a meeting so we can go through the prologue detail.

The outbound calls to the LoC are made through url-downloader right?
Traffic I'm not too familiar with squid (the http proxy used by url-downloader) but would there be a way to add rate limits for some domains there?
Another possible solution would be something like:

• Add url-downloader to the service mesh
• Add redis-backed rate limiting to the service mesh
• Set custom policies for the different external URLs used by citoid

However this would be costly in engineering time, adding a lot of complexity to the request path, and probably low-value for reuse.

Additional parameters for a new run:

• Correlate with KAPOW scores to see if interesting information pops up
• Correlate with known_client to see if we need to implement known_client ratelimit bypass
• Estimate bandwidth savings avg(response_size)*blocked_requests, possibly estimate over X-Is-Browser and top clients

From analysis of requests done in T434179: Data analysis request for media rate limiting, current limit of 600MB/min would:

• Block 0.0088% of clients, 0.0335% of requests
• Mostly in the low-medium X-Is-Browser

In conclusion, error-params.php is used to configure php7-fatal-error.php to bubble up a counter of uncaught exceptions that are not rendered by mediawiki. I think if only to avoid a broken counter and confusion about what is in use, merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1049625 to close T356814 is the best course of action.

I didn't delete the failed job properly, my bad.

Transient issue, the following run completed correctly.

cleanup-upload-stash-29800860   Failed     0/1           32h        32h
cleanup-upload-stash-29802300   Complete   1/1           35m        8h

I don't have a particularly strong opinion on the pull-side access, because IMO the idea is that the images don't show up on the public registry but not necessarily to gate pull access from production in general.
On the push side, maybe gitlab trusted runners can have access to restricted, but I'd be in favour of maybe testing that out in a separate namespace at first?

Just merged, we'll reopen if it doesn't help.

In T433547#12238921, @HCoplin-WMF wrote:

@Clement_Goubert -- we're potentially leaning towards just doing the testing live, since the new routes don't exist in the Gateway anyway. Do you see any risk with that? Should we start with local config first?

In T435364#12237928, @colewhite wrote:

The only usage of error-params.php I can locate is https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/profile/files/mediawiki/php/php7-fatal-error.php

Is php7-fatal-error.php in use anywhere anymore?

In T427668#12236539, @Blake wrote:

Hmmmm, I'm not sure how to actually point the new grafana dashboard at mw-pretrain... is there a way to filter the dashboard on namespace?

I'll put together a patch for changing the service state. Thanks!

It seems to have lasted for about 24 hours then stopped. I've tried to match with operations or alerts in production but can't find a smoking gun.

This is done, we can investigate adding a fourth machine for load spreading and turning up the replication factor in MediaWiki to tolerate two down servers at the same time (currently one) if needed, but that would be the subject of a new task.

I think setting it to send to the exporter should be fine then.

@jasmine_ Can you take care of adding the servers to site.pp and preseed.yml please?

@Joe @Krinkle I'm trying to remember two things:

• Is this file used in "normal operations", meaning called by MediaWiki to handle fatal errors in production, or is it only used to trigger errors at-will during testing/debugging?
• Is sending these errors to statsd an integral part of that flow, and can it be changed to what MediaWiki already does in its normal operation: sending to statsd-prometheus-exporter ?

I'm rerunning the couple jobs that failed and didn't have a subsequent successful run.

In T427999#12229182, @Ladsgroup wrote:

So now it's taking 500 operations per second like it's nothing. Everything looks fine but from time to time, it can't connect to any of the redis instances: https://logstash.wikimedia.org/goto/956345055892636f37fb651ab4395c7a it's quite rare but just flagging it in case it's easy to fix.

That particular instance of the job did fail, but there's one running right now that seems fine

Looks like something is wrong with the prometheus export, I can't find the new hosts in the redis_up metric, will dig further.

All jobs should now be re-running or completed.

List of failed jobs left to rerun:

In T435056#12220764, @Ladsgroup wrote:

My guess is T434979: Periodic script failure due to "Fatal error: no version entry for `Please`"

I'm rerunning the failed jobs I can find btw.

Caused by an error in global runner script, should be fixed now, so re-running manually.