Create Task
Maniphest T432217

wikifunctions Memcached: High incidence of failures on MemcachedWrapper::set operations
Open, In Progress, HighPublicBUG REPORT
Actions

Description

Description

There's a high incidence of the error MediaWiki\Extension\WikiLambda\Cache\MemcachedWrapper::set: failed to set broadcast prefixed /*/wf-wan/WikiLambdaFunctionCall::zobject|Z1K1|Z7%23274546,Z7K1|Z14280%23260512,Z14280K1|Z1K1|Z13518%23282417,Z13518K1|12345,,Z14280K2|Z1553%23215642,,doValidate|1, on eqiad with error 10: SERVER ERROR

https://logstash.wikimedia.org/goto/7a6494a2c5b2dbfc88166187ae072afb

Probably due to values being too large. We should log size, diagnose this and figure out possible solutions.

Completion checklist

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Increase logging to diagnose MemcachedWrapper::set frequent failureGenoveva Galarzamediawiki/extensions/WikiLambdamaster+113 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

gengh created this task.Jul 15 2026, 10:20 AM
gengh renamed this task from <Short one-liner describing the problem> to wikifunctions Memcached: High incidence of MemcachedWrapper::set operations.
gengh renamed this task from wikifunctions Memcached: High incidence of MemcachedWrapper::set operations to wikifunctions Memcached: High incidence of failures on MemcachedWrapper::set operations.
Jdforrester-WMF subscribed.Jul 15 2026, 10:23 AM

Are you sure this is an output-too-large issue? Z14280 is display-natural-number and its output would be a few dozen bytes, wouldn't it? From my quick digging a few weeks ago I think this is more a memcached-busy signal, maybe? But I'm not sure.

Jdforrester-WMF added a comment.Jul 15 2026, 10:24 AM

(Also I'd say that this is definitely in the scope of T431428 and so OKR work.)

gengh added a comment.Jul 15 2026, 10:37 AM

Are you sure this is an output-too-large issue? Z14280 is display-natural-number and its output would be a few dozen bytes, wouldn't it? From my quick digging a few weeks ago I think this is more a memcached-busy signal, maybe? But I'm not sure.

It might be, this is a WikilambdaFunctionCall key, so its storing the result and the metadata. If there's an error, the metadata can get long. However, I would like to add some better logging to diagnose the problem rather than speculate. We can also see if it's being rejected due to busyness!

Btw, I was searching for a task before opening this one, but I didn't find one. You said you were looking into this, do you know if there's task? happy to close this one.

(Also I'd say that this is definitely in the scope of T431428 and so OKR work.)

Yes!!! agreed.

Jdforrester-WMF edited projects, added: Abstract Wikipedia team (27Q1 (Jul–Sep)), OKR-Work; removed: Abstract Wikipedia team.Jul 15 2026, 10:39 AM
In T432217#12123334, @gengh wrote:

Are you sure this is an output-too-large issue? Z14280 is display-natural-number and its output would be a few dozen bytes, wouldn't it? From my quick digging a few weeks ago I think this is more a memcached-busy signal, maybe? But I'm not sure.

It might be, this is a WikilambdaFunctionCall key, so its storing the result and the metadata. If there's an error, the metadata can get long. However, I would like to add some better logging to diagnose the problem rather than speculate. We can also see if it's being rejected due to busyness!

Fair!

Btw, I was searching for a task before opening this one, but I didn't find one. You said you were looking into this, do you know if there's task? happy to close this one.

No, just poking.

(Also I'd say that this is definitely in the scope of T431428 and so OKR work.)

Yes!!! agreed.

Will tag as such.

Jdforrester-WMF added a parent task: T431428: [Hypothesis] DE2.1.3 - Durable fragments.Jul 15 2026, 10:39 AM
gengh claimed this task.Jul 15 2026, 10:53 AM
gerritbot added a comment.Jul 15 2026, 10:54 AM

Change #1311011 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] Increase logging to diagnose MemcachedWrapper::set frequent failure

https://gerrit.wikimedia.org/r/1311011

gerritbot added a project: Patch-For-Review.Jul 15 2026, 10:54 AM
Jdforrester-WMF changed the task status from Open to In Progress.Jul 15 2026, 2:32 PM
Jdforrester-WMF triaged this task as High priority.
Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (27Q1 (Jul–Sep)) board.
gerritbot added a comment.Jul 15 2026, 4:03 PM

Change #1311011 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Increase logging to diagnose MemcachedWrapper::set frequent failure

https://gerrit.wikimedia.org/r/1311011

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2026, 4:31 PM
ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.12; 2026-07-21).Jul 15 2026, 5:00 PM
gengh added a comment.EditedThu, Jul 23, 11:24 AM

After release of 1.47.0-wmf.12 I can observe the following from the logs:

Time frame: Jul 22, 2026 @ 08:00:00.000 - Jul 23, 2026 @ 08:00:00.000
https://logstash.wikimedia.org/goto/2c8c4787ee45ddb3d91d85649461caf9

There doesn't seem to be any significant distinction between keys, this is the distribution of operations, failures and successes through different keys

KeyTotal setsSuccessful sets% successFailed sets% failureorigin
WikiLambdaObjectStorage36133693.07%256.93%wikifunctionswiki
WikiLambdaClientFunctionCall97296098.77%121.23%many (see error distribution below)
WikiLambdaAbstractFragment16,36816,10798.41%2611.59%abstractwiki
WikiLambdaFunctionCall13,88512,94893.25%9376.75%wikifunctionswiki
All31,59630,35996.08%1,8285.79%-

Origin wiki of WikiLambdaClientFunctionCall failures

keytotalwikifunctionswikibnwiktionaryabstractwikihrwiktionarytestwiki
WikiLambdaClientFunctionCall97296.40%1.70%0.80%0.40%0.30%

Namespace distribution of operations, successes and failures:

Namespace% of operations% of successes% of failures
mw-wikifunctions58.8%58.9%74.3%
mw-jobrunner29.9%30.0%18.6%
mw-api-int7.6%7.3%7.1%
mw-api-ext3.7%3.7%0.0%

Writes to our Memcached instance from function-orchestrator look healthy:

keytotal ops% from totalsuccessfailure% failure
all137490100.00%137479110.01%
_DEF3431724.96%3431070.02%
_ALLDEPS10288774.83%10288340.00%
_WIKIDATA_DEF2860.21%28600.00%

Collected stats on failure for different keys show no sign of size-related issue (not even remotely).
Logs show "server: unavailable", which means that Memcached::getStats returned false and may indicate some connection failure.

WikiLambdaObjectStorage:

{"size_serialized":2514,"size_json":2839,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}
{"size_serialized":17811,"size_json":22009,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}

WikiLambdaClientFunctionCall

{"size_serialized":798,"size_json":1106,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}
{"size_serialized":95,"size_json":78,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}

WikiLambdaAbstractFragment

{"size_serialized":287,"size_json":324,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}
{"size_serialized":146,"size_json":153,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}

WikiLambdaFunctionCall

{"size_serialized":2819,"size_json":3395,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}
{"size_serialized":2888,"size_json":3464,"compression_enabled":true,"compression_type":2,"serializer":1,"server":"unavailable"}

This info seems to point at a transient, unavailability issue only happening in wikifunctionswiki,

99of9 subscribed.Thu, Jul 23, 12:53 PM
RLazarus subscribed.Thu, Jul 30, 3:09 PM
Jdforrester-WMF mentioned this in T399098: MediaWiki\Extension\WikiLambda\WikifunctionsClientStore::fetchFromFunctionCallCache cache miss while fetching {key}.Thu, Aug 20, 5:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits