Create Task
Maniphest T393024

lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

What happens?:

  • For some reason the webbrowser blocks scripts and style sheets loaded from tools-static.wmflabs.org.

What should have happened instead?:

  • The scripts and style sheets should be loaded correctly.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

  • The browser console log on Firefox (137.0.2 on Ubuntu) shows that Javascipt and css from tools-static.wmflabs.org is blocked with the error message: NS_BINDING_ABORTED same for Firefox 138.0 on MacOS
  • Chromium (135.0.7049.114 on Ubuntu) has the same error with the error message ERR_BLOCKED_BY_ORB
  • Safari seems not to be affected

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dynamicproxy: Declare functions as localoperations/puppetproduction+2 -2
dynamicproxy: Add missing redis_shutdown() calloperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

GPSLeo created this task.Apr 30 2025, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2025, 3:46 PM

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

JJMC89 edited projects, added Toolforge; removed Cloud-Services.Apr 30 2025, 3:50 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptApr 30 2025, 3:50 PM
taavi renamed this task from All tools using tools-static.wmflabs.org broken to lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request.Apr 30 2025, 3:50 PM
taavi added a parent task: T379175: Enable IPv6 for the Cloud VPS web proxy.
taavi edited projects, added Cloud-VPS; removed Toolforge.
taavi triaged this task as High priority.Apr 30 2025, 3:54 PM
taavi subscribed.

Thanks for the report! This seems to be a problem with the new proxies that T379175: Enable IPv6 for the Cloud VPS web proxy is introducing. The particular error being logged is P75678, which I thought https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/0e4620fe503a5cc46b118492d27528dbc8b8f5ce%5E%21/ fixed but apparently not.

For now I've dropped the AAAA record from tools-static.wmflabs.org which means that you should be routed to the working old proxies as soon as DNS TTLs expire.

Stashbot added a comment.Apr 30 2025, 3:54 PM

Mentioned in SAL (#wikimedia-cloud) [2025-04-30T15:54:46Z] <taavi> project-proxy drop AAAA record from tools-static T393024

taavi claimed this task.May 6 2025, 10:45 AM
taavi added a comment.May 6 2025, 11:16 AM

One thing to note is that this is only happening with HTTP 2.0 requests, and seemingly only for requests that are not the first request on a given connection.

gerritbot added a comment.May 6 2025, 11:25 AM

Change #1142572 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Add missing redis_shutdown() call

https://gerrit.wikimedia.org/r/1142572

gerritbot added a project: Patch-For-Review.May 6 2025, 11:25 AM

Change #1142572 merged by Majavah:

[operations/puppet@production] dynamicproxy: Add missing redis_shutdown() call

https://gerrit.wikimedia.org/r/1142572

Maintenance_bot removed a project: Patch-For-Review.May 6 2025, 11:30 AM
taavi added a comment.May 6 2025, 12:06 PM

I tried to:

  • Handle possible Lua errors in the code, and set lua_socket_log_errors off; instead (as recommended in the lua-nginx-redis upstream README)
  • Use :close() instead of :set_keepalive()

Neither of those had any impact. Now I've set loglevel verbose in the Redis configuration in the hopes that that will be more helpful.

taavi added a comment.May 6 2025, 12:11 PM
In T393024#10795574, @taavi wrote:

Now I've set loglevel verbose in the Redis configuration in the hopes that that will be more helpful.

Nothing useful in there either. I've also tried to set the code to talk to an old Redis server (proxy-03) with no impact, suggesting that the issue is within Nginx and not the Redis server.

gerritbot added a comment.May 6 2025, 1:31 PM

Change #1142598 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Declare functions as local

https://gerrit.wikimedia.org/r/1142598

gerritbot added a project: Patch-For-Review.May 6 2025, 1:31 PM
gerritbot added a comment.May 6 2025, 1:35 PM

Change #1142598 merged by Majavah:

[operations/puppet@production] dynamicproxy: Declare functions as local

https://gerrit.wikimedia.org/r/1142598

taavi added a comment.May 6 2025, 1:42 PM

The latest patch seems to have done the trick. I'm confident enough that I'm moving some tools-static traffic back to the new proxy to validate the fix with a higher amount of traffic. I'll check back tomorrow and will close the task if there are no additional reports of issues or logged errors.

Maintenance_bot removed a project: Patch-For-Review.May 6 2025, 2:31 PM
taavi closed this task as Resolved.May 7 2025, 8:02 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits