Missing user-agent error message from Varnish is malformed xml (img tag not being closed)
Closed, InvalidPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Calling this URL to request information about Freya Ridings from the api endpoint in xml format:
https://en.wikipedia.org/w/api.php?format=xml&action=query&prop=extracts&exlimit=1&titles=Freya_Ridings

Returns the xml below. Line 31 which starts "<a href="https://www.wikimedia.org"><img id="logo" ..." - the "<img" tag is not properly closed so xml parsing fails.

This applies to all other urls tried on the wikipedia site this morning.

<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<title>Wikimedia Error</title>
<style>
* { margin: 0; padding: 0; }
body { background: #fff; font: 15px/1.6 sans-serif; color: #333; }
.content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 640px; }
.footer { clear: both; margin-top: 14%; border-top: 1px solid #e5e5e5; background: #f9f9f9; padding: 2em 0; font-size: 0.8em; text-align: center; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
.content-text { overflow: hidden; overflow-wrap: break-word; word-wrap: break-word; -webkit-hyphens: auto; -moz-hyphens: auto; -ms-hyphens: auto; hyphens: auto; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645ad; text-decoration: none; }
a:hover { text-decoration: underline; }
code { font-family: sans-serif; }
summary { font-weight: bold; cursor: pointer; }
details[open] { background: #970302; color: #dfdedd; }
.text-muted { color: #777; }
@media (prefers-color-scheme: dark) {
  a { color: #9e9eff; }
  body { background: transparent; color: #ddd; }
  .footer { border-top: 1px solid #444; background: #060606; }
  #logo { filter: invert(1) hue-rotate(180deg); }
  .text-muted { color: #888; }
}
</style>
<meta name="color-scheme" content="light dark">
<div class="content" role="main">
<a href="https://www.wikimedia.org"><img id="logo" src="https://www.wikimedia.org/static/images/wmf-logo.png" srcset="https://www.wikimedia.org/static/images/wmf-logo-2x.png 2x" alt="Wikimedia" width="135" height="101">
</a>
<h1>Error</h1>
<div class="content-text">

<p>Our servers are currently under maintenance or experiencing a technical issue</p>
</div>
</div>
<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class="text-muted"><code>Request served via cp3069 cp3069, Varnish XID 155453481<br>Upstream caches: cp3069 int<br>Error: 403, Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119 at Tue, 19 Aug 2025 10:15:33 GMT<br><details><summary>Sensitive client information</summary>IP address: 87.102.46.73</details></code></p>
</div>
</html>

What happens?:

Line 31 which starts "<a href="https://www.wikimedia.org"><img id="logo" ..." - the "<img" tag is not properly closed so xml parsing fails.

What should have happened instead?:

The xml should be returned with the <img> tag being closed prior to the </a> tag closure.

Other information (browser name/version, screenshots, etc.):

Tested using Windows API 'HTTPClient.GetAsync' and verified using Chromium and viewing the returned response directly.

Event Timeline

Thanks for reporting this! Tagging MediaWiki-Action-API though might be a parser issue.

Aklapper renamed this task from Wikipedia XML API queries are returning malformed xml including an img tag that is not being closed to All Wikipedia XML API queries return malformed xml (img tag not being closed).Aug 19 2025, 11:05 AM

Follow up: Looks like this only affects calls that are now being rejected due to the new user-agent limiting T400119

Calls with a valid user agent are returning xml that parses fine, so I suspect its limited to the system that is applying the new limits to the calls rather than more widespread.

Follow up: Looks like this only affects calls that are now being rejected due to the new user-agent limiting T400119

Calls with a valid user agent are returning xml that parses fine, so I suspect its limited to the system that is applying the new limits to the calls rather than more widespread.

Yeah, I can repo with curl -s -a 'https://en.wikipedia.org/w/api.php?format=xml&action=query&prop=extracts&exlimit=1&titles=Freya_Ridings' -H 'User-Agent:' > a.xml && google-chrome a.xml. It looks like the static error message has bad XML (over a valid HTML5 void element).

Bot XML parsers could probably handle getting this kind of HTML back, though they might fail on void elements or such. The structure wouldn't match anything anyway, so the bot would mostly have to rely on the status code.

Seems like relevant code is ./modules/profile/files/mediawiki/php/php7-fatal-error.php in operations/puppet.

It's hard to "fix" by closing the tag (XHTML style) since that isn't really HTML5 style (though browsers handle it via "do what I mean").

Assuming such bots have a simple accept-encoding (such as just application/xml or application/json), perhaps php7-fatal-error.php could use that to return XML or JSON variants instead of HTML. The script currently relies on PHP's default content-type header ("content-type: text/html; charset=utf-8").

In any case, bots should be weary of parsing the response for 4XX/5XX errors with an unexpected content-type.

@PeterJarrett What issues did problem cause for you? Where you running a bot?

It's just a little desktop application running under Windows.Net that lets me quickly see wikipedia articles for music artists - when you select a song and click to see the Wikipedia info it calls the api to read the xml to show the slug of summary text in a little window. In this case, it (was) using the default HttpClient object which doesn't include a user-agent header - I've now changed it so that it includes a user agent so it's all working again as no longer getting the (slightly invalid) error message :-)

It's hard to "fix" by closing the tag (XHTML style) since that isn't really HTML5 style (though browsers handle it via "do what I mean").

Void tags with a slash are explicitly valid HTML5.

Pppery renamed this task from All Wikipedia XML API queries return malformed xml (img tag not being closed) to Missing user-agent error message from Varnish is malformed xml (img tag not being closed).Sep 23 2025, 5:03 AM
Pppery edited projects, added: Traffic; removed: MediaWiki-Action-API.

This doesn't seem to be Traffic related, unless I am missing something. The CDN is simply rendering the output generated by the app layer.

taavi subscribed.

That error message comes from the CDN, not from the app layer. Anyway, the error page is served with a content-type: text/html header and that page is valid HTML, so it's up to the client to handle that response instead of blindly assuming it'll be XML. I don't think it's reasonable to expect the CDN to format errors in different formats following app-specific query parameters.