forgot to mention that the reindex was started yesterday on the two other clusters (eqiad, cloudelastic)

A/B test results on other wikis: https://people.wikimedia.org/~dcausse/T404858-completion-default-sort-2.html

Going to start the reindex today

went with the approach of enabling on the five georgian wikis at once, please let me know if a more conservative approach (one wiki first) is preferable.

The update process has been fixed.
Existing stale data in the search index will get fixed when:

• a new revision of the page is created
• a template change propagates
• when the continuous cleanup mechanism processes a page with stale data

In T410758#11409444, @doctaxon wrote:

In T410758#11408729, @dcausse wrote:

Unless you noticed that that the regex got a lot slower recently and that more queries are timing out I think it is safer to keep the 15s internal timeout.

But Wikipedia content is more and more increasing and it will reach one time that 15s are not safer any more. What then? Shall we decrease the timeout more and more? I guess we need a search machine that is able to handle all the big content properly if CirrusSearch does not.

CirrusSearch has to be careful when specifying timeouts of a regex query.
Regex queries are particularly costly and may cause a lot of stress on the servers if not properly protected.
The 15s timeouts has been setup for this, to ensure that the search backend return before any other timeouts are applied otherwise this might mean that a costly query mist continue to run outside of the concurrency protection (T152895).
Unless you noticed that that the regex got a lot slower recently and more queries I think it is safer to keep the 15s internal timeout.

Thanks for reporting this, I think there are two different issues that allowed such suggestions to appear:

• defaultsort is indeed not properly removed from the search index when it's erased, a null value unfortunately tells the system to ignore it when updating it, this needs to be fixed for this field
• defaultsort values are allowed to help completion only if they match a particular pattern, this pattern seems too permissive and should be corrected to limit the possibility of such vandalism to impact search suggestions in the future

Should be ready once 1.46.0-wmf.3 is deployed, earliest would be Thursday nov 20 but probably safer to wait til the following week in case we rollback.

In T408533#11383532, @achou wrote:

Hi @pfischer @dcausse, ML team wants to follow up on the initial ingestion process. As you mentioned before, the Search platform team has a manual script for this purpose. Can the ML team execute this on our end (e.g., in statbox)? Or can only the Search team execute it?

it is a bit cumbersome to run unfortunately and some adaptations have to be made (we only used it to backfill article countries). The script is in stat1009.eqiad.wmnet:~dcausse/articlecountry:

• backfill_articlecountry.scala the spark job that reads hdfs://analytics-hadoop/user/dcausse/topic_model/wiki-region-groundtruth/regions-cirrus-upload.tsv.gz and convert it to classification.prediction.articlecountry weighted tags, this one would have to be adapted based on your source data
• wiki.lst: the list of wikis to filter on
• backfill.sh the shell script that orchestrates all this

Indeed, the new debian package wmf-opensearch-search-plugins version 1.3.20+12 has to be installed to run the lastest cirrus version. We generally maintain the cirrussearch-opensearch-image docket image that is used by MW developers and our cirrus integration test suite, but here I think that you install opensearch on the existing quibble image and thus refreshing this image with the new version of the plugin is indeed what should be needed.

In T409898#11378808, @Gehel wrote:

Do we need any specific plugins on this instance? At the moment, we're working on a minimal OpenSearch deployment, with no additional plugins, meant for the non-Search use cases.

This should be fixed, I can see the partial search response instead of the error.

@TJones the change should be live on hewiki and ruwiki, could you draft a message for the tech news possibly by adding some text to https://meta.wikimedia.org/wiki/Tech/News/2025/48?

In T410007#11377822, @doctaxon wrote:

In T410007#11377698, @dcausse wrote:

... there is now a component failing earlier than the allowed 50s.

How we can find out the component failing?

We received notifications from users that the search API which is configured to allow 50s timeouts to support costly search requests is now failing at 15s with an upstream request timeout (T410007). The user reported that the behavior started to change around nov 11th which is apparently when we started to roll out this new route on group2 wikis. I'm not 100% sure that this change is the cause of this new behavior but IIUC on all wikis except enwiki we now route api.php requests to the rest-gateway. If I'm not mistaken the rest-gateway has a default timeout of 15s which might explain this new behavior? Are there ways to vary this timeout based on the target action API?

Indeed, the internal timeout should be 50s to allow the regex to run. It is possible that something changed in the request flow that there is now a component failing earlier than the allowed 50s.

In T409469#11374597, @tchin wrote:

Would we also need to explicitly create the topics in main? Is auto topic creation enabled there?

If pushing to kafka-main you might need to increase broker's message.max.bytes see T344688.

In T409070#11350778, @EBernhardson wrote:

It might be the only reasonable way is to remove anchored trigram support from REL1_45

I think this is now fixed, the behavior of items and lexemes should be the same.
The API response looks like this now (on L7 when searching for L7):

{

Draft uploaded at: https://people.wikimedia.org/~dcausse/T404858-completion-default-sort-en-fr-he.html

In T393966#11325354, @RKemper wrote:

@dcausse In this updated version of the SLI we don't want to count throttled requests as either a success or failure, but rather exclude them entirely. However I'm having a bit of trouble understanding how all the pieces fit together.

Briefly, when a request comes in from a user and hits the throttling filter, does that avoid the request ever ultimately hitting blazegraph itself? In other words, we have the metrics of blazegraph_queries_done and blazegraph_queries_error which are scraped by the prometheus blazegraph exporter, and I want to know if blazegraph_queries_error implicitly contains the throttled requests in its count or not.

if it looks like request -> throttling filter -> (if not throttled) blazegraph then I think those are already excluded and therefore I don't have to do anything. But if the throttled request still makes it to blazegraph and blazegraph issues the 4xx at that point then they would be included. I think it's the former but figured you might be able to shed some light here.

In T401021#11334869, @achou wrote:

@dcausse, regarding the weighted search tag, IIUC, for a few mock records, the simplest way to produce events to CirrusSearch would be using kafkacat like P84613, right?

yes the event you crafted is perfect it's just missing the meta section, also please use event-gate to push them (otherwise you would bypass the various event platform validations):

curl -H"User-Agent: achou-T401021/wmf" -H"Content-Type: application/json" -XPOST https://eventgate-main.discovery.wmnet:4492/v1/events -d '[events]'

Event:

{
  "meta": {
    "stream": "mediawiki.cirrussearch.page_weighted_tags_change.v1",
    "domain": "test.wikipedia.org"
  },
  "dt": "2025-11-03T10:56:00Z",
  "wiki_id": "testwiki",
  "page": {
    "page_id": 1,
    "page_title": "SomePage",
    "namespace_id": 0,
    "is_redirect": false
  },
  "weighted_tags": {
    "set": {
      "recommendation.tone": [
        {
          "tag": "exists",
          "score": 1.0
        }
      ]
    }
  }
}

In T403775#11334379, @thiemowmde wrote:

An extra Advanced-Search ticket for something that's currently technically impossible is not useful.

If only touching wgEnableEventBus to add TYPE_EVENT is there a risk to start producing data to unrelated streams where downstream consumers might be a bit puzzled to see things from loginwiki? If yes perhaps constraining loginwiki to only produce to mediawiki.product_metrics.suggested_investigations_interaction should be preferred?

In T407514#11322446, @Gehel wrote:

In T407514#11322205, @dcausse wrote:

• exclude .DS_Store from this duplicate-finder plugin in the parent pom, but we'd have to re-add it for projects that override the default settings

Couldn't the child pom use combine.child as described in the doc to not need to re-declare? This might not be obvious to child projets, so it might need some documentation. (I haven't checked the actual use case, so I might be missing something).

In general, the approach we've tried to take was to make projects as easy to build as possible. You should be able to checkout a project and run ./mvnw clean verify and have it work. Needing to add specific -D arguments breaks that rule.

In T407514#11320956, @TJones wrote:

I often open the target folder to find the jar so I can deploy it locally. I also use the Finder with Java projects to recursively open folders to get all the way to the bottom quickly. (Anything that relies on people not opening folders is doomed to fail eventually.)

In T407514#11319096, @TJones wrote:

I believe we should find what is causing them to appear in the first place

That would be me opening folders. The MacOS Finder creates .DS_Store files to record information about the folder—how it is sorted, the position of icons, where the window is located, etc. https://en.wikipedia.org/wiki/.DS_Store#Purpose_and_location :

@TJones could you try to replicate this step by step from a fresh repository (cleanly fetched from gerrit and no target folders).
By default I think maven is supposed to ignore these folders when moving them to the target folders. I believe we should find what is causing them to appear in the first place because I'm afraid that fixing duplicate-finder-maven-plugin might be too late because it'd mean that these .DS_Store folders already have leaked into the build directories which could cause them to be published silently.
If on the other hand you found third-party jars (dependencies we use) with such folder in them this is a different story, we might possibly try to raise the issue to the lib owner and use the workaround you suggested to make our build pass (but haven't hit this problem myself so hopefully it's not the case).

In T405475#11317979, @Lydia_Pintscher wrote:

but this is perhaps good enough for now

Yeah I think so. And maybe we can have a ticket for adapting the EntityIdSearchHelper for Lexemes? (I'd do it but I don't understand enough of the details to create a meaningful ticket.)

I believe the issue is that we generally prefer EntityIdSearchHelper when matching lexeme IDs, this search helper does not have any customization for lexemes. A simple approach is to prefer the Cirrus version of the hits which contains the expected set of metadata, this might not fully solve the issue in cases where the Lexeme ID is searched rapidly after being created (searched before the search engine is updated) but this is perhaps good enough for now?

In T406920#11307950, @RKemper wrote:

The only thing I'm not sure about is one of your tests was this: for endpoint in $(cat wdqs_hosts.lst | grep ^w); do echo -n "$endpoint: "; curl -s -XPOST --data-urlencode [email protected] http://$endpoint/bigdata/namespace/categories/sparql?format=json | jq '.results.bindings[] | .C.value'; done but none of these have a .C. Not sure if that is intended or not.

Sorry I did not paste this query which is:

SELECT (COUNT(*) AS ?C) WHERE {
	<https://commons.wikimedia.org/wiki/Category:Current_De_Havilland_Canada_aircraft_of_Universal_Air> ?p <https://commons.wikimedia.org/wiki/Category:Universal_Air_current_fleet> .
}

I tested this it and all nodes seem to have the missing link.
I would consider this ticket done.

We should wait for https://gerrit.wikimedia.org/r/c/search/extra/+/1198569 I think

I think the problem is in the extra plugin, I could reproduce it when the weighted_tags is current null in opensearch with the following bulk sequence:

{"index": {"_index": "my_database_content", "_id": "10000"}}
{}
{"update": {"_index": "my_database_content", "_id": "10000"}}
{"script":{"source":"super_detect_noop","lang":"super_detect_noop","params":{"handlers":{"weighted_tags":"multilist","version":"documentVersion"},"source":{"version":1,"weighted_tags":["mytag/__DELETE_GROUPING__","myothertag/somedata|2"]}}},"upsert":{"version":1}}

CirrusSearch may provide such sorting options via T40403. I believe that this ticket refers to the UI work required to happen when this capability is available from CirrusSearch, given that there is no such selector in Special:Search I have the impression that the intent of this ticket was to adapt Advanced-Search, @thiemowmde am I missing something?

Opened https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/12 that should have all this.

Looking at the lag data of the category graph it seems to me that around june 12 2015 we started to have more lag issues:

Actually the graph seems outdated on all 4 endpoints I tested.
The query:

SELECT ?out WHERE {
      SERVICE mediawiki:categoryTree {
          bd:serviceParam mediawiki:start <https://commons.wikimedia.org/wiki/Category:Aircraft_of_Universal_Air> .
          bd:serviceParam mediawiki:direction "Reverse" .
          bd:serviceParam mediawiki:depth 5 .
      }
} ORDER BY ASC(?depth)
LIMIT 200

Should return more than two items:

for endpoint in wdqs-internal-main.svc.eqiad.wmnet wdqs-main.svc.eqiad.wmnet wdqs-internal-main.svc.codfw.wmnet wdqs-main.svc.codfw.wmnet; do
  echo -n "$endpoint: ";
  curl -s -XPOST --data-urlencode [email protected] https://$endpoint/bigdata/namespace/categories/sparql?format=json | jq '.results.bindings | length';
done

wdqs-internal-main.svc.eqiad.wmnet: 2
wdqs-main.svc.eqiad.wmnet: 2
wdqs-internal-main.svc.codfw.wmnet: 2
wdqs-main.svc.codfw.wmnet: 2