Wikidata Query Service/Migration/Rewrite of MWAPI

MWAPI is one aspect of search on Blazegraph. Another is Blazegraph’s search (bds:) predicates, which exist in the Wikidata fork.

But, this page is focused on MWAPI. Why? bds:search requires a full-text (Lucene) index to be built over all the literals. That is not done for Wikidata. Without the index, the predicates (bds:search and its modifying predicates, bds:matchAllTerms, bds:relevance, bds:min/maxRelevance, bds:rank and bds:min/maxRank) either return nothing or error out. So, instead of bds:search, there is MWAPI.

SERVICE wikibase:mwapi { ... } is a Blazegraph-specific extension to WDQS that calls the MediaWiki Action API and binds its results to SPARQL variables. The service is not part of the SPARQL 1.1 standard since it is not possible to call a REST API within a query.

MWAPI’s bridges the MediaWiki Action API into SPARQL - adding full-text search via CirrusSearch (backed by an Elasticsearch/OpenSearch cluster) over any Wikimedia project including Wikidata, label/alias lookup via wbsearchentities, and structural queries (categories, backlinks, transclusions) via generators.

The following table briefly highlights the different searches possible using the MWAPI service on WDQS and the corresponding “actions” if using the MWAPI directly^[1]:

Service	MWAPI Action	Inputs	Outputs	Description
Generator	action=query& generator=<name>	generator, prop, pprop	title, item, pageid, lastrevid, timestamp	Generators retrieve a base set of pages or items. The property modules fetch specific data for the retrieved items.
Categories	action=query& prop=categories	titles, cllimit	category, title	Lists the categories that a given page belongs to.
Search	action=query& list=search	srsearch, srwhat, srlimit	title	Provides full-text CirrusSearch on the target wiki.
EntitySearch	action= wbsearchentities	search, language, type, limit	item, label	Performs Wikidata entity label and alias search.

Overall, this is a huge topic and MWAPI’s functionality cannot be replicated using standard SPARQL 1.1 syntax. This is because the behavior depends on relevance ranking, case and accent folding, and reliance on data outside Wikidata.

The scenarios where rewrites are theoretically possible (although not equivalent) are:

EntitySearch/wbsearchentities requests
Search/CirrusSearch addressed to www.wikidata.org

These specific scenarios are briefly discussed in later portions of this document. But, there is no way to achieve full equivalency.

Note that the only reliable and efficient rewrite is to separately call the MediaWiki Action API and then feed the results into a subsequent SPARQL query (which is discussed in the next section).

Using the MediaWiki Action API

This section describes the "two-step pattern" (Action API -> SPARQL query) referenced above. It performs the search/generator step using the MediaWiki endpoint (https://www.wikipedia.org/w/api.php), collects the Wikidata entities (QIDs) and then uses these in a SPARQL 1.1 query with a VALUES block.

The pattern can be accomplished using `curl` plus `jq`^[2] via the command line, or using a programming language such as Python. The Python code is also discussed below, and included in MWAPI.ipynb, since it can implement robust pagination (continuation processing) for bulk data handling. Also, setting the properties of the API call is easier than manipulating the command line.

Requirements and Constraints of Using the API

Mandatory (defined by Wikimedia policy):
- Set a `User-Agent` with a contact URL/email
  - Missing or generic UAs will immediately fail with an authorization (403) error
Correctness (as specified by the API contract):
- Use format=json&formatversion=2
- Never assume that a request is complete
  - MWAPI may return a continue element in the JSON response
  - Clients must append a sub-element of “continue” as a parameter to subsequent requests, to fetch the next batch of data
Per-call limits for MWAPI with WDQS:
- wbsearchentities: 7 (default for web clients and regular users); 50 (API and bot users); Max 50
- Search: 10 (default); Max 500 (standard users); Max 5,000 (bots); Hard-coded max 10000 rows (with use of continuation and `sroffset`)
Robustness (client-side):
- Always check for an `error` key before using the query results
Etiquette/Being a good-citizen (recommended for bulk processing):
- Timeout handling
- `maxlag=5` + backoff on a `maxlag` error. See API Etiquette for more information.

Use of curl and jq

The following should be sufficient for a one-off or a small result set. This approach has no dependencies beyond `curl` and `jq`:

1. Set the user-agent. For example, UA='mwapi-substitute/0.5 (User: awesterinen-ctr@wikimedia.org). For more information, see user-agent policy and API usage guidelines.

2. Use the Action API to search Wikidata for specific entities (QIDs), then use jq to parse the returned JSON and extract them to a file for for persistence and ongoing use in WDQS calls.

For example, to get all QIDs whose “en”/”mul” labels or aliases include “Einstein”, execute the following command:

curl -s -G 'https://www.wikidata.org/w/api.php' -H "User-Agent: $UA" --data-urlencode 'action=wbsearchentities' --data-urlencode 'search=Einstein' --data-urlencode 'language=en' --data-urlencode 'type=item' --data-urlencode 'format=json' --data-urlencode 'formatversion=2' | jq -r '.search[].id' > qids.txt

The service returns 7 results^[3]: Q16834800, Q937, Q901448, Q1309274, Q26854017, Q1892, Q28739649
Note that the full JSON response to the wbsearchentities request contains much more information than just the QIDs:

3. Build a VALUES string from the QIDs, to use in the SPARQL query.

For example, VALUES=$(paste -sd' ' <(sed 's/^/wd:/' qids.txt)) results in the string, “wd:Q16834800 wd:Q937 wd:Q901448 wd:Q1309274 wd:Q26854017 wd:Q1892 wd:Q28739649”.

4. Include the results in a request to WDQS. For example:

curl -s -G 'https://query.wikidata.org/sparql' -H "User-Agent: $UA" -H 'Accept: application/sparql-results+json' --data-urlencode "query=PREFIX wd: <http://www.wikidata.org/entity/> PREFIX rdfs:<http://www.w3.org/2000/01/rdf-schema#> SELECT ?item ?label WHERE { VALUES ?item { $VALUES } ?item rdfs:label ?label . FILTER(LANG(?label)='en') }"

Or, the same example above, as input to the WDQS UI:

SELECT ?item ?label WHERE {
   VALUES ?item { wd:Q16834800 wd:Q937 wd:Q901448 wd:Q1309274
                  wd:Q26854017 wd:Q1892 wd:Q28739649 }
   ?item rdfs:label ?label. FILTER(LANG(?label) = "en")
}

The examples above return the following results:

Using the command line is awkward exactly where Python is useful. Python allows for handling continuations, errors or mandatory lags, or chunking a large number of VALUES.

Use of Python for Pagination, Chunking and Error Handling

The code in the Jupyter notebook, MWAPI.ipynb, implements the pattern from above, and adds support for larger limits, continuation, CirrusSearch and generator processing.

There are examples of using MWAPI for Wikidata label/alias search (similar to what is shown for `curl` above), CirrusSearch and several generators. Note that the generators used in the notebook return Wikipedia articles. The Python code converts those page references to Wikidata QIDs as part of the generator-to-qids function.

After obtaining the QIDs, these can then be fed into SPARQL 1.1 queries (using the VALUES keyword). The code to perform the queries is also included in the second to last cell of the notebook (as a query with a very minimal result set - basically, returning the “instance of” entities for the QID, and the labels of the QID and the “instance of” entity). Also, a Python-only “pretty print” routine is shown in the notebook, in the last cell.

An example of the output for the wbsearchentities results is shown below:

item	type	label	typeLabel
Q136164891	Q3331189	Einstein	version, edition or translation
Q11452	Q214070	general relativity	physical law
Q59151	Q173227	cosmological constant	physical constant
Q138088407	Q1542966	Einstein-Gymnasium Neuenhagen	gymnasium
Q135050506	Q7889	Einstein's Cats	video game
Q17712	Q131647	Albert Einstein Medal	medallion
Q35875	Q24034552	mass–energy equivalence	mathematical concept
Q131938472	Q11707	Einstein - Aarau	restaurant
Q131938473	Q11707	Einstein Bistro - St. Gallen	restaurant
Q35875	Q33104303	mass–energy equivalence	concept in physics
Q1892	Q11344	einsteinium	chemical element

Qualified Query Rewrites for MWAPI

The section above explained how to generically call MWAPI and transform/use the results. However, there are some specific MWAPI queries that appear to be possible to transform to SPARQL 1.1 compliant rewrites. These are discussed below, along with explaining their lack of equivalency.

EntitySearch/wbsearchentities

The MWAPI service's EntitySearch template (a wrapper for the wbsearchentities Action API) performs a start-anchored, language-aware, relevance match over Wikidata items’ labels and aliases. It returns the top-ranked entities whose label or alias begins with the specified search string, in the selected and fallback languages.

A similar intent can be expressed in plain SPARQL 1.1 by adding the following to a query that binds the ?item variable. For example:

?item rdfs:label|skos:altLabel ?match .
FILTER(LANG(?match) IN ("en", "mul"))
FILTER(STRSTARTS(LCASE(STR(?match)), "some_text"))

Unfortunately, there is no string index behind STRSTARTS nor is there any support for accent folding^[4] or relevance scoring. The SPARQL engine evaluates the specified filter on every label or alias for the items that are bound in the query. If the query sufficiently constrains ?item (such as a structural constraint like ?item wdt:P31 wd:<TYPE> where there are less than 2M possible items), the query may execute within the timeout period. However, if there are no bounds or their evaluation results in many millions of possible ?items, the query will certainly timeout.

As an example, here is a MWAPI EntitySearch query (and its results) for entities with labels/aliases starting with “saint”. Note that there is no criteria restricting the kinds of entities returned.

  SELECT ?item ?label WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api "EntitySearch" ;
                    wikibase:endpoint "www.wikidata.org" ;
                    mwapi:search "saint" ;
                    mwapi:language "en" .
    ?item  wikibase:apiOutputItem mwapi:item .
    ?label wikibase:apiOutput  mwapi:label .
  }
}

The following top-ranked^[5] results are reported:

Executing an equivalent query using SPARQL 1.1 would certainly time out since all Wikidata entities’ labels/aliases would be FILTERed.

Further complicating matters, another difference in the results occurs when different spellings, accents and diacritics must be taken into account. As noted earlier, EntitySearch/wbsearchentities results are accent folded. This means that the string “Curie” matches “Curié”, and the string “Lodz” matches “Łódź”. To do this sort of filtering with SPARQL would require rewriting the filter as:

?item rdfs:label|skos:altLabel ?match .
FILTER(LANG(?match) IN ("en", "mul"))
BIND(LCASE(?match) AS ?lc)
FILTER(STRSTARTS(LCASE(STR(?lc)), "some_text") || 
       STRSTARTS(LCASE(STR(?lc)), "some_alternate_version_of_text") || 
       … )

The above would be difficult to correctly and fully state.

In this case, the only resort is to separately call the MWAPI, and then use the results in a separate query.

CirrusSearch of Wikidata

Every reason to discourage rewriting EntitySearch/wbsearchentities applies to Search/CirrusSearch as well, plus several additional reasons exist that are unique to the broader feature set which CirrusSearch exposes.

The shared reasons are:

No case folding in SPARQL → Need to FILTER LCASE
No accent folding → Need to FILTER on multiple patterns, or use some REPLACE/REGEX patterns
No language fallback → Need explicit LANG IN ("en", "mul", …) statements
No relevance ranking → SPARQL has no concept of “top-ranked” and so can return large result (or intermediate result) sets
Unindexed-scan ceiling → As above, can easily time out for queries that have no bounding/constraining criteria

Additional reasons for lack of equivalency in SPARQL 1.1 are:

Tokenization includes word-boundary semantics → For example, inlabel: theater matches "Dream Theater"; This could be accomplished using SPARQL’s CONTAINS keyword but that could return unwanted results
- For example, consider “theatergoer”: MWAPI would not match this string since it would not match a word boundary, but SPARQL CONTAINS would indicate a match
Stemming → For example, “theory” matches "theorical" in CirrusSearch whereas SPARQL CONTAINS would not
morelike:QID → Finds items considered “similar” to the specified one; There is no equivalent SPARQL rendering
Fuzzy matching (via a trailing ~) → Supports Damerau-Levenshtein edit distance which has no corollary in SPARQL
Regex with stem/case awareness → CirrusSearch's inlabel:/.../ does full-string matching, (it requires the pattern to consume the entire field value), whereas SPARQL REGEX does substring matching (the pattern just has to appear somewhere in the literal)
- For example, consider the labels, “Theater”, “Dream Theater” and “Theatergoer”
- CirrusSearch’s regex might search for inlabel:/theater/i, and would only match “Theater”
- SPARQL's REGEX equivalent must be written as FILTER(REGEX(?label, "^theater$", "i")) to force full-string matching
Text search across labels, aliases, descriptions, and the labels of statement values in one pass → Can be reproduced in SPARQL with explicit UNION over rdfs:label, skos:altLabel, schema:description, and joined statement-value labels

There is a limited case where rewriting to SPARQL 1.1 is possible - when the CirrusSearch query (against the Wikidata endpoint) uses the haswbstatement structural keyword with (optionally) haslabel: or hasdescription:.

For example, consider the case of searching for female astronauts:

SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api "Search" ;
             wikibase:endpoint "www.wikidata.org" ;
             mwapi:srsearch "haswbstatement:P106=Q11631 haswbstatement:P21=Q6581072" ;                                
             mwapi:srlimit "max" .
   ?bare_qid wikibase:apiOutput mwapi:title .   
  }
  BIND(IRI(CONCAT(STR(wd:), ?bare_qid)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Before looking at the results, it is important to examine the query. Executing MWAPI “Search” against the Wikidata endpoint returns a single result - mwapi:title. But, it only returns that result when specifically requested as the apiOutput - as is done in line 7. The “title” that is returned is a Wikidata QID - but not an IRI, just the string, “Q###”. To use this result in subsequent triples in the query, it must be converted to an IRI. That is accomplished in line 9.

The above query returns results as shown below:

To execute the equivalent SPARQL 1.1 query, it must be expressed as:

SELECT * WHERE { 
   ?item p:P106/ps:P106 wd:Q11631 ;
             p:P21/ps:P21 wd:Q6581072 .                                
   OPTIONAL { ?item rdfs:label ?label . FILTER(LANG(?label) = "en") }                  
   OPTIONAL { ?item rdfs:label ?label . FILTER(LANG(?label) = "mul") }
}

Why? Because haswbstatement is rank-blind. It can return deprecated statements. These must be accounted for in the rewritten SPARQL by examining the qualified properties/statements.

It is valuable to note that the SPARQL query executes in less time (on both Blazegraph and QLever) than the corresponding MWAPI request.

As another example, consider the example of searching for female astronauts that also have a label in German. Using the MWAPI, the query is:

SELECT ?item ?itemLabel WHERE {
 SERVICE wikibase:mwapi {
   bd:serviceParam wikibase:api "Search" ;
            wikibase:endpoint "www.wikidata.org" ;
            mwapi:srsearch "haswbstatement:P106=Q11631 haswbstatement:P21=Q6581072 
                            haslabel:de" ;                                
            mwapi:srlimit "max" .
   ?bare_qid wikibase:apiOutput mwapi:title .   
 }
 BIND(IRI(CONCAT(STR(wd:), ?bare_qid)) AS ?item)
 SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

125 results are returned.

The equivalent (and faster) SPARQL 1.1 query is:

SELECT ?item ?itemLabel WHERE { 
   ?item p:P106/ps:P106 wd:Q11631 ;
             p:P21/ps:P21 wd:Q6581072 .       
   ?item rdfs:label ?germanLabel . FILTER(LANG(?germanLabel) = "de") 
   OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?label) = "en") }                  
   OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?label) = "mul") }
}

Query Rewrite Pattern

The only cleanly rewritable MWAPI queries use CirrusSearch of Wikidata and the keywords: haswbstatement:, haslabel:, and hasdescription:. They can include boolean composition operators (AND/OR/-) and use the syntax, [property_PID=entity_QID], for searching qualified statements. The rewrite is being reassessed as to its value given the restrictions and lack of equivalency.

The rewrite pattern follows this algorithm:

Parse the CirrusSearch query into clauses by:
- Splitting by parentheses
- Examining the contents of each:
  - Recurse if there are multiple levels of parentheses to reach the inner clauses
  - Split on the top-level boolean operators: whitespace AND and OR
  - For each leaf:
    - Strip a leading - (negation) if present (it doesn't change whether the search is rewritable or not)
    - Make sure that the leaf starts with one of: haswbstatement:, haslabel:, or hasdescription:
      - If not, then the query cannot be rewritten and requires an external call to MWAPI and subsequent use of the results in a SPARQL query
Apply per-clause rewrite rules:
- haswbstatement:P##=Q## → ?item p:P##/ps:P## wd:Q## .
- hswbstatement:P##=Q##[P**=Q**] → ?item p:P## ?statement. ?statement ps:P## wd:Q## ; pq:P** wd:Q** .
- haslabel:LANG → FILTER EXISTS { ?item rdfs:label ?l . FILTER(LANG(?l)="LANG") }
- hasdescription:LANG → FILTER EXISTS { ?item schema:description ?d . FILTER(LANG(?d)="LANG") }
- - (negation) → FILTER NOT EXISTS { body_rewritten_using_rules }
Recreate the boolean structure
- AND → Multiple triple patterns / filters in the same WHERE block
- OR → Wrap the alternatives in { … } UNION { … } and add DISTINCT to the outer SELECT to dedupe
- Grouping ( … ) → Preserved by { … } blocks around the rewritten body
Add LIMIT 500 to approximate the Search “standard user” cap

Note that even this rewrite will not be equivalent since CirrusSearch returns the results ranked by relevance, and SPARQL returns the entire set of results.

Examining a specific example, consider the CirrusSearch query for female astronauts without a French language description:

SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api "Search" ;
            wikibase:endpoint "www.wikidata.org" ;
            mwapi:srsearch 
               "haswbstatement:P106=Q11631                                              
                haswbstatement:P21=Q6581072 -hasdescription:fr" ;
            mwapi:srlimit "max" .
    ?title wikibase:apiOutput mwapi:title .
  }
  BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

This produces 52 results. A portion of the output is shown:

The rewritten query is:

SELECT DISTINCT ?item ?itemLabel WHERE {
    ?item p:P106/ps:P106 wd:Q11631 ;       # rank-blind: astronaut
              p:P21 /ps:P21  wd:Q6581072 .     # rank-blind: female
  FILTER NOT EXISTS {                            # negated hasdescription:fr
     ?item schema:description ?d . FILTER(LANG(?d) = "fr") }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = "en") }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = "mul") }
} LIMIT 500

Which also produces 52 results.

Alternately, consider the query for people who received a PhD from MIT:

SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api "Search" ;
                    wikibase:endpoint "www.wikidata.org" ;
                    mwapi:srsearch "haswbstatement:P69=Q49108[P512=Q849697]" ;
                    mwapi:srlimit "max" .
    ?title wikibase:apiOutput mwapi:title .
  }
  BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Which returns 67 results.

This is translated to the SPARQL query (which also produces 67 results):

SELECT DISTINCT ?item ?itemLabel WHERE {
  ?item p:P69 ?s . ?s ps:P69 wd:Q49108 ; pq:P512 wd:Q849697 .
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = "en") }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = "mul") }
} LIMIT 500

Footnotes

↑ From https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Supported_services
↑ jq is a lightweight, command-line tool to process, filter and extract data from JSON. See https://jqlang.org/.
↑ 7 is the default number returned when a limit is not specified.
↑ A text preprocessing technique that strips accent marks and diacritics
↑ ‘top-ranked’ is determined by sitelink count, statement count, matching the label versus the alias, matching the specified language versus the fallback, etc. This ordering cannot be replicated in SPARQL. Also, note that 50 is the limit for the number of items returned by EntitySearch/wbsearchentities.

[1] From https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Supported_services

[2] q is a lightweight, command-line tool to process, filter and extract data from JSON. See https://jqlang.org/.

[3] 7 is the default number returned when a limit is not specified.

[4] A text preprocessing technique that strips accent marks and diacritics

[5] ‘top-ranked’ is determined by sitelink count, statement count, matching the label versus the alias, matching the specified language versus the fallback, etc. This ordering cannot be replicated in SPARQL. Also, note that 50 is the limit for the number of items returned by EntitySearch/wbsearchentities.

[1]