WDQS scaling issues meetings in February

[dpriskorn]

Context

https://lists.wikimedia.org/hyperkitty/list/[email protected]/thread/KPA3CTQG2HCJO55EFZVNINGVFQJAHT4W/

Question

Is it a good idea to participate and deliver our perspective?

[Daniel-Mietchen]

@dpriskorn Probably yes.

For the record, this relates to

• Assess implications of Blazegraph failure playbook for Scholia #1721
  and we are having a hackathon next week, as per
• Scholia hackathon on 14 and 16 February 2022 #1807

[fnielsen]

1. WDQS scaling community meeting 1/2: SPARQL query features - Thursday, February 17 · 18:00 UTC
2. WDQS scaling community meeting 2/2: RDF store backend needs - Monday, February 21 · 18:00 UTC

[Daniel-Mietchen]

See also https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Feb_2022_scaling_community_meetings

[dpriskorn]

I created the request for bot flag for So9qBot earlier and met resistance presumably because of fear of breaking/overloading blazegraph and thus render WDQS unusable for everyone.

Since I helped raise the issue in the Wikidata Telegram channel and with the product manager, a lot has happened.

WMF started analyzing the issue in depth and now we have a disaster playbook. 🥳

WMF also very recently began the process of evaluating alternatives to blazegraph and now we have a rough timeline of 2-3 years until the problem is completely solved, assuming a competent team of engineers are dedicated to the task and funded appropriately.

I intend to finish the original idea of asseeibot soon so that it can add new items for each DOI found in Wikipedia, which we are currently missing in Wikidata.

This bot, if approved by the community, will increase the number of scientific items by 10-15% at a pace the community can control.
If all direct references to the DOIs found are also imported the number of items will probably double to 80M over time.

I also plan to create an importer for refcat from IA:

This first release of the Refcat dataset contains over 1.3 billion citations extracted from over 60 million metadata records and over 120 million scholarly artifacts [...]

This will bring our collection of papers in Wikidata up to a level around 120M in total. 🎉

Nobody knows if these two imports will push blazegraph past the breaking point, but that is not a big issue IMO since only 1% of all queries in WDQS are affected and since the data just existing and being curated and editable by any scientist in the world will probably be an epic game changer. I predict that a majority of scientists are gonna want to be well represented in our graph within 2-5 years.

If WDQS breaks, the playbook is enacted and we can continue working, but without SPARQL support from WMF during an interim period. This will of course affect Scholia a lot. We would have to set up our own sparql endpoint with the data somewhere. My hope is that we will succeed in getting a working endpoint up in less that 2 weeks with the help of IA and others in the Scholia and WikiCite communities.

The upside of this for Scholia is that all the new papers and citations would take us to a level of completeness that is on par with closed commercial databases currently used by scientists,but which completely lacks both the openness and empowerment of Wikidata and the graph powered features for finding author networks and new papers efficiently.

Since it takes roughly 3 months in average according to a study I read for a scientific article to be included in Wikipedia my bot alone would keep us only 3 months behind the bleeding edge of science publications.

We would have to find another approach to get closer to real time import of scientific papers as they are published.

[Daniel-Mietchen]

The notes from the first meeting on Thursday Feb 17 about SPARQL query features sit at https://etherpad.wikimedia.org/p/R5n382Ld0Vvykc7Ak3iH .

I could not attend but @fnielsen and @dpriskorn did, and I went through the notes later on, particularly adding examples of Scholia queries that time out:

• https://scholia.toolforge.org/author/Q9047#academic-tree (usually works, but at its limits)
• https://scholia.toolforge.org/author/Q46826387#publications-per-year
• https://scholia.toolforge.org/author/Q46826387#coauthors
• https://scholia.toolforge.org/author/Q55004959#list-of-publications
• https://scholia.toolforge.org/author/Q55004959#publications-per-year
• https://scholia.toolforge.org/author/Q46826387/missing#missing-coauthor-items
• https://scholia.toolforge.org/authors/Q53405108,Q53382329,Q57956212,Q42225039,Q57956362,Q42666666,Q53407393#co-author
• https://scholia.toolforge.org/country/Q35
• https://scholia.toolforge.org/country/Q668
• https://scholia.toolforge.org/location/Q583725#nearby-locations-as-topics-in-works
• https://scholia.toolforge.org/location/Q84/topic/Q41112
• https://scholia.toolforge.org/organization/Q13371
• https://scholia.toolforge.org/organization/Q13371##employees-and-affiliated
• https://scholia.toolforge.org/organization/Q13371#most-cited-papers
• https://scholia.toolforge.org/organizations/Q1269766,Q193196#citations-per-year
• https://scholia.toolforge.org/organizations/Q1269766,Q193196#citations-to-works
• https://scholia.toolforge.org/publisher/Q746413#citations-works
• https://scholia.toolforge.org/sponsor/Q60346#project-budgets
• https://scholia.toolforge.org/topic/Q15#publications-per-year
• https://scholia.toolforge.org/topic/Q41719#author-scores
• https://scholia.toolforge.org/topic/Q41719#coauthor-graph
• https://scholia.toolforge.org/topic/Q41719#co-occurring
• https://scholia.toolforge.org/topic/Q41719#co-occurring-map
• https://scholia.toolforge.org/topic/Q41719#publications-per-year
• https://scholia.toolforge.org/topic/Q180556#most-cited-authors
• https://scholia.toolforge.org/topic/Q180556#top-cited
• https://scholia.toolforge.org/topic/Q1492760#author-scores
• https://scholia.toolforge.org/topic/Q1492760#country-citation-graph
• https://scholia.toolforge.org/topic/Q1492760#publications-per-year
• https://scholia.toolforge.org/topic/Q18123741#publications-per-year
• https://scholia.toolforge.org/topic/Q181600/missing#missing-author-items
• https://scholia.toolforge.org/topic/Q1492760/missing#missing-author-items
• https://scholia.toolforge.org/venue/Q26150#recently-published-works
• https://scholia.toolforge.org/venue/Q26535#recently-published-works
• https://scholia.toolforge.org/venue/Q939416#recently-published-works
• https://scholia.toolforge.org/venue/Q564954#articles-by-intention
• https://scholia.toolforge.org/venues/Q180445,Q192864,Q1146531 (all three queries)
• https://scholia.toolforge.org/P577 (first two queries)

The next meeting on RDF store backend needs is today at 18:00 UTC, and I'll try to be there.

[Daniel-Mietchen]

The WDQS scaling call just ended, and I found it useful. Notes in https://etherpad.wikimedia.org/p/yPUhyhbmXglC_Magx0Go .

[fnielsen]

Blogpost: "What SPARQL keywords do we use in Scholia?"

[Daniel-Mietchen]

Official summary: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS-scaling-update-feb-2022