Out of memory on the qlever public endpoint

[WolfgangFahl]

I'd like to optimize the query of properties of humans but can't run the non optimized one due to an out of memory condition:

https://qlever.cs.uni-freiburg.de/wikidata/tnooBa

[hannahbast]

This is exactly the kind of query for which QLever has the special ql:has-predicate predicate: https://qlever.cs.uni-freiburg.de/wikidata/xgizMg . It's also the kind of query used by the QLever UI for the suggestions of predicates.

Of course, a query like the one you posted should automatically be executed using ql:has-predicate internally (without the user having to specify or even know about this). This is on our TODO list and we can do it, once our new SPARQL parser is finished.

PS: Note that in the ql:has-predicate query, the DISTINCT is missing. With ql:has-predicate, the DISTINCT is implicit, which you can check for yourself if you write it explicitly. However, for some strange reason (which will hopefully be fixed soon), the query currently runs much longer with the explicit DISTINCT (it should be the other way round, since the query is harder without the DISTINCT).

@joka921 @Qup42

[WolfgangFahl]

@hannahbast - very helpful hint i have created WolfgangFahl/pyLoDStorage#98 to address this.

[WolfgangFahl]

http://wikidata.bitplan.com/
with

now works for e.g. human:

for scholarl article i still get the memory error

[joka921]

Can you please send us the concrete query that you send to QLever which makes you trouble.
I cannot infer the relevant information from your screenshot which seems to be about the (already fixed) case for humans.

I just tried this query for scholarly articles, which worked. It uses the exact same pattern as the "humans" query, but replaces wd:Q5 with wd:Q13442814. Is this the correct entity for scholarly articles, or which one did you use?

[WolfgangFahl]

I dont' think the property query is the problem but the first one asking for the instance of counts having count>1

QueryBadFormed: a bad request has been sent to the endpoint, probably the sparql query is bad formed. Response: b'{\n "exception": "Tried to allocate 14029MB, but only 11549MB were available. Clear the cache or allow more memory for QLever during startup",\n "query": "# get the most frequently used properties for\\n# Q13442814:scholarly article\\nPREFIX rdf: \\nPREFIX rdfs: \\nPREFIX schema: \\nPREFIX wd: \\nPREFIX wdt: \\nPREFIX wikibase: \\nPREFIX xsd: \\n\\nSELECT ?prop ?propLabel ?count WHERE {\\n {\\n SELECT ?p (COUNT(DISTINCT ?item) AS ?count) WHERE {\\n ?item wdt:P31 wd:Q13442814. \\n ?item ql:has-predicate ?p \\n } GROUP BY ?p\\n }\\n ?prop wikibase:directClaim ?p.\\n ?prop rdfs:label ?propLabel.\\n FILTER(LANG(?propLabel) = \\"en\\").\\n FILTER(?count >1519319). \\n}\\nORDER BY DESC (?count)\\n",\n "resultsize": 0,\n "status": "ERROR",\n "time": {\n "computeResult": 6956,\n "total": 6956\n }\n}'

[joka921]

Can you please always send the queries that caused your problems in a well-formatted way. It is often really time consuming for us to extract this information from your messages.

[joka921]

The following query works:
TryIt

The point (which should be documented, and we should get the other case to work...) is:
When using ql:has-predicate you must write COUNT(?item) instead of COUNT(DISTINCT ?item). I agree that this is somewhat unintuitive, as the DISTINCT is in fact part of the semantics of ql:has-predicate. We should get the "ordinary" syntax that triggers the same implementation back to work.

[WolfgangFahl]

@joka921 thanks for the reply

https://qlever.cs.uni-freiburg.de/wikidata/Y5WES4 also has the memory issue ...

[joka921]

For this specific query, I currently use the following workaround (with some caveats, see below):
TryIt.

The problem here is, that the fully materialized wdt:P31/wdt:P279 is too large for QLever, and that QLever currently doesn't support "direct/lazy aggregates" or keeping intermediate results in RAM (both are on our roadmap).

The workaround will only yield the correct results as long as the first predicate (wdt:P31) in your case is functional (each entity has at most one type). But in this case it should be correct. However I have found the result of this query to be not that interested, as super generic types ("entity" or "object") are highly overrepresented:)

[WolfgangFahl]

The query is very helpful but only shows 100 results. If i extend the limit the behavior of the UI is strange:
https://qlever.cs.uni-freiburg.de/wikidata/iuP6lt
First it shows:

and when you click on the button:

[hannahbast]

This is indeed misleading, though easily explained: Without the LIMIT, the result would have 96,986 rows, which QLever has indeed computed in its internal representation. Due to the LIMIT, the full result has only 500 rows.

The UI should realize this and change the button label accordingly. I will try to fix this.

BTW: This issue is still called "Out of memory ...". Is this resolved? It's misleading when a discussion has a completely unrelated title.

[hannahbast]

Fixed by QLever UI commit 0b7135

[WolfgangFahl]

@hannahbast
BTW: This issue is still called "Out of memory ...". Is this resolved? It's misleading when a discussion has a completely unrelated title.
Indeed - i opened a specific issue for the out of memory issue at #730 .