Speed Up Dataset List Request

[fm3]

• Slightly speeds up the dataset list request by moving the filter logic to composed SQL.
• An optional boolean parameter compact can be used to get a more compact response with less information. This speeds up the route significantly more, as the complete response is constructed with just one SQL query.
• the optional isEditable GET parameter in the dataset list request has been removed (it was, to our best knowledge, unused)
• The front-end does not yet use this new compact version, but this is planned, cc @philippotto
• Note that the work by @normanrz in the now superseded [Reference] Dataset list query #6766 paved the way for this PR :)
• Also slipped in here: bugfix for folder access query (make visible folders that include public datasets)
• Also slipped in here: bugfix for annotation list total count header

URL of deployed dev instance (used for testing):

• https://fastdatasetlist.webknossos.xyz

Steps to test:

• query http://localhost:9000/api/datasets?compact=true
• should be fast (compared to no compact=true)
• should still work with filters like isActive, searchQuery, folderId, … (except isEditable, that one has been removed)
• isEditable boolean field should be correct (that one needs most scrutiny I’d say)

Issues:

• fixes speed up dataset list request #6089
• fixes Dataset Folder Tab: Public datasets are not shown if user has otherwise no access #6773

---

• Updated (unreleased) changelog
• Updated documentation if applicable
• Ready for review

[normanrz]

Very good stuff. Looking forward to seeing the perf improvements!

I am wondering if a complex query like this (or parts of it) might be better placed in a view for reusability and debuggability.

[normanrz]

We may need to revise that in the future. There needs to be a way to revoke access to a dataset even for the person that happens to have uploaded the data.

[fm3]

If you agree we can just remove this from the relevant places now. In fact, to be allowed to upload (to a folder), you need to be admin/dataset manager or folder team manager already. So this change will not affect many (and there it may actually be intended)

[fm3]

I created #6780 – I decided not to do it right away because I need to double-check that this does not break the uploading process itself

[normanrz]

Btw this could also be implemented as a view: a view of all dataset+user combinations with an attribute whether the user can read/write the dataset. The benefit of a view would be that we can easily examine it with pgadmin et al and it could potentially be used by multiple queries.

[fm3]

I agree about the views. However, we need to figure out a way to handle them during migrations. The existing userInfos view is already frequently a source of trouble (needs to be dropped + recreated in evolutions). I would not like to have to do this for two more views all the time 😬 Maybe there is a way to get around this.

[normanrz]

What's the problem with dropping/recreating?

[fm3]

Not a conceptual problem, but time and again we wrote evolutions with only their semantics in mind, then saw them fail because the view handling was not yet included. It of course makes the evolutions a lot bigger as well. None of this totally blocks this, but I would only go for it if we see concrete benefit from creating more views

[fm3]

Another question: I spoke to @jstriebel and they said the current webknossos-libs don’t use any of the fields that "compact" removes. So in theory, we could just breaking-change the list query once the front-end uses only the compact version. However, this would break the versioning in https://docs.webknossos.org/webknossos/rest_api.html (even more). What are your feelings on this? Do you know if there are still users querying the Rest API directly, without our libs?
Related #5840

[normanrz]

What are your feelings on this? Do you know if there are still users querying the Rest API directly, without our libs?

I would mention it in migration guide and break it. I'd say that the REST API is only officially supported through our official libraries.