Allow renaming datasets & dataset with duplicate names #8075

MichaelBuessemeyer · 2024-09-12T16:02:08Z

Further Notes:

Quite some of the line changes are the result of moving the ObjectId class to the utils package so that all wk backend servers have access to this class.

URL of deployed dev instance (used for testing):

https://allowdatasetrenaming.webknossos.xyz

Steps to test:

Give two datasets the same name and check whether annotations and so on works
Test whether the task system still works with duplicate dataset names
check dataset upload
- dataset upload
- add remote
- compose
check moving datasets between folders (single & multiple at once)
check whether the renaming works
when there are duplicate names, the correct dataset should open upon view it (not another one with the same name)
Test worker jobs (tested by @fm3)
Test legacy routes
And much much more 🙈

TODOs:

I would also suggest to

Execute the screenshot tests once for this branch. Take care to change the branch name in the CI config (or local command), so that the tests are actually ran on this branch.

Looks much better now :) Did you re-test that exceptions are caught as expected?
undo application.conf & snapshot changes -> in progress

Issues:

(Please delete unneeded items, merge only when none are left open)

Updated changelog
Updated migration guide if applicable
Updated documentation if applicable
Needs to execute a mirgation
Needs datastore update after deployment

…ir dataset addressing scheme

… the check for already existing name)

…gaId - Includes moving ObjectId to uitls package

…store & add legacy routes - Undo renaming DataSourceId to LegacyDataSourceId

…ow-dataset-renaming

…ataset object

…d required legacy adaption

- no legacy routes needed as not used by wklibs

…egacy controller

MichaelBuessemeyer · 2024-11-26T14:36:57Z

I now removed the dataSet field in jsonified task object and added the legacy adaption.
I checked routes that needed adaptation and came up with that the following do not need adaptation or do they?

- /tasks/:id/assign                                     controllers.TaskController.assignOne
- /taskTypes/:id/tasks                                  controllers.TaskController.listTasksForType
- /tasks/list                                           controllers.TaskController.listTasks
- /user/tasks/peek                                      controllers.TaskController.peekNext
- /tasks/:id                                            controllers.TaskController.update
- /annotations/:id/addAnnotationLayer                   controllers.AnnotationController.addAnnotationLayerWithoutType
- /annotations/:typ/:id/addAnnotationLayer              controllers.AnnotationController.addAnnotationLayer
- /datasets/:datasetId/createExplorational              controllers.AnnotationController.createExplorational
- /annotations/:id/downsample                           controllers.AnnotationController.downsampleWithoutType
- /annotations/:typ/:id/downsample                      controllers.AnnotationController.downsample
- /annotations/:typ/:id/duplicate                       controllers.AnnotationController.duplicate
- /annotations/:typ/:id/editLockedState                 controllers.AnnotationController.editLockedState
- /annotations/:typ/:id/finish                          controllers.AnnotationController.finish
- /datasets/:datasetId/sandbox/:typ                     controllers.AnnotationController.getSandbox
- /annotations/:typ/:id/info                            controllers.AnnotationController.info
- /annotations/:typ/:id/makeHybrid                      controllers.AnnotationController.makeHybrid
- /annotations/:typ/:id/merge/:mergedTyp/:mergedId      controllers.AnnotationController.merge
- /annotations/:typ/:id/reopen                          controllers.AnnotationController.reopen
- /annotations/:typ/:id/reset                           controllers.AnnotationController.reset
- /annotations/:typ/:id/transfer                        controllers.AnnotationController.transfer

I'll do testing of the new legacy routes later

fm3 · 2024-11-26T14:42:33Z

I checked routes that needed adaptation and came up with that the following do not need adaptation or do they?

Looks right to me!

…ow-dataset-renaming

- also remove unised injections and imports

MichaelBuessemeyer · 2024-11-27T10:06:24Z

Hi @fm3,

I added the required legacy routes to remove dataSet from the json serialized task objects in the newest API version. Please find the following links & curl script to test these routes with the dev instance. All ids included & dataset names should exist on the dev instance and therefore just clicking the links should work. As these are legacy routes, the results should include the legacy field dataSet in each task object. Some routes return an annotation which itself has a task.

For the curl commands, you first need to fill in the id cookie before you can run the scripts.

Before testing here is one more important thing to double check: The legacy routes now include both: dataSet and datasetName. Can wklibs handle / ignore new unexpected json fields?

Checklist for testing new Legacy Routes

Testing create & update task -> :shurg: I did this manually in firefox with the edit & resent feature on the dev instance.

In case you want to test more complicated routes:

curl command for checking task creation (setting id cookie is required)

curl 'https://allowdatasetrenaming.webknossos.xyz/api/tasks' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9,de;q=0.8' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -H 'cookie: id=<fill-me-in>' \
  -H 'origin: https://allowdatasetrenaming.webknossos.xyz' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://allowdatasetrenaming.webknossos.xyz/tasks/create' \
  -H 'sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' \
  --data-raw '[{"taskTypeId":"63721e2cef0100470266c485","neededExperience":{"domain":"sampleExp","value":1},"pendingInstances":1,"projectName":"sampleProject","boundingBox":null,"dataSet":"kiwi","editPosition":[0,0,0],"editRotation":[0,0,0],"baseAnnotation":null}]'

Test task update: (setting id cookie is required)

curl 'https://allowdatasetrenaming.webknossos.xyz/api/tasks/6746ecc6010000d80015ea8c' \
  -X 'PUT' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9,de;q=0.8' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -H 'cookie: id=<fill-me-in>' \
  -H 'origin: https://allowdatasetrenaming.webknossos.xyz' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://allowdatasetrenaming.webknossos.xyz/tasks/6746ecc6010000d80015ea8c/edit' \
  -H 'sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' \
  --data-raw '{"taskTypeId":"63721e2cef0100470266c485","neededExperience":{"domain":"sampleExp","value":1},"pendingInstances":2,"projectName":"sampleProject","boundingBox":null,"datasetId":"kiwi","editPosition":[0,0,0],"editRotation":[0,0,0]}'

fm3

LGTM & Works for me =) Thanks! I added two more small comments. Also, have a look at the merge conflicts. None of that should be much work. Let’s ship it!

fm3 · 2024-11-27T10:14:23Z

app/controllers/LegacyApiController.scala

                                    sil: Silhouette[WkEnv])(implicit ec: ExecutionContext, bodyParsers: PlayBodyParsers)
    extends Controller {

+  /* to provide v8, remove legacy routes */


Not sure I understand this comment. What legacy routes are removed? (I mean, the PR removes some, but this comment appears to be about the following code lines). Maybe it should be something like “to provide v8, add dataSet to task json?” Or maybe remove these comments altogether.

Ah damn it. This is a mixture of a todo comment which I forgot to remove and copy & paste mistake.

I renamed all such occurrences to simple /* provide vX */

Or maybe remove these comments altogether.

Something like these comments were present previously (I think). Thus, I kept them to "order" the code a little :)

I can also remove them in case you prefer that :)

fm3 · 2024-11-27T10:15:39Z

CHANGELOG.unreleased.md

@@ -26,6 +27,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Fix a bug where dataset uploads would fail if the organization directory on disk is missing. [#8230](https://github.com/scalableminds/webknossos/pull/8230)

 ### Removed
+- Removed legacy routes for versions 2,3 and 4. [#8075](https://github.com/scalableminds/webknossos/pull/8075)


Suggested change

- Removed legacy routes for versions 2,3 and 4. [#8075](https://github.com/scalableminds/webknossos/pull/8075)

- Removed support for HTTP API versions 3 and 4. [#8075](https://github.com/scalableminds/webknossos/pull/8075)

(v2 was already removed previously) Could you also mention this in the migration guide please? (If someone has old code interacting with the HTTP API, they need to update)

…ow-dataset-renaming

- fix changelog entry - add migration entry about dropped api versions

MichaelBuessemeyer · 2024-11-27T11:09:40Z

merge conflicts & your two comments are done now. So it's ready for 🚢

fm3 · 2024-11-27T11:19:34Z

🎉 🤞

WIP: Adjust schema to allow duplicate dataset names & implement new u…

5ffa251

…ir dataset addressing scheme

MichaelBuessemeyer added backend frontend new feature labels Sep 12, 2024

MichaelBuessemeyer self-assigned this Sep 12, 2024

MichaelBuessemeyer and others added 21 commits September 16, 2024 17:13

reimplement proper dataset name checking route (still keep leave away…

d34bdeb

… the check for already existing name)

WIP: implement wk core backend routes to only use datasetId and no or…

49432a9

…gaId - Includes moving ObjectId to uitls package

WIP: finish using dataset id in wk core backend and dataspath in data…

5d3f8bd

…store & add legacy routes - Undo renaming DataSourceId to LegacyDataSourceId

WIP: Fix backend compilation

0c111cb

Fix backend compilation

d8ad983

WIP: Adapt frontend to new api

10724e7

WIP: adapt frontend to new routes

1a8349a

WIP: Adjust frontend to newest api

6b54a7b

first kinda working version

394c023

Try update schema and evolution

62bbd75

fix evolution & add first version of reversion (needs to be tested)

f74a7f1

fix frontend tests

3a203a8

format backend

a65c7fb

fix dataSets.csv

2234b5b

fix e2e tests

3570545

Merge branch 'master' of github.com:scalableminds/webknossos into all…

cbf4470

…ow-dataset-renaming

format backend

50e918a

fix frontend

90990b5

remove occurences of displayName access / variables in context of a d…

1b0aec3

…ataset object

fixed verion routes

7a79ea4

fix reserveUploadRoute

5443912

fm3 mentioned this pull request Oct 7, 2024

Rename organization_name to organization_id in worker args #8038

Closed

Michael Büßemeyer added 3 commits October 8, 2024 11:55

rename orga_name in jobs to orga_id

8224f6a

format code

35d62a9

fix finishUploadRoute

1b6ff5c

Michael Büßemeyer added 6 commits November 26, 2024 14:09

remove dataSet field from task json object returned from server -> ad…

7afa16a

…d required legacy adaption

also rename dataSet in publication routes to dataset

3e5df15

- no legacy routes needed as not used by wklibs

add changelog entry about dropping legacy routes

6211451

bump api version

bb4a473

remove old routes with api version lower than 5 & resort methods in l…

065379e

…egacy controller

refresh snapshots

45d3bc4

Michael Büßemeyer added 4 commits November 26, 2024 22:48

Merge branch 'master' of github.com:scalableminds/webknossos into all…

48850c1

…ow-dataset-renaming

fix legacy create task route

b42ccbc

- also remove unised injections and imports

fix annotation info legacy route

b9dad1d

remove unused import

b382cb4

MichaelBuessemeyer requested a review from fm3 November 27, 2024 10:06

fm3 approved these changes Nov 27, 2024

View reviewed changes

Michael Büßemeyer added 2 commits November 27, 2024 11:26

Merge branch 'master' of github.com:scalableminds/webknossos into all…

539fb64

…ow-dataset-renaming

- fix in code comments

43e814a

- fix changelog entry - add migration entry about dropped api versions

MichaelBuessemeyer merged commit 6c0a472 into master Nov 27, 2024
3 checks passed

MichaelBuessemeyer deleted the allow-dataset-renaming branch November 27, 2024 13:46

MichaelBuessemeyer mentioned this pull request Nov 27, 2024

Rename resolution to mag in local variables #8168

Merged

15 tasks

fm3 mentioned this pull request Nov 27, 2024

Fix accessing public datasets #8242

Merged

This was referenced Nov 29, 2024

Add missing isValidNewName legacy route #8252

Merged

Unified annotation versioning #7917

Merged

Add switch orga to legacy routes #8257

Merged

Fix legacy support for outdated displayName field of datasets #8263

Merged

This was referenced Dec 16, 2024

Fix nml upload with only ds name in nml by using user's orga as fallback #8277

Merged

Fix appending task bbox to user bbox in annotation nml upload #8280

Merged

Remove unused legacy routes #8293

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow renaming datasets & dataset with duplicate names #8075

Allow renaming datasets & dataset with duplicate names #8075

MichaelBuessemeyer commented Sep 12, 2024 •

edited

Loading

MichaelBuessemeyer commented Nov 26, 2024

fm3 commented Nov 26, 2024

MichaelBuessemeyer commented Nov 27, 2024

fm3 left a comment

fm3 Nov 27, 2024

MichaelBuessemeyer Nov 27, 2024

MichaelBuessemeyer Nov 27, 2024

fm3 Nov 27, 2024 •

edited

Loading

MichaelBuessemeyer commented Nov 27, 2024

fm3 commented Nov 27, 2024 •

edited

Loading

	- Removed legacy routes for versions 2,3 and 4. [#8075](https://github.com/scalableminds/webknossos/pull/8075)
	- Removed support for HTTP API versions 3 and 4. [#8075](https://github.com/scalableminds/webknossos/pull/8075)

Allow renaming datasets & dataset with duplicate names #8075

Allow renaming datasets & dataset with duplicate names #8075

Conversation

MichaelBuessemeyer commented Sep 12, 2024 • edited Loading

Further Notes:

URL of deployed dev instance (used for testing):

Steps to test:

TODOs:

Issues:

MichaelBuessemeyer commented Nov 26, 2024

fm3 commented Nov 26, 2024

MichaelBuessemeyer commented Nov 27, 2024

Checklist for testing new Legacy Routes

In case you want to test more complicated routes:

curl command for checking task creation (setting id cookie is required)

Test task update: (setting id cookie is required)

fm3 left a comment

Choose a reason for hiding this comment

fm3 Nov 27, 2024

Choose a reason for hiding this comment

MichaelBuessemeyer Nov 27, 2024

Choose a reason for hiding this comment

MichaelBuessemeyer Nov 27, 2024

Choose a reason for hiding this comment

fm3 Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

MichaelBuessemeyer commented Nov 27, 2024

fm3 commented Nov 27, 2024 • edited Loading

MichaelBuessemeyer commented Sep 12, 2024 •

edited

Loading

fm3 Nov 27, 2024 •

edited

Loading

fm3 commented Nov 27, 2024 •

edited

Loading