Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Listing trash items is slow #9866

Open
butonic opened this issue Aug 20, 2024 · 5 comments
Open

Listing trash items is slow #9866

butonic opened this issue Aug 20, 2024 · 5 comments
Labels
Type:Bug

Comments

@butonic
Copy link
Member

butonic commented Aug 20, 2024 •
edited
Loading

On a local instance listing 10k items in the trash takes ~1s.

I instrumented the code a little... 
listTrashRoot glob 614.791µs trashRoot /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash subTrees 256
  [... one of the subtrees looks like this ...]
  subtree glob 3.839909ms nodePath /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02
      readTrashLink Readlink 5.831µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/20/cf/-adca-4bcf-bbfe-a405cee3a4d9
      readTrashLink EvalSymlinks 78.069µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/20/cf/-adca-4bcf-bbfe-a405cee3a4d9
      readTrashLink 105.575µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/20/cf/-adca-4bcf-bbfe-a405cee3a4d9
    match delta 308.41µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/20/cf/-adca-4bcf-bbfe-a405cee3a4d9
      readTrashLink Readlink 10.096µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/40/8a/-75bf-4fe0-9688-959dc3d375fc
      readTrashLink EvalSymlinks 94.001µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/40/8a/-75bf-4fe0-9688-959dc3d375fc
      readTrashLink 103.445µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/40/8a/-75bf-4fe0-9688-959dc3d375fc
    match delta 266.984µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/04/40/8a/-75bf-4fe0-9688-959dc3d375fc
      readTrashLink Readlink 8.834µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/08/70/41/-8cb9-4fc8-8fbe-ab094f48cfbe
      readTrashLink EvalSymlinks 93.629µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/08/70/41/-8cb9-4fc8-8fbe-ab094f48cfbe
      readTrashLink 100.965µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/08/70/41/-8cb9-4fc8-8fbe-ab094f48cfbe
    match delta 264.575µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/08/70/41/-8cb9-4fc8-8fbe-ab094f48cfbe
      readTrashLink Readlink 6.775µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/09/39/3f/-cac7-4b14-b963-41d10a5ab3b9
      readTrashLink EvalSymlinks 106.362µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/09/39/3f/-cac7-4b14-b963-41d10a5ab3b9
      readTrashLink 115.268µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/09/39/3f/-cac7-4b14-b963-41d10a5ab3b9
    match delta 287.344µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/09/39/3f/-cac7-4b14-b963-41d10a5ab3b9
      readTrashLink Readlink 8.149µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0e/c1/14/-bb25-442d-bb21-b8b87948a7c5
      readTrashLink EvalSymlinks 108.301µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0e/c1/14/-bb25-442d-bb21-b8b87948a7c5
      readTrashLink 117.004µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0e/c1/14/-bb25-442d-bb21-b8b87948a7c5
    match delta 289.116µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0e/c1/14/-bb25-442d-bb21-b8b87948a7c5
      readTrashLink Readlink 8.191µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0f/9e/c7/-758c-4562-a5b9-acbcac635814
      readTrashLink EvalSymlinks 109.153µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0f/9e/c7/-758c-4562-a5b9-acbcac635814
      readTrashLink 117.939µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0f/9e/c7/-758c-4562-a5b9-acbcac635814
    match delta 294.467µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/0f/9e/c7/-758c-4562-a5b9-acbcac635814
      readTrashLink Readlink 7.926µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/12/97/02/-7755-4477-86bc-dcf39e2f2c8f
      readTrashLink EvalSymlinks 107.666µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/12/97/02/-7755-4477-86bc-dcf39e2f2c8f
      readTrashLink 116.478µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/12/97/02/-7755-4477-86bc-dcf39e2f2c8f
    match delta 283.024µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/12/97/02/-7755-4477-86bc-dcf39e2f2c8f
      readTrashLink Readlink 8.132µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/1b/5b/41/-8099-4e24-9bf4-cfa51331ced5
      readTrashLink EvalSymlinks 112.099µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/1b/5b/41/-8099-4e24-9bf4-cfa51331ced5
      readTrashLink 122.074µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/1b/5b/41/-8099-4e24-9bf4-cfa51331ced5
    match delta 311.866µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/1b/5b/41/-8099-4e24-9bf4-cfa51331ced5
      readTrashLink Readlink 9.208µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/21/23/1d/-8ab0-44a6-914a-5dd54a5ae581
      readTrashLink EvalSymlinks 114.048µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/21/23/1d/-8ab0-44a6-914a-5dd54a5ae581
      readTrashLink 148.206µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/21/23/1d/-8ab0-44a6-914a-5dd54a5ae581
    match delta 549.414µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/21/23/1d/-8ab0-44a6-914a-5dd54a5ae581
      readTrashLink Readlink 9.625µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/23/34/16/-d1f6-4435-93d9-07b43fdffc73
      readTrashLink EvalSymlinks 116.287µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/23/34/16/-d1f6-4435-93d9-07b43fdffc73
      readTrashLink 127.51µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/23/34/16/-d1f6-4435-93d9-07b43fdffc73
    match delta 605.32µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/23/34/16/-d1f6-4435-93d9-07b43fdffc73
      readTrashLink Readlink 9.165µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/25/14/78/-5f92-456a-8ccf-5cbdf904e688
      readTrashLink EvalSymlinks 99.964µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/25/14/78/-5f92-456a-8ccf-5cbdf904e688
      readTrashLink 107.357µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/25/14/78/-5f92-456a-8ccf-5cbdf904e688
    match delta 394.299µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/25/14/78/-5f92-456a-8ccf-5cbdf904e688
      readTrashLink Readlink 9.639µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/28/98/f0/-bd52-4a45-bc96-ddb07de18a91
      readTrashLink EvalSymlinks 101.154µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/28/98/f0/-bd52-4a45-bc96-ddb07de18a91
      readTrashLink 110.03µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/28/98/f0/-bd52-4a45-bc96-ddb07de18a91
    match delta 688.895µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/28/98/f0/-bd52-4a45-bc96-ddb07de18a91
      readTrashLink Readlink 22.759µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/29/64/01/-da28-465d-8e90-52e9d3558fb7
      readTrashLink EvalSymlinks 98.562µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/29/64/01/-da28-465d-8e90-52e9d3558fb7
      readTrashLink 116.102µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/29/64/01/-da28-465d-8e90-52e9d3558fb7
    match delta 602.48µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/29/64/01/-da28-465d-8e90-52e9d3558fb7
      readTrashLink Readlink 7.852µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2b/c1/ab/-8390-4a3a-a302-8ecf87bbb5a0
      readTrashLink EvalSymlinks 95.559µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2b/c1/ab/-8390-4a3a-a302-8ecf87bbb5a0
      readTrashLink 102.778µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2b/c1/ab/-8390-4a3a-a302-8ecf87bbb5a0
    match delta 297.747µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2b/c1/ab/-8390-4a3a-a302-8ecf87bbb5a0
      readTrashLink Readlink 9.965µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2f/d8/2a/-8a33-4516-89a8-890d812b35f0
      readTrashLink EvalSymlinks 129.156µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2f/d8/2a/-8a33-4516-89a8-890d812b35f0
      readTrashLink 138.891µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2f/d8/2a/-8a33-4516-89a8-890d812b35f0
    match delta 304.132µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/2f/d8/2a/-8a33-4516-89a8-890d812b35f0
      readTrashLink Readlink 8.541µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/32/39/7f/-d2aa-4387-8d8f-4f8795e842d4
      readTrashLink EvalSymlinks 92.988µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/32/39/7f/-d2aa-4387-8d8f-4f8795e842d4
      readTrashLink 99.64µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/32/39/7f/-d2aa-4387-8d8f-4f8795e842d4
    match delta 327.802µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/32/39/7f/-d2aa-4387-8d8f-4f8795e842d4
      readTrashLink Readlink 8.203µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/36/56/3f/-f914-4242-82f1-f1bc29f7d497
      readTrashLink EvalSymlinks 106.073µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/36/56/3f/-f914-4242-82f1-f1bc29f7d497
      readTrashLink 112.908µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/36/56/3f/-f914-4242-82f1-f1bc29f7d497
    match delta 351.739µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/36/56/3f/-f914-4242-82f1-f1bc29f7d497
      readTrashLink Readlink 8.032µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3b/ad/11/-4f57-4bd4-878a-49caf4e4ad66
      readTrashLink EvalSymlinks 92.722µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3b/ad/11/-4f57-4bd4-878a-49caf4e4ad66
      readTrashLink 98.716µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3b/ad/11/-4f57-4bd4-878a-49caf4e4ad66
    match delta 365.531µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3b/ad/11/-4f57-4bd4-878a-49caf4e4ad66
      readTrashLink Readlink 10.018µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3f/6c/96/-9fe8-4409-97f2-4923d4457a13
      readTrashLink EvalSymlinks 111.928µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3f/6c/96/-9fe8-4409-97f2-4923d4457a13
      readTrashLink 122.396µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3f/6c/96/-9fe8-4409-97f2-4923d4457a13
    match delta 543.082µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/3f/6c/96/-9fe8-4409-97f2-4923d4457a13
      readTrashLink Readlink 8.424µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/42/72/2a/-10a4-49e5-aac8-434a52ce8610
      readTrashLink EvalSymlinks 98.099µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/42/72/2a/-10a4-49e5-aac8-434a52ce8610
      readTrashLink 106.767µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/42/72/2a/-10a4-49e5-aac8-434a52ce8610
    match delta 305.665µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/42/72/2a/-10a4-49e5-aac8-434a52ce8610
      readTrashLink Readlink 7.193µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/9c/af/-954d-4323-a646-ece8f43df715
      readTrashLink EvalSymlinks 115.802µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/9c/af/-954d-4323-a646-ece8f43df715
      readTrashLink 122.472µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/9c/af/-954d-4323-a646-ece8f43df715
    match delta 351.111µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/9c/af/-954d-4323-a646-ece8f43df715
      readTrashLink Readlink 9.39µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/e4/2f/-0e4a-4787-bd12-6f554831952b
      readTrashLink EvalSymlinks 102.837µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/e4/2f/-0e4a-4787-bd12-6f554831952b
      readTrashLink 111.64µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/e4/2f/-0e4a-4787-bd12-6f554831952b
    match delta 307.516µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/43/e4/2f/-0e4a-4787-bd12-6f554831952b
      readTrashLink Readlink 8.672µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4a/f3/a8/-8e48-453d-991c-65f349f4ce0a
      readTrashLink EvalSymlinks 96.085µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4a/f3/a8/-8e48-453d-991c-65f349f4ce0a
      readTrashLink 103.071µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4a/f3/a8/-8e48-453d-991c-65f349f4ce0a
    match delta 353.276µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4a/f3/a8/-8e48-453d-991c-65f349f4ce0a
      readTrashLink Readlink 9.093µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4b/7c/ba/-8f05-4cd9-a2a7-63b1d06eea7e
      readTrashLink EvalSymlinks 94.721µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4b/7c/ba/-8f05-4cd9-a2a7-63b1d06eea7e
      readTrashLink 102.539µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4b/7c/ba/-8f05-4cd9-a2a7-63b1d06eea7e
    match delta 358.058µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/4b/7c/ba/-8f05-4cd9-a2a7-63b1d06eea7e
      readTrashLink Readlink 8.758µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/53/32/14/-3560-4598-8adb-8e03ecd2b9dc
      readTrashLink EvalSymlinks 105.151µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/53/32/14/-3560-4598-8adb-8e03ecd2b9dc
      readTrashLink 114.345µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/53/32/14/-3560-4598-8adb-8e03ecd2b9dc
    match delta 311.042µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/53/32/14/-3560-4598-8adb-8e03ecd2b9dc
      readTrashLink Readlink 8.559µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/54/90/84/-aa75-4c74-944e-2d2097de3cb9
      readTrashLink EvalSymlinks 93.117µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/54/90/84/-aa75-4c74-944e-2d2097de3cb9
      readTrashLink 101.305µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/54/90/84/-aa75-4c74-944e-2d2097de3cb9
    match delta 298.375µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/54/90/84/-aa75-4c74-944e-2d2097de3cb9
      readTrashLink Readlink 7.085µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/57/aa/a9/-f09f-46ff-ba2e-4858dcfc805f
      readTrashLink EvalSymlinks 87.625µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/57/aa/a9/-f09f-46ff-ba2e-4858dcfc805f
      readTrashLink 95.659µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/57/aa/a9/-f09f-46ff-ba2e-4858dcfc805f
    match delta 433.951µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/57/aa/a9/-f09f-46ff-ba2e-4858dcfc805f
      readTrashLink Readlink 10.562µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/5b/16/b9/-679a-4ac2-9871-1782e007fa61
      readTrashLink EvalSymlinks 131.081µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/5b/16/b9/-679a-4ac2-9871-1782e007fa61
      readTrashLink 265.125µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/5b/16/b9/-679a-4ac2-9871-1782e007fa61
    match delta 492.978µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/5b/16/b9/-679a-4ac2-9871-1782e007fa61
      readTrashLink Readlink 8.59µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/60/a6/41/-78fa-4052-9829-7282fd54349a
      readTrashLink EvalSymlinks 105.244µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/60/a6/41/-78fa-4052-9829-7282fd54349a
      readTrashLink 186.472µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/60/a6/41/-78fa-4052-9829-7282fd54349a
    match delta 362.692µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/60/a6/41/-78fa-4052-9829-7282fd54349a
      readTrashLink Readlink 9.315µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/74/e0/32/-526a-47c5-aada-1ce36d74a560
      readTrashLink EvalSymlinks 110.095µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/74/e0/32/-526a-47c5-aada-1ce36d74a560
      readTrashLink 120.523µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/74/e0/32/-526a-47c5-aada-1ce36d74a560
    match delta 361.685µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/74/e0/32/-526a-47c5-aada-1ce36d74a560
      readTrashLink Readlink 8.93µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/37/13/-7ebc-4a03-a681-9db26d068ff5
      readTrashLink EvalSymlinks 96.33µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/37/13/-7ebc-4a03-a681-9db26d068ff5
      readTrashLink 234.036µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/37/13/-7ebc-4a03-a681-9db26d068ff5
    match delta 454.575µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/37/13/-7ebc-4a03-a681-9db26d068ff5
      readTrashLink Readlink 8.953µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/ff/40/-1e1d-462c-aa6a-32bca6f4c7b8
      readTrashLink EvalSymlinks 122.834µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/ff/40/-1e1d-462c-aa6a-32bca6f4c7b8
      readTrashLink 133.291µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/ff/40/-1e1d-462c-aa6a-32bca6f4c7b8
    match delta 311.852µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/75/ff/40/-1e1d-462c-aa6a-32bca6f4c7b8
      readTrashLink Readlink 9.766µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/76/4e/e8/-f424-4736-8462-b99f13b2847b
      readTrashLink EvalSymlinks 94.826µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/76/4e/e8/-f424-4736-8462-b99f13b2847b
      readTrashLink 104.237µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/76/4e/e8/-f424-4736-8462-b99f13b2847b
    match delta 404.004µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/76/4e/e8/-f424-4736-8462-b99f13b2847b
      readTrashLink Readlink 8.778µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/41/e5/-921e-4c39-8f0a-551016b39d94
      readTrashLink EvalSymlinks 96.638µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/41/e5/-921e-4c39-8f0a-551016b39d94
      readTrashLink 103.767µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/41/e5/-921e-4c39-8f0a-551016b39d94
    match delta 379.788µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/41/e5/-921e-4c39-8f0a-551016b39d94
      readTrashLink Readlink 8.308µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/53/65/-9915-48f1-996b-80c0cdec82dd
      readTrashLink EvalSymlinks 103.693µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/53/65/-9915-48f1-996b-80c0cdec82dd
      readTrashLink 112.467µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/53/65/-9915-48f1-996b-80c0cdec82dd
    match delta 309.909µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/78/53/65/-9915-48f1-996b-80c0cdec82dd
      readTrashLink Readlink 21.17µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/8c/44/c7/-1962-4ff8-ab25-61863660c479
      readTrashLink EvalSymlinks 105.257µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/8c/44/c7/-1962-4ff8-ab25-61863660c479
      readTrashLink 115.502µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/8c/44/c7/-1962-4ff8-ab25-61863660c479
    match delta 379.046µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/8c/44/c7/-1962-4ff8-ab25-61863660c479
      readTrashLink Readlink 8.205µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/a5/70/31/-fe2c-4aee-9e1f-cea81b06974f
      readTrashLink EvalSymlinks 119.665µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/a5/70/31/-fe2c-4aee-9e1f-cea81b06974f
      readTrashLink 131.138µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/a5/70/31/-fe2c-4aee-9e1f-cea81b06974f
    match delta 459.122µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/a5/70/31/-fe2c-4aee-9e1f-cea81b06974f
      readTrashLink Readlink 10.691µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ab/f3/1a/-3179-4a9c-b3b8-815ae15e0041
      readTrashLink EvalSymlinks 120.74µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ab/f3/1a/-3179-4a9c-b3b8-815ae15e0041
      readTrashLink 130.886µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ab/f3/1a/-3179-4a9c-b3b8-815ae15e0041
    match delta 363.839µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ab/f3/1a/-3179-4a9c-b3b8-815ae15e0041
      readTrashLink Readlink 10.009µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/b4/cb/04/-6179-4fec-8f26-29c289ac9663
      readTrashLink EvalSymlinks 147.246µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/b4/cb/04/-6179-4fec-8f26-29c289ac9663
      readTrashLink 157.63µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/b4/cb/04/-6179-4fec-8f26-29c289ac9663
    match delta 439.52µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/b4/cb/04/-6179-4fec-8f26-29c289ac9663
      readTrashLink Readlink 10.321µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c0/e0/81/-1064-47b5-bad3-c2dfaca530a4
      readTrashLink EvalSymlinks 144.916µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c0/e0/81/-1064-47b5-bad3-c2dfaca530a4
      readTrashLink 158.215µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c0/e0/81/-1064-47b5-bad3-c2dfaca530a4
    match delta 354.696µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c0/e0/81/-1064-47b5-bad3-c2dfaca530a4
      readTrashLink Readlink 8.347µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c2/95/98/-bec5-498a-9ba8-c2b26c7edaf2
      readTrashLink EvalSymlinks 157.486µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c2/95/98/-bec5-498a-9ba8-c2b26c7edaf2
      readTrashLink 167.138µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c2/95/98/-bec5-498a-9ba8-c2b26c7edaf2
    match delta 450.069µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c2/95/98/-bec5-498a-9ba8-c2b26c7edaf2
      readTrashLink Readlink 9.308µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c4/b0/2d/-bd9c-4532-b985-fc39db9d0d7d
      readTrashLink EvalSymlinks 111.25µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c4/b0/2d/-bd9c-4532-b985-fc39db9d0d7d
      readTrashLink 121.348µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c4/b0/2d/-bd9c-4532-b985-fc39db9d0d7d
    match delta 301.746µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/c4/b0/2d/-bd9c-4532-b985-fc39db9d0d7d
      readTrashLink Readlink 9.189µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/d4/61/56/-599a-4be4-aa95-32f23c851289
      readTrashLink EvalSymlinks 112.127µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/d4/61/56/-599a-4be4-aa95-32f23c851289
      readTrashLink 123.78µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/d4/61/56/-599a-4be4-aa95-32f23c851289
    match delta 604.464µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/d4/61/56/-599a-4be4-aa95-32f23c851289
      readTrashLink Readlink 7.648µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/e1/17/c0/-08af-4cef-90dc-b1ebc4a2218e
      readTrashLink EvalSymlinks 116.25µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/e1/17/c0/-08af-4cef-90dc-b1ebc4a2218e
      readTrashLink 126.5µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/e1/17/c0/-08af-4cef-90dc-b1ebc4a2218e
    match delta 517.245µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/e1/17/c0/-08af-4cef-90dc-b1ebc4a2218e
      readTrashLink Readlink 10.481µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/eb/d6/a0/-3329-477e-b577-f5fd1706e62d
      readTrashLink EvalSymlinks 101.121µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/eb/d6/a0/-3329-477e-b577-f5fd1706e62d
      readTrashLink 322.029µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/eb/d6/a0/-3329-477e-b577-f5fd1706e62d
    match delta 811.569µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/eb/d6/a0/-3329-477e-b577-f5fd1706e62d
      readTrashLink Readlink 9.559µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ee/fc/b3/-c32e-4116-9622-10c89d7755d1
      readTrashLink EvalSymlinks 120.12µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ee/fc/b3/-c32e-4116-9622-10c89d7755d1
      readTrashLink 251.945µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ee/fc/b3/-c32e-4116-9622-10c89d7755d1
    match delta 552.256µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/ee/fc/b3/-c32e-4116-9622-10c89d7755d1
      readTrashLink Readlink 23.494µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f2/d5/7f/-85de-4319-aba0-2688290661fd
      readTrashLink EvalSymlinks 106.581µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f2/d5/7f/-85de-4319-aba0-2688290661fd
      readTrashLink 116.084µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f2/d5/7f/-85de-4319-aba0-2688290661fd
    match delta 396.642µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f2/d5/7f/-85de-4319-aba0-2688290661fd
      readTrashLink Readlink 8.46µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f5/ef/e8/-21c9-4f07-9427-e59271751b7c
      readTrashLink EvalSymlinks 107.275µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f5/ef/e8/-21c9-4f07-9427-e59271751b7c
      readTrashLink 119.144µs path /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f5/ef/e8/-21c9-4f07-9427-e59271751b7c
    match delta 416.685µs item /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02/f5/ef/e8/-21c9-4f07-9427-e59271751b7c
  subtree delta 24.097212ms nodePath /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash/02 matches 48
[...]  
listTrashRoot delta 1.415534386s trashRoot /home/jfd/.ocis/storage/users/spaces/f7/fbf8c8-139b-4376-b307-cf0a8c2d0d9c/trash items 11714
for that subtree the glob took 3.839909ms but fetching the metadata of every trash item increased the time spent on the subtree to 24.097212ms

the 256 subtrees are crawled concurrently with ~14-51ms per subtree
@butonic
Copy link
Member Author

butonic commented Aug 20, 2024

While trying to upload and delete thousands of files I noticed that we are still not using async metadata propgation:

            "OCIS_DECOMPOSEDFS_PROPAGATOR": "async",
            "STORAGE_USERS_ASYNC_PROPAGATOR_PROPAGATION_DELAY": "10s",

without this the propagation happens in sync, which sometimes causes stale NFS handles:

[pod/storageusers-764bf68f45-6fcks/storageusers] {"level":"error","pid":1,"pid":1,"method":"sync.Propagate","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"6047a05c-b19e-4a6e-9745-6f27a310859b","sizeDiff":0,"error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","parent filename":"/var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock","time":"2024-08-20T15:22:13Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/decomposedfs.go:343","message":"Propagation failed. Could not open metadata for parent with lock."}
[pod/storageusers-764bf68f45-6fcks/storageusers] {"level":"error","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"b4b9cca6-7270-4a5a-9bb3-0d2a1a4ed96f","error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","time":"2024-08-20T15:22:13Z","message":"could not propagate etag change"}
[pod/storageusers-764bf68f45-6fcks/storageusers] {"level":"error","pid":1,"pid":1,"method":"sync.Propagate","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"6047a05c-b19e-4a6e-9745-6f27a310859b","sizeDiff":0,"error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","parent filename":"/var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock","time":"2024-08-20T15:22:16Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/decomposedfs.go:343","message":"Propagation failed. Could not open metadata for parent with lock."}
[pod/storageusers-764bf68f45-6fcks/storageusers] {"level":"error","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"a5e0053d-32c8-467e-8b41-b091b25abf75","error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","time":"2024-08-20T15:22:16Z","message":"could not propagate etag change"}
[pod/storageusers-764bf68f45-9rght/storageusers] {"level":"error","pid":1,"pid":1,"method":"sync.Propagate","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"6047a05c-b19e-4a6e-9745-6f27a310859b","sizeDiff":0,"error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","parent filename":"/var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock","time":"2024-08-20T15:22:19Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/decomposedfs.go:343","message":"Propagation failed. Could not open metadata for parent with lock."}
[pod/storageusers-764bf68f45-9rght/storageusers] {"level":"error","spaceid":"afabfc40-6cdc-4444-8c15-da201b23d46e","nodeid":"31d30f03-16ea-42b1-8a3d-2b641dc3ead0","error":"Lock /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/nodes/af/ab/fc/40/-6cdc-4444-8c15-da201b23d46e.mlock: stale NFS file handle","time":"2024-08-20T15:22:19Z","message":"could not propagate etag change"}

after enabling the async propagation I no longer see the stale NFS error. It seems to at least take the pressure of the filesystem. In any case ... this needs to be investigated. Failing to propagate breaks the metadata.

@butonic
Copy link
Member Author

butonic commented Aug 20, 2024

so on a kubernetes deploment with NFS latency kicks in:

And it looks as bad as you might guess... 
listTrashRoot glob 13.882323ms trashRoot /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash subTrees 256

[...]

  subtree glob 12.249648543s nodePath /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f
      readTrashLink Readlink 61.067085ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/05/21/74/-4c0b-4645-9ff7-5dfa5b930a25
      readTrashLink EvalSymlinks 756.193267ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/05/21/74/-4c0b-4645-9ff7-5dfa5b930a25
      readTrashLink 756.202134ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/05/21/74/-4c0b-4645-9ff7-5dfa5b930a25
    match delta 882.514314ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/05/21/74/-4c0b-4645-9ff7-5dfa5b930a25
      readTrashLink Readlink 68.509172ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/0e/3b/33/-91bc-42ae-b862-36564ffa9220
      readTrashLink EvalSymlinks 692.540435ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/0e/3b/33/-91bc-42ae-b862-36564ffa9220
      readTrashLink 692.627747ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/0e/3b/33/-91bc-42ae-b862-36564ffa9220
    match delta 812.619013ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/0e/3b/33/-91bc-42ae-b862-36564ffa9220
      readTrashLink Readlink 67.532546ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/12/1e/d5/-e809-42fc-b9df-a40937454a06
      readTrashLink EvalSymlinks 680.013655ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/12/1e/d5/-e809-42fc-b9df-a40937454a06
      readTrashLink 680.11783ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/12/1e/d5/-e809-42fc-b9df-a40937454a06
    match delta 807.620952ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/12/1e/d5/-e809-42fc-b9df-a40937454a06
      readTrashLink Readlink 54.533087ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1c/49/61/-e68b-480d-a436-961af35b32b6
      readTrashLink EvalSymlinks 652.810208ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1c/49/61/-e68b-480d-a436-961af35b32b6
      readTrashLink 652.841828ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1c/49/61/-e68b-480d-a436-961af35b32b6
    match delta 763.900356ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1c/49/61/-e68b-480d-a436-961af35b32b6
      readTrashLink Readlink 64.677627ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1d/7f/e9/-5a8e-43a9-bbaf-b5381bb2549d
      readTrashLink EvalSymlinks 728.890257ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1d/7f/e9/-5a8e-43a9-bbaf-b5381bb2549d
      readTrashLink 728.904473ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1d/7f/e9/-5a8e-43a9-bbaf-b5381bb2549d
    match delta 852.821047ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/1d/7f/e9/-5a8e-43a9-bbaf-b5381bb2549d
      readTrashLink Readlink 64.892958ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/38/d9/d2/-36c9-4738-b618-5c2f32209e1f
      readTrashLink EvalSymlinks 644.939316ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/38/d9/d2/-36c9-4738-b618-5c2f32209e1f
      readTrashLink 645.294457ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/38/d9/d2/-36c9-4738-b618-5c2f32209e1f
    match delta 774.279844ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/38/d9/d2/-36c9-4738-b618-5c2f32209e1f
      readTrashLink Readlink 52.753007ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/3a/66/da/-6420-4914-a1dd-7574a0ca20a3
      readTrashLink EvalSymlinks 661.973356ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/3a/66/da/-6420-4914-a1dd-7574a0ca20a3
      readTrashLink 661.986429ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/3a/66/da/-6420-4914-a1dd-7574a0ca20a3
    match delta 768.143726ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/3a/66/da/-6420-4914-a1dd-7574a0ca20a3
      readTrashLink Readlink 51.124459ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/40/b1/98/-d068-45fb-a70a-3376e734ce00
      readTrashLink EvalSymlinks 612.980167ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/40/b1/98/-d068-45fb-a70a-3376e734ce00
      readTrashLink 612.989625ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/40/b1/98/-d068-45fb-a70a-3376e734ce00
    match delta 718.287163ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/40/b1/98/-d068-45fb-a70a-3376e734ce00
      readTrashLink Readlink 49.587243ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/45/fe/bf/-3a0c-4783-98ce-747bfd5b48f2
      readTrashLink EvalSymlinks 614.549575ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/45/fe/bf/-3a0c-4783-98ce-747bfd5b48f2
      readTrashLink 614.559633ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/45/fe/bf/-3a0c-4783-98ce-747bfd5b48f2
    match delta 714.978362ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/45/fe/bf/-3a0c-4783-98ce-747bfd5b48f2
      readTrashLink Readlink 52.831584ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/47/6a/8a/-2734-48ac-a90f-37273aaae472
      readTrashLink EvalSymlinks 651.51988ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/47/6a/8a/-2734-48ac-a90f-37273aaae472
      readTrashLink 651.544135ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/47/6a/8a/-2734-48ac-a90f-37273aaae472
    match delta 759.602065ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/47/6a/8a/-2734-48ac-a90f-37273aaae472
      readTrashLink Readlink 57.39063ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/4c/ce/01/-3bb3-4fe8-8cdc-856e5f12d491
      readTrashLink EvalSymlinks 666.584398ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/4c/ce/01/-3bb3-4fe8-8cdc-856e5f12d491
      readTrashLink 666.600248ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/4c/ce/01/-3bb3-4fe8-8cdc-856e5f12d491
    match delta 779.351643ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/4c/ce/01/-3bb3-4fe8-8cdc-856e5f12d491
      readTrashLink Readlink 77.623306ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/50/6e/6a/-a446-4a8e-9316-12f4b5994446
      readTrashLink EvalSymlinks 708.574466ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/50/6e/6a/-a446-4a8e-9316-12f4b5994446
      readTrashLink 708.597438ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/50/6e/6a/-a446-4a8e-9316-12f4b5994446
    match delta 837.437185ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/50/6e/6a/-a446-4a8e-9316-12f4b5994446
      readTrashLink Readlink 67.865325ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/54/27/19/-e8ef-421b-bf69-760124842ac8
      readTrashLink EvalSymlinks 672.821214ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/54/27/19/-e8ef-421b-bf69-760124842ac8
      readTrashLink 672.834368ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/54/27/19/-e8ef-421b-bf69-760124842ac8
    match delta 794.542567ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/54/27/19/-e8ef-421b-bf69-760124842ac8
      readTrashLink Readlink 57.626889ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5a/48/7c/-0701-4fd6-a549-a6fad3736e9d
      readTrashLink EvalSymlinks 661.952376ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5a/48/7c/-0701-4fd6-a549-a6fad3736e9d
      readTrashLink 661.962355ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5a/48/7c/-0701-4fd6-a549-a6fad3736e9d
    match delta 788.860543ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5a/48/7c/-0701-4fd6-a549-a6fad3736e9d
      readTrashLink Readlink 49.903661ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5f/17/47/-f837-4c9f-ab97-269d0a396928
      readTrashLink EvalSymlinks 618.248904ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5f/17/47/-f837-4c9f-ab97-269d0a396928
      readTrashLink 618.262398ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5f/17/47/-f837-4c9f-ab97-269d0a396928
    match delta 721.55554ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/5f/17/47/-f837-4c9f-ab97-269d0a396928
      readTrashLink Readlink 51.881848ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/61/cf/1d/-b9af-446d-b7dc-6e67d7672599
      readTrashLink EvalSymlinks 617.26793ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/61/cf/1d/-b9af-446d-b7dc-6e67d7672599
      readTrashLink 617.280883ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/61/cf/1d/-b9af-446d-b7dc-6e67d7672599
    match delta 724.882595ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/61/cf/1d/-b9af-446d-b7dc-6e67d7672599
      readTrashLink Readlink 53.51246ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/63/02/10/-b672-477c-9af3-3b42cada4a29
      readTrashLink EvalSymlinks 647.202233ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/63/02/10/-b672-477c-9af3-3b42cada4a29
      readTrashLink 647.229474ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/63/02/10/-b672-477c-9af3-3b42cada4a29
    match delta 761.041251ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/63/02/10/-b672-477c-9af3-3b42cada4a29
      readTrashLink Readlink 56.34748ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/65/6e/d5/-7452-42a3-b3cc-de0bc12850e2
      readTrashLink EvalSymlinks 669.869676ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/65/6e/d5/-7452-42a3-b3cc-de0bc12850e2
      readTrashLink 669.897658ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/65/6e/d5/-7452-42a3-b3cc-de0bc12850e2
    match delta 788.177072ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/65/6e/d5/-7452-42a3-b3cc-de0bc12850e2
      readTrashLink Readlink 57.152167ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/68/b9/9a/-f95f-48a7-a58d-6a9140099605
      readTrashLink EvalSymlinks 705.493467ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/68/b9/9a/-f95f-48a7-a58d-6a9140099605
      readTrashLink 705.513164ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/68/b9/9a/-f95f-48a7-a58d-6a9140099605
    match delta 833.488987ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/68/b9/9a/-f95f-48a7-a58d-6a9140099605
      readTrashLink Readlink 53.52339ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/6d/8f/eb/-ba9c-4706-82a0-4e6374ba874b
      readTrashLink EvalSymlinks 683.483096ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/6d/8f/eb/-ba9c-4706-82a0-4e6374ba874b
      readTrashLink 683.527257ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/6d/8f/eb/-ba9c-4706-82a0-4e6374ba874b
    match delta 803.530758ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/6d/8f/eb/-ba9c-4706-82a0-4e6374ba874b
      readTrashLink Readlink 57.662005ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/72/99/d6/-3d5f-4adf-8010-dc161a613f66
      readTrashLink EvalSymlinks 645.355088ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/72/99/d6/-3d5f-4adf-8010-dc161a613f66
      readTrashLink 645.369145ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/72/99/d6/-3d5f-4adf-8010-dc161a613f66
    match delta 752.848611ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/72/99/d6/-3d5f-4adf-8010-dc161a613f66
      readTrashLink Readlink 60.42982ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/86/60/2b/-6735-4220-9b10-8d8962523e3c
      readTrashLink EvalSymlinks 624.226646ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/86/60/2b/-6735-4220-9b10-8d8962523e3c
      readTrashLink 624.240683ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/86/60/2b/-6735-4220-9b10-8d8962523e3c
    match delta 742.385447ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/86/60/2b/-6735-4220-9b10-8d8962523e3c
      readTrashLink Readlink 48.961238ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/8a/1b/5d/-e49f-4dab-bd80-f017b9115858
      readTrashLink EvalSymlinks 658.052255ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/8a/1b/5d/-e49f-4dab-bd80-f017b9115858
      readTrashLink 658.062244ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/8a/1b/5d/-e49f-4dab-bd80-f017b9115858
    match delta 766.695355ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/8a/1b/5d/-e49f-4dab-bd80-f017b9115858
      readTrashLink Readlink 53.176336ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/93/76/ce/-b36c-4a61-937f-e049576f85ff
      readTrashLink EvalSymlinks 625.434774ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/93/76/ce/-b36c-4a61-937f-e049576f85ff
      readTrashLink 625.452426ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/93/76/ce/-b36c-4a61-937f-e049576f85ff
    match delta 741.030528ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/93/76/ce/-b36c-4a61-937f-e049576f85ff
      readTrashLink Readlink 59.99395ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/97/ed/04/-2a45-44d3-ba0b-12452fd43627
      readTrashLink EvalSymlinks 629.215221ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/97/ed/04/-2a45-44d3-ba0b-12452fd43627
      readTrashLink 629.256248ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/97/ed/04/-2a45-44d3-ba0b-12452fd43627
    match delta 744.029012ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/97/ed/04/-2a45-44d3-ba0b-12452fd43627
      readTrashLink Readlink 56.251232ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/99/1a/85/-d725-4ff5-9fcc-76104a3db871
      readTrashLink EvalSymlinks 647.133896ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/99/1a/85/-d725-4ff5-9fcc-76104a3db871
      readTrashLink 647.163221ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/99/1a/85/-d725-4ff5-9fcc-76104a3db871
    match delta 758.354116ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/99/1a/85/-d725-4ff5-9fcc-76104a3db871
      readTrashLink Readlink 55.681753ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a0/d1/84/-a137-4279-87b3-302ef509db19
      readTrashLink EvalSymlinks 676.309299ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a0/d1/84/-a137-4279-87b3-302ef509db19
      readTrashLink 676.341789ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a0/d1/84/-a137-4279-87b3-302ef509db19
    match delta 785.111042ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a0/d1/84/-a137-4279-87b3-302ef509db19
      readTrashLink Readlink 61.801591ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a8/38/76/-7d62-4beb-90e8-af88d8d57860
      readTrashLink EvalSymlinks 644.194281ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a8/38/76/-7d62-4beb-90e8-af88d8d57860
      readTrashLink 644.206193ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a8/38/76/-7d62-4beb-90e8-af88d8d57860
    match delta 753.425903ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/a8/38/76/-7d62-4beb-90e8-af88d8d57860
      readTrashLink Readlink 53.760111ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/ab/9e/57/-46b6-496e-9141-6960a1c53855
      readTrashLink EvalSymlinks 627.610909ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/ab/9e/57/-46b6-496e-9141-6960a1c53855
      readTrashLink 627.688734ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/ab/9e/57/-46b6-496e-9141-6960a1c53855
    match delta 737.837271ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/ab/9e/57/-46b6-496e-9141-6960a1c53855
      readTrashLink Readlink 66.646189ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/b7/1c/49/-30ca-48e9-8b99-e9e95d6c12d8
      readTrashLink EvalSymlinks 626.021255ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/b7/1c/49/-30ca-48e9-8b99-e9e95d6c12d8
      readTrashLink 626.042934ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/b7/1c/49/-30ca-48e9-8b99-e9e95d6c12d8
    match delta 743.898721ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/b7/1c/49/-30ca-48e9-8b99-e9e95d6c12d8
      readTrashLink Readlink 48.032443ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c5/7e/27/-2f18-4fd8-98b1-7fa26786ba7c
      readTrashLink EvalSymlinks 635.673088ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c5/7e/27/-2f18-4fd8-98b1-7fa26786ba7c
      readTrashLink 635.686534ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c5/7e/27/-2f18-4fd8-98b1-7fa26786ba7c
    match delta 738.334426ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c5/7e/27/-2f18-4fd8-98b1-7fa26786ba7c
      readTrashLink Readlink 58.49894ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c6/8d/c6/-3f30-461f-8889-848d49c81ecd
      readTrashLink EvalSymlinks 636.324069ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c6/8d/c6/-3f30-461f-8889-848d49c81ecd
      readTrashLink 636.334188ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c6/8d/c6/-3f30-461f-8889-848d49c81ecd
    match delta 751.339776ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c6/8d/c6/-3f30-461f-8889-848d49c81ecd
      readTrashLink Readlink 51.144718ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c7/42/c7/-d35d-414f-abea-2b25e2a008b6
      readTrashLink EvalSymlinks 647.501139ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c7/42/c7/-d35d-414f-abea-2b25e2a008b6
      readTrashLink 647.513572ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c7/42/c7/-d35d-414f-abea-2b25e2a008b6
    match delta 758.462828ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/c7/42/c7/-d35d-414f-abea-2b25e2a008b6
      readTrashLink Readlink 51.184822ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/cb/2e/82/-d5e3-43aa-bb54-fae9fa07450e
      readTrashLink EvalSymlinks 751.093686ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/cb/2e/82/-d5e3-43aa-bb54-fae9fa07450e
      readTrashLink 751.10633ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/cb/2e/82/-d5e3-43aa-bb54-fae9fa07450e
    match delta 870.592308ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/cb/2e/82/-d5e3-43aa-bb54-fae9fa07450e
      readTrashLink Readlink 65.356089ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/da/41/bb/-2789-4ce7-a7d6-9484ef8add19
      readTrashLink EvalSymlinks 807.844528ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/da/41/bb/-2789-4ce7-a7d6-9484ef8add19
      readTrashLink 807.887148ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/da/41/bb/-2789-4ce7-a7d6-9484ef8add19
    match delta 949.838382ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/da/41/bb/-2789-4ce7-a7d6-9484ef8add19
      readTrashLink Readlink 66.657549ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/dc/0c/82/-afbc-41cb-8949-3763e49e2850
      readTrashLink EvalSymlinks 778.421044ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/dc/0c/82/-afbc-41cb-8949-3763e49e2850
      readTrashLink 778.433267ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/dc/0c/82/-afbc-41cb-8949-3763e49e2850
    match delta 909.428001ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/dc/0c/82/-afbc-41cb-8949-3763e49e2850
      readTrashLink Readlink 69.532124ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/de/6d/6e/-8767-4b8d-aad0-415742d6a314
      readTrashLink EvalSymlinks 645.222543ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/de/6d/6e/-8767-4b8d-aad0-415742d6a314
      readTrashLink 645.236449ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/de/6d/6e/-8767-4b8d-aad0-415742d6a314
    match delta 768.928936ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/de/6d/6e/-8767-4b8d-aad0-415742d6a314
      readTrashLink Readlink 51.978548ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9
      readTrashLink EvalSymlinks 673.557302ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9
      readTrashLink 673.592438ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9
    match delta 776.385289ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9
      readTrashLink Readlink 50.030686ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f2/08/6c/-9035-4dbc-9d6e-94aa1b58a366
      readTrashLink EvalSymlinks 611.349025ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f2/08/6c/-9035-4dbc-9d6e-94aa1b58a366
      readTrashLink 611.369924ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f2/08/6c/-9035-4dbc-9d6e-94aa1b58a366
    match delta 714.477981ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f2/08/6c/-9035-4dbc-9d6e-94aa1b58a366
      readTrashLink Readlink 56.113777ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f6/cf/d9/-21d9-4c92-80ae-bcf1c3745997
      readTrashLink EvalSymlinks 725.536912ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f6/cf/d9/-21d9-4c92-80ae-bcf1c3745997
      readTrashLink 725.799871ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f6/cf/d9/-21d9-4c92-80ae-bcf1c3745997
    match delta 841.259022ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f6/cf/d9/-21d9-4c92-80ae-bcf1c3745997
      readTrashLink Readlink 56.221136ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f9/32/27/-b4b3-42bb-ae18-ef4e12f0a69b
      readTrashLink EvalSymlinks 660.962426ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f9/32/27/-b4b3-42bb-ae18-ef4e12f0a69b
      readTrashLink 660.984538ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f9/32/27/-b4b3-42bb-ae18-ef4e12f0a69b
    match delta 778.471588ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/f9/32/27/-b4b3-42bb-ae18-ef4e12f0a69b
      readTrashLink Readlink 49.486214ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/fe/d0/90/-533e-4804-aece-93b83c5e5bae
      readTrashLink EvalSymlinks 642.810548ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/fe/d0/90/-533e-4804-aece-93b83c5e5bae
      readTrashLink 642.823432ms path /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/fe/d0/90/-533e-4804-aece-93b83c5e5bae
    match delta 753.374718ms item /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/fe/d0/90/-533e-4804-aece-93b83c5e5bae
  subtree delta 45.075275968s nodePath /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f matches 42

[...]

listTrashRoot delta 1m3.546910539s trashRoot /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash items 4529

reading a single symlink takes 600+ ms 😞

@butonic
Copy link
Member Author

butonic commented Aug 20, 2024 •
edited
Loading

now ... I don't think we need to use symlinks here ... we should be able to store the timestamp as part of the trash path. no need to use a symlink. we can just use /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9.T.2022-02-16T14:38:11.769917408Z instead of /var/lib/ocis/storage/users/spaces/af/abfc40-6cdc-4444-8c15-da201b23d46e/trash/2f/e2/37/5c/-0ecb-4c45-8082-46b8246a27d9 and reading it from the link.

the downside is that when listing the trash with a key we need to look for a file and ignore the timestamp in the path.

and we would need a migration step ... to rewrite the symlinks

@butonic
Copy link
Member Author

butonic commented Aug 20, 2024

or we think about putting dir listings in the cache ...

@butonic
Copy link
Member Author

butonic commented Aug 20, 2024 •
edited
Loading

hm nats can use watch to get a list of all keys using a prefix, see: nats-io/nats.go#957

keys would take the form base64(nodeid)+ '.' + base64({childname}) ...

@butonic butonic mentioned this issue Aug 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type:Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@butonic