[oCIS] Batch restore can fail sometimes

[saw-jan]

Steps to reproduce

1. Upload 20-30 files
2. Delete all files (some requests might fail)
3. From trashbin Restore all files

Expected behavior

expected all files are restored

Actual behavior

some requests failed

HTTP log

MOVE /remote.php/dav/trash-bin/admin/55fffca2-a7d2-47e5-95b8-4e6957b318bd HTTP/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer eyJhbGciOiJQUzI1NiIsImtpZCI6IiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJ3ZWIiLCJleHAiOjE2NTExMjk5MjIsImp0aSI6Iko2UWRRMldPbk5XcHFVZ29xMU1HTGMzQ1JkUld1aHhXIiwiaWF0IjoxNjUxMTI5MzIyLCJpc3MiOiJodHRwczovL2hvc3QuZG9ja2VyLmludGVybmFsOjkyMDAiLCJzdWIiOiJmNnF2RDdoMmE1SFJsOWl6QEVDQjJOQzBCenJfTG54ZmJrZ3BsNFZ1UzJYOXM3T25zZnpVRFBONHVOU2RfanhFZ19sV0EzNGpzdXlXYW51RnREUHRIRlEiLCJsZy50IjoiMSIsInNjcCI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwibGcuaSI6eyJkbiI6IkFkbWluIiwiaWQiOiJjbj1hZG1pbixvdT11c2VycyxkYz1vY2lzLGRjPXRlc3QiLCJ1biI6ImFkbWluIn0sImxnLnAiOiJpZGVudGlmaWVyLWxkYXAifQ.pKKtdvwmJ-YlUUVHchdF7Lh_OuCmgDP5s4oD78DylXFy5hSIUbc_nG2RZFu-bzfv29RTq2HFEBdtSqScxEIodSLMqvFZJXTLCycU70pxc7lFzNzV625cB1yZbY2xUhNhnxLPgv2911GfdXhQG16-QtjsiZQhy3WXXI-rz1vcu3eZmmHnn1-hRUBYFew_082FWtl1Wg32Z5V6poLbT97sV-Z4oLVwGYxv5Msc21Mgbq4ZovoydQhWaaQqoquA3nx08xl102jm3dLYG8D9ibzF7OcGWVYX96mSruffqwCWxxnPtj__EIXDGtAMx8irQQ-nrHUF_jQmALgbIQGLqRt9Ag
Connection: keep-alive
Destination: https://host.docker.internal:9200/remote.php/dav/files/admin/textfile.txt
Host: host.docker.internal:9200
OCS-APIREQUEST: true
Origin: https://host.docker.internal:9200
Overwrite: F
Referer: https://host.docker.internal:9200/files/trash/personal?items-per-page=100
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36
X-Request-ID: 5b2941e2-9796-4aec-a152-005e5b51d6b7
X-Requested-With: XMLHttpRequest
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Content-Length: 0
Content-Security-Policy: default-src 'none';
Date: Thu, 28 Apr 2022 07:05:19 GMT
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-Xss-Protection: 1; mode=block

Browser console log

MOVE https://host.docker.internal:9200/remote.php/dav/trash-bin/admin/55fffca2-a7d2-47e5-95b8-4e6957b318bd 404 (Not Found)

Error: Unknown error
    at e.exports.buildHttpErrorFromDavResponse (vendor.js:116824:13061)
    at vendor.js:116824:45293
    at async web-app-files.js:4317:15
    at async run (vendor.js:34900:31)

oCIS log

The server yields the following log on request failure

{
  "level":"error",
  "service":"storage",
  "pkg":"rgrpc",
  "traceid":"00000000000000000000000000000000",
  "error":"remove /var/lib/ocis/storage/users/spacetypes/personal/ddc2004c-0977-11eb-9d3f-a793888cd0f8.flock: no such file or directory",
  "node":"marshaling error: json: unsupported value: encountered a cycle via *node.Node",
  "time":"2022-04-28T07:05:19Z",
  "message":"could not convert to storage space"
}
{
  "level":"error",
  "service":"storage",
  "pkg":"rgrpc",
  "traceid":"00000000000000000000000000000000",
  "error":"remove /var/lib/ocis/storage/users/spaces/dd/c2004c-0977-11eb-9d3f-a793888cd0f8/nodes/dd/c2/00/4c/-0977-11eb-9d3f-a793888cd0f8.flock: no such file or directory",
  "status":{
    "code":15,
    "message":"error listing spaces",
    "trace":"00000000000000000000000000000000"
  },
  "filters":[
    {
      "type":2,
      "Term":{
        "Id":{
          "opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8!ddc2004c-0977-11eb-9d3f-a793888cd0f8"
        }
      }
    }
  ],
  "time":"2022-04-28T07:05:19Z",
  "message":"failed to list storage spaces"
}

[individual-it]

This looks like an ocis issue to me. @kulmann @butonic what do you think moving to ocis?

@saw-jan please skip those tests for now for ocis and add a todo here to unskip the tests again

[saw-jan]

[QA] TODO:

• unskip the tests related to this issue (skipped in [tests-only][full-ci] Skip tests on oCIS related to batch request web#6828)

[kobergj]

Reason is the flock issue. Closing this in favor of #3757