[OCIS] run acceptance tests with EOS #156

individual-it · 2020-05-07T07:48:44Z

API
UI

how to run see: https://owncloud.github.io/ocis/eos/#how-to-do-it

individual-it · 2020-05-12T10:08:32Z

I cannot make it work same issue as owncloud/ocis#257

individual-it · 2020-05-13T11:06:04Z

maybe can be unblocked by #34

individual-it · 2020-06-05T07:43:48Z

I managed to use an external LDAP server and fire it up with EOS & OCIS. see: owncloud-docker/compose-playground#28

Next issue is the skeleton folder, currently using local storage it just copies the skeletonfolder into the destination, we need an other approach for EOS

individual-it · 2020-06-05T08:14:58Z

one more issue is the DAV name-space:
/webdav works as expected
/dav/files has an extra folder layer that is the first letter of the name of the user E.g. the content of Alice is inside /remote.php/dav/files/A/Alice/

individual-it · 2020-06-09T04:10:07Z

fixing the name-space worked by setting REVA_STORAGE_EOS_LAYOUT: "{{.Username}}"

individual-it · 2020-06-09T04:43:29Z

Still I cannot upload:
curl -k -u user0:123456 -X PUT https://localhost:9200/remote.php/dav/files/user0/newfile.txt -d "123" -v

results in and error 500
logs:

ocis          | 2020-06-09T04:42:04Z DBG eos: wrap external=/user0/newfile.txt internal=/eos/dockertest/reva/users/user0/newfile.txt pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | nslcd: [90cde7] DEBUG: connection from pid=15 uid=0 gid=0
ocis          | nslcd: [90cde7] <passwd="user0"> DEBUG: myldap_search(base="dc=owncloud,dc=com", filter="(&(objectClass=posixAccount)(uid=user0))")
ocis          | nslcd: [90cde7] <passwd="user0"> DEBUG: ldap_result(): uid=user0,ou=TestUsers,dc=owncloud,dc=com
ocis          | nslcd: [90cde7] <passwd="user0"> DEBUG: ldap_result(): end of results (1 total)
ocis          | 2020-06-09T04:42:04Z INF eos cmd args="[/usr/bin/eos -r 30000 5000 file info /eos/dockertest/reva/users/user0/newfile.txt -m --comment 70a2295a1ef12aa137196ccc025821a1]" env=[EOS_MGM_URL=root://mgm-master.testnet:1094] err="error: cannot stat '/eos/dockertest/reva/users/user0/newfile.txt'\n (errc=2) (No such file or directory)\n" exit=2 pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z WRN root/go/pkg/mod/github.com/cs3org/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:457 > file not found pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z INF unary code=OK end="09/Jun/2020:04:42:04 +0000" from=tcp://127.0.0.1:48994 pkg=rgrpc service=reva start="09/Jun/2020:04:42:04 +0000" time_ns=22309165 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/cs3.storage.provider.v1beta1.ProviderAPI/Stat user-agent=grpc-go/1.26.0
ocis          | 2020-06-09T04:42:04Z INF unary code=OK end="09/Jun/2020:04:42:04 +0000" from=tcp://127.0.0.1:42828 pkg=rgrpc service=reva start="09/Jun/2020:04:42:04 +0000" time_ns=24024653 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/cs3.gateway.v1beta1.GatewayAPI/Stat user-agent=grpc-go/1.26.0
ocis          | 2020-06-09T04:42:04Z DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/InitiateFileUpload pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z INF unary code=OK end="09/Jun/2020:04:42:04 +0000" from=tcp://127.0.0.1:42836 pkg=rgrpc service=reva start="09/Jun/2020:04:42:04 +0000" time_ns=103821 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
ocis          | 2020-06-09T04:42:04Z DBG eos: wrap external=/user0/newfile.txt internal=/eos/dockertest/reva/users/user0/newfile.txt pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z DBG eos: NewUpload info={"ID":"","IsFinal":false,"IsPartial":false,"MetaData":{"dir":"/eos/dockertest/reva/users/user0","filename":"newfile.txt"},"Offset":0,"PartialUploads":null,"Size":3,"SizeIsDeferred":false,"Storage":null} pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z DBG eos: resolved filename info={"ID":"","IsFinal":false,"IsPartial":false,"MetaData":{"dir":"/eos/dockertest/reva/users/user0","filename":"newfile.txt"},"Offset":0,"PartialUploads":null,"Size":3,"SizeIsDeferred":false,"Storage":null} pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z INF file upload data-server=http://localhost:9160/data/de7bb989-e845-4f36-ad0e-75b12add5b94 fn=/eos/user0/newfile.txt pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1 xs="map[md5:100 unset:1000]"
ocis          | 2020-06-09T04:42:04Z INF unary code=OK end="09/Jun/2020:04:42:04 +0000" from=tcp://127.0.0.1:48994 pkg=rgrpc service=reva start="09/Jun/2020:04:42:04 +0000" time_ns=517825 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/cs3.storage.provider.v1beta1.ProviderAPI/InitiateFileUpload user-agent=grpc-go/1.26.0
ocis          | 2020-06-09T04:42:04Z INF upload is routed directly to data server - skiping datagateway pkg=rgrpc service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z INF unary code=OK end="09/Jun/2020:04:42:04 +0000" from=tcp://127.0.0.1:42828 pkg=rgrpc service=reva start="09/Jun/2020:04:42:04 +0000" time_ns=1575374 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/cs3.gateway.v1beta1.GatewayAPI/InitiateFileUpload user-agent=grpc-go/1.26.0
ocis          | 2020-06-09T04:42:04Z DBG adding token to header header=x-access-token pkg=rhttp service=reva token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNTkxNjgxMzI0LCJpYXQiOjE1OTE2Nzc3MjQsImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJ1c2VyIjp7ImlkIjp7ImlkcCI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJvcGFxdWVfaWQiOiJ1c2VyMCJ9LCJ1c2VybmFtZSI6InVzZXIwIiwibWFpbCI6InVzZXIwQGV4YW1wbGUub3JnIn19.cAW_Is_nNWt3miSuoGzR5CoIE7RggaBGokgJmcbfxQU traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z INF access token is already provided pkg=rhttp service=reva traceid=782ce4ece8b49b84ca985ce9532812af
ocis          | 2020-06-09T04:42:04Z DBG http routing: head=data tail=/de7bb989-e845-4f36-ad0e-75b12add5b94 svc=data pkg=rhttp service=reva
ocis          | [tusd] 2020/06/09 04:42:04 event="RequestIncoming" method="PATCH" path="/de7bb989-e845-4f36-ad0e-75b12add5b94" requestId="" 
ocis          | 2020-06-09T04:42:04Z INF tusd routing: path=/de7bb989-e845-4f36-ad0e-75b12add5b94 pkg=rhttp service=reva traceid=782ce4ece8b49b84ca985ce9532812af
ocis          | [tusd] 2020/06/09 04:42:04 event="ChunkWriteStart" id="de7bb989-e845-4f36-ad0e-75b12add5b94" maxSize="3" offset="0" 
ocis          | [tusd] 2020/06/09 04:42:04 event="ChunkWriteComplete" id="de7bb989-e845-4f36-ad0e-75b12add5b94" bytesWritten="3" 
ocis          | nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable
ocis          | nslcd: [ef438d] DEBUG: connection from pid=16 uid=0 gid=0
ocis          | nslcd: [ef438d] <passwd="user0"> DEBUG: myldap_search(base="dc=owncloud,dc=com", filter="(&(objectClass=posixAccount)(uid=user0))")
ocis          | nslcd: [ef438d] <passwd="user0"> DEBUG: ldap_result(): uid=user0,ou=TestUsers,dc=owncloud,dc=com
ocis          | nslcd: [ef438d] <passwd="user0"> DEBUG: ldap_result(): end of results (1 total)
ocis          | [tusd] 2020/06/09 04:42:04 event="ResponseOutgoing" status="500" method="PATCH" path="/de7bb989-e845-4f36-ad0e-75b12add5b94" error="eosclient: error while executing command: exit status 54" requestId="" 
ocis          | 2020-06-09T04:42:04Z ERR http end="09/Jun/2020:04:42:04 +0000" host=127.0.0.1 method=PATCH pkg=rhttp proto=HTTP/1.1 service=reva size=57 start="09/Jun/2020:04:42:04 +0000" status=500 time_ns=18832094 traceid=782ce4ece8b49b84ca985ce9532812af uri=/data/de7bb989-e845-4f36-ad0e-75b12add5b94 url=/data/de7bb989-e845-4f36-ad0e-75b12add5b94
ocis          | 2020-06-09T04:42:04Z ERR Could not start TUS upload error="unexpected status code: 500" pkg=rhttp service=reva traceid=70a2295a1ef12aa137196ccc025821a1
ocis          | 2020-06-09T04:42:04Z ERR http end="09/Jun/2020:04:42:04 +0000" host=127.0.0.1 method=PUT pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="09/Jun/2020:04:42:04 +0000" status=500 time_ns=109875011 traceid=70a2295a1ef12aa137196ccc025821a1 uri=/remote.php/dav/files/user0/newfile.txt url=/remote.php/dav/files/user0/newfile.txt
ocis          | 2020-06-09T04:42:25Z DBG Refreshing external service-registration service={"endpoints":[],"metadata":null,"name":"com.owncloud.reva","nodes":[{"address":"0.0.0.0:9142","id":"com.owncloud.reva-aaf759f1-8666-4084-a241-aca117f91dca","metadata":{"broker":"http","protocol":"grpc","registry":"mdns","server":"grpc","transport":"grpc"}}],"version":""}

individual-it · 2020-06-09T11:53:51Z

an other issue is the cleanup of files after deleting the user
@butonic any idea how we could get rid of all user files/folders when deleting the user?

butonic · 2020-06-10T08:06:04Z

to manipulate the tree in eos we need the eos binary. it does support eos rm -rf.
AFAIK @felixboehm and @jnweiger are planning to build a docker container that can be used to get the binary in any os.

for file up or download xrdcopy or xrdcp is needed, AFAICT it should be possible to wrap it in a docker contdainer like eos.

xrdcp example: xrdcp /tmp/myfile xroot://foo.bar.com//data/myfile

xrdcp and xrdcopy seem to do the same thing. maybe @labkode or @madsi1m can shed a light on this.

for direct http upload eos needs to be configured: http://eos-docs.web.cern.ch/eos-docs/configuration/http_tpc.html#http-s-support-and-http-tpc-with-token-authentication

maybe the aarnet docker containers already do that. IIRC at one point there was http appearing in the output of eos vid ls.

individual-it · 2020-06-15T04:56:16Z

deleting works fine with a command, upload could be solved by WebDav upload, see: owncloud/core#37525

individual-it · 2020-07-06T03:02:15Z

a lot of tests have been adjusted in owncloud/core#37575

API tests can be run with https://github.com/owncloud-docker/compose-playground/tree/master/examples/eos-compose-acceptance-tests

implement in CI

dpakach · 2020-07-22T03:52:02Z

The Ocis docker image fails with error

ocis          | make: Nothing to be done for `build'.
ocis          | bin/ocis: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by bin/ocis)

butonic · 2020-08-17T14:05:11Z

@dpakach try running make clean before starting the container, otherwise it will pick up the binary you complied, which might nor work inside the container.

dpakach · 2020-08-25T07:12:45Z

Setting up the CI setup with eos backend doesn't seem to be possible at the moment. The docker-compose setup in the ocis repo runs all the containers in the privileged mode which works fine in local setup, but in drone, we cannot run the containers in the privileged mode. So moving this one to blocked for now.
@butonic @felixboehm , If you know a way to run it without the privileged flag then please inform me.

PVince81 · 2020-08-27T10:03:48Z

please recheck with use eos driver for /eos namespace ocis#484 as we had to change some storage settings for EOS to make sharing work correctly

dpakach · 2020-09-16T05:45:51Z

A new POC for testing EOS in CI in owncloud/ocis#519
But this is blocked at the moment by owncloud/ocis#586
The docker-compose setup needs to be fully reliable to set it in CI.

phil-davis · 2022-05-19T05:16:41Z

@individual-it do you think that this will ever be able to happen?

What do we do about this issue?

individual-it · 2022-05-19T06:06:05Z

as per discussion with @wkloucek I'm closing this

there is some effort to get EOS tests running in reva [tests-only] EOS tests cs3org/reva#2318
if they would run there, there is no need to have them also here in ocis

individual-it closed this as completed Jun 5, 2020

individual-it reopened this Jun 5, 2020

dpakach self-assigned this Jul 7, 2020

PVince81 changed the title ~~run acceptance tests with EOS~~ [OCIS] run acceptance tests with EOS Aug 18, 2020

PVince81 transferred this issue from owncloud/ocis Aug 18, 2020

phil-davis added the QA-team label Sep 14, 2020

individual-it mentioned this issue Feb 1, 2021

expected failures on EOS storage get out of date owncloud/ocis#1549

Closed

exalate-issue-sync bot added the p3-medium label Mar 1, 2021

dpakach removed their assignment Mar 17, 2021

individual-it closed this as completed May 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OCIS] run acceptance tests with EOS #156

[OCIS] run acceptance tests with EOS #156

individual-it commented May 7, 2020

individual-it commented May 12, 2020

individual-it commented May 13, 2020

individual-it commented Jun 5, 2020

individual-it commented Jun 5, 2020

individual-it commented Jun 9, 2020

individual-it commented Jun 9, 2020

individual-it commented Jun 9, 2020

butonic commented Jun 10, 2020 •

edited

Loading

individual-it commented Jun 15, 2020

individual-it commented Jul 6, 2020

dpakach commented Jul 22, 2020

butonic commented Aug 17, 2020 •

edited

Loading

dpakach commented Aug 25, 2020

PVince81 commented Aug 27, 2020 •

edited

Loading

dpakach commented Sep 16, 2020

phil-davis commented May 19, 2022

individual-it commented May 19, 2022

[OCIS] run acceptance tests with EOS #156

[OCIS] run acceptance tests with EOS #156

Comments

individual-it commented May 7, 2020

individual-it commented May 12, 2020

individual-it commented May 13, 2020

individual-it commented Jun 5, 2020

individual-it commented Jun 5, 2020

individual-it commented Jun 9, 2020

individual-it commented Jun 9, 2020

individual-it commented Jun 9, 2020

butonic commented Jun 10, 2020 • edited Loading

individual-it commented Jun 15, 2020

individual-it commented Jul 6, 2020

dpakach commented Jul 22, 2020

butonic commented Aug 17, 2020 • edited Loading

dpakach commented Aug 25, 2020

PVince81 commented Aug 27, 2020 • edited Loading

dpakach commented Sep 16, 2020

phil-davis commented May 19, 2022

individual-it commented May 19, 2022

butonic commented Jun 10, 2020 •

edited

Loading

butonic commented Aug 17, 2020 •

edited

Loading

PVince81 commented Aug 27, 2020 •

edited

Loading