[full-ci] Store settings via metadata #3232

kobergj · 2022-02-24T14:18:41Z

Description

Encourages settings service to store its data via the metadata service

Related Issue

Fixes https://jira.owncloud.com/browse/OCIS-2346

Motivation and Context

How Has This Been Tested?

new unit tests
existing acceptance tests cover functionality

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Technical debt
Tests only (no source changes)

Checklist:

Code changes
Unit tests added
Acceptance tests added
Documentation ticket raised:

Signed-off-by: jkoberg <[email protected]>

ownclouders · 2022-02-25T16:41:35Z

💥 Acceptance test localApiTests-apiSpaces-ocis failed. Further test are cancelled...

Signed-off-by: jkoberg <[email protected]>

wkloucek

I couldn't get it working. Eg. as admin I am not allowed to add / remove users:

When using SETTINGS_STORE_TYPE=filesystem it works

settings/pkg/service/v0/service.go

kobergj · 2022-03-14T08:57:11Z

@wkloucek Creating users works fine for me. Currently it is necessary to set OCIS_URL envvar because of the IDP of the service user. Could you try if this is the issue?

C0rby · 2022-03-14T10:55:25Z

Yeah, it doesn't work on my machine either. I can login but I can't access the settings/accounts UI.
I get the following errors in my log:

2022-03-14T11:50:36+01:00 ERR error initializing metadata client error="error: permission denied: permission denied" service=ocis
2022-03-14T11:50:45+01:00 WRN Failed to execute error="{\"id\":\"com.owncloud.api.thumbnails\",\"code\":404,\"detail\":\"Unsupported file type\",\"status\":\"Not Found\"}" duration=3.591365 method=Thumbnails.GetThumbnail service=thumbnails
2022-03-14T11:51:04+01:00 ERR could not read link, skipping error="readlink /home/corby/.ocis/storage/users/spacetypes/personal/ddc2004c-0977-11eb-9d3f-a793888cd0f8.flock: no such file or directory" match=/home/corby/.ocis/storage/users/spacetypes/personal/ddc2004c-0977-11eb-9d3f-a793888cd0f8.flock pkg=rgrpc service=storage traceid=00000000000000000000000000000000
2022-03-14T11:51:16+01:00 ERR could not read link, skipping error="readlink /home/corby/.ocis/storage/users/spacetypes/personal/ddc2004c-0977-11eb-9d3f-a793888cd0f8.flock: no such file or directory" match=/home/corby/.ocis/storage/users/spacetypes/personal/ddc2004c-0977-11eb-9d3f-a793888cd0f8.flock pkg=rgrpc service=storage traceid=00000000000000000000000000000000
2022-03-14T11:52:15+01:00 ERR error initializing metadata client error="error: not found: stat: error: not found: f1bdd61a-da7c-49fc-8203-0558109d1b4f/settings" service=ocis
2022-03-14T11:52:21+01:00 ERR error initializing metadata client error="error: not found: stat: error: not found: f1bdd61a-da7c-49fc-8203-0558109d1b4f/settings" service=ocis
2022-03-14T11:52:29+01:00 ERR error initializing metadata client error="error: not found: stat: error: not found: f1bdd61a-da7c-49fc-8203-0558109d1b4f/settings" service=ocis

Started oCIS like this: OCIS_URL=https://localhost:9200 OCIS_INSECURE=true PROXY_ENABLE_BASIC_AUTH=true OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=t rue OCIS_LOG_LEVEL=warn bin/ocis server

kobergj · 2022-03-14T12:21:45Z

@C0rby this is happening when the settings folder could not be created properly. Could you remove ~/.ocis/storage/metadata/spaces/f1 and try again?

C0rby · 2022-03-14T12:28:38Z

Starting with a clean storage and the OCIS_URL works.

kobergj · 2022-03-14T12:39:40Z

Ok cool thanks 👍 Imho it is acceptable to have to remove metadata/spaces/f1 if it already exists (it shouldn't normally)

Should we do something about the OCIS_URL? I can hardcode the idp instead of taking it from env...

Signed-off-by: jkoberg <[email protected]>

need to review again

settings/pkg/config/defaults/defaultconfig.go

wkloucek · 2022-03-14T17:39:36Z

settings/pkg/config/defaults/defaultconfig.go

+		Metadata: config.Metadata{
+			GatewayAddress:    "127.0.0.1:9142",
+			StorageAddress:    "127.0.0.1:9215",
+			ServiceUserID:     "ddc2004c-0977-11eb-9d3f-a793888cd0f8",


should we really use the admin user here? why can't we use a system user like for the accounts service (95cb8724-03b2-11eb-a0a6-c33ef8ef53ad)?

The service user wasn't working properly before as it was creating 10k files on a simple acceptance test. It seems to be fixed though. Let's see what the pipeline says...

Well. Seems like it still doesn't work 😞

@wkloucek I found a way around it. Can you check again?

Adding and removing it again with each ListAccounts() call was a huge overhead. This is a temporary workaround, the whole service is gonna be replaced by the idm service soon anyway.

Signed-off-by: jkoberg <[email protected]>

sonarqubecloud · 2022-03-15T11:17:43Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

60.5% Coverage
9.1% Duplication

Merge: b0ea1de d54f75d Author: Willy Kloucek <[email protected]> Date: Tue Mar 15 21:49:58 2022 +0100 Merge pull request #3232 from kobergj/StoreSettingsViaMetadata [full-ci] Store settings via metadata

kobergj added 9 commits February 24, 2022 15:09

Settings: add metadata store

59527e1

Signed-off-by: jkoberg <[email protected]>

Merge branch 'master' into StoreSettingsViaMetadata

534e561

Settings: activate metadata store to see tests failing

0d4f8df

Signed-off-by: jkoberg <[email protected]>

add changelog

297a006

Signed-off-by: jkoberg <[email protected]>

Merge branch 'master' into StoreSettingsViaMetadata

25191fe

make assignment unittests green

a762e3c

Signed-off-by: jkoberg <[email protected]>

make bundle unit tests green

45c85f4

Signed-off-by: jkoberg <[email protected]>

make values unit tests green

7210266

Signed-off-by: jkoberg <[email protected]>

use cs3storage in metadatastorage

11a8b75

Signed-off-by: jkoberg <[email protected]>

kobergj added 6 commits February 26, 2022 13:18

lazy initialize metadataclient

2452d01

Signed-off-by: jkoberg <[email protected]>

delay creation of default roles

588a731

Signed-off-by: jkoberg <[email protected]>

return listaccounts permission when mdc is not yet set

e23448f

Signed-off-by: jkoberg <[email protected]>

add test for appendSetting

4c22c07

Signed-off-by: jkoberg <[email protected]>

rework initialization logic

492419e

Signed-off-by: jkoberg <[email protected]>

return default values when mdc is not initialized

a873fdc

Signed-off-by: jkoberg <[email protected]>

kobergj force-pushed the StoreSettingsViaMetadata branch from 3048563 to a873fdc Compare March 1, 2022 09:55

implement permission unit tests

7fcf6f5

Signed-off-by: jkoberg <[email protected]>

kobergj force-pushed the StoreSettingsViaMetadata branch from 1962eeb to 7fcf6f5 Compare March 1, 2022 11:15

refine initialization logic again

f642801

Signed-off-by: jkoberg <[email protected]>

kobergj force-pushed the StoreSettingsViaMetadata branch from af5c9f5 to f642801 Compare March 1, 2022 13:54

kobergj added 9 commits March 1, 2022 15:22

unitests for list values

a647f56

Signed-off-by: jkoberg <[email protected]>

ReadSetting unit test & implementation

c80022b

Signed-off-by: jkoberg <[email protected]>

create id for values if not given

7731de9

Signed-off-by: jkoberg <[email protected]>

Merge branch 'master' into StoreSettingsViaMetadata

fd78a66

add new defaults

62ec3cf

Signed-off-by: jkoberg <[email protected]>

downgrade reva temporarilly

0d49316

Signed-off-by: jkoberg <[email protected]>

use feature reva

1cde2fa

Signed-off-by: jkoberg <[email protected]>

Merge branch 'master' into StoreSettingsViaMetadata

085dcd9

make it configurable

15ac5d8

Signed-off-by: jkoberg <[email protected]>

kobergj marked this pull request as ready for review March 8, 2022 10:28

kobergj requested review from kulmann and pascalwengerter as code owners March 8, 2022 10:28

resolve conflicts

1a9d446

wkloucek force-pushed the StoreSettingsViaMetadata branch from fa8364b to 1a9d446 Compare March 11, 2022 15:24

wkloucek previously requested changes Mar 11, 2022

View reviewed changes

settings/pkg/service/v0/service.go Show resolved Hide resolved

Merge branch 'master' into StoreSettingsViaMetadata

2d4d90a

Signed-off-by: jkoberg <[email protected]>

kobergj force-pushed the StoreSettingsViaMetadata branch from e8c251b to 2d4d90a Compare March 14, 2022 12:48

Merge branch 'master' into StoreSettingsViaMetadata

65b3c97

wkloucek reviewed Mar 14, 2022

View reviewed changes

settings/pkg/config/defaults/defaultconfig.go Outdated Show resolved Hide resolved

wkloucek reviewed Mar 14, 2022

View reviewed changes

aduffeck and others added 2 commits March 15, 2022 10:07

Only add the service user to the index once (lazily)

edf8452

Adding and removing it again with each ListAccounts() call was a huge overhead. This is a temporary workaround, the whole service is gonna be replaced by the idm service soon anyway.

use service user / hardcode idp

b7c934b

Signed-off-by: jkoberg <[email protected]>

kobergj force-pushed the StoreSettingsViaMetadata branch from 6eeed53 to b7c934b Compare March 15, 2022 09:37

don't store service user data in metadataservice

d54f75d

Signed-off-by: jkoberg <[email protected]>

kobergj requested a review from wkloucek March 15, 2022 13:12

wkloucek approved these changes Mar 15, 2022

View reviewed changes

wkloucek merged commit 5a67a20 into owncloud:master Mar 15, 2022

wkloucek mentioned this pull request Mar 16, 2022

move settings persistence to metadata-storage owncloud/ocis-charts#21

Merged

kobergj deleted the StoreSettingsViaMetadata branch March 16, 2022 08:15

rhafer mentioned this pull request Mar 18, 2022

Setttings service (and share manager) require a hardcoded internal service user #3343

Closed

micbar mentioned this pull request Mar 29, 2022

Release 1.19.0 Tech Preview #3377

Closed

22 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[full-ci] Store settings via metadata #3232

[full-ci] Store settings via metadata #3232

kobergj commented Feb 24, 2022 •

edited

Loading

ownclouders commented Feb 25, 2022 •

edited

Loading

wkloucek left a comment •

edited

Loading

kobergj commented Mar 14, 2022

C0rby commented Mar 14, 2022

kobergj commented Mar 14, 2022

C0rby commented Mar 14, 2022

kobergj commented Mar 14, 2022

wkloucek Mar 14, 2022 •

edited

Loading

kobergj Mar 15, 2022

kobergj Mar 15, 2022

kobergj Mar 15, 2022

sonarqubecloud bot commented Mar 15, 2022

[full-ci] Store settings via metadata #3232

[full-ci] Store settings via metadata #3232

Conversation

kobergj commented Feb 24, 2022 • edited Loading

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

ownclouders commented Feb 25, 2022 • edited Loading

wkloucek left a comment • edited Loading

Choose a reason for hiding this comment

kobergj commented Mar 14, 2022

C0rby commented Mar 14, 2022

kobergj commented Mar 14, 2022

C0rby commented Mar 14, 2022

kobergj commented Mar 14, 2022

wkloucek Mar 14, 2022 • edited Loading

Choose a reason for hiding this comment

kobergj Mar 15, 2022

Choose a reason for hiding this comment

kobergj Mar 15, 2022

Choose a reason for hiding this comment

kobergj Mar 15, 2022

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 15, 2022

kobergj commented Feb 24, 2022 •

edited

Loading

ownclouders commented Feb 25, 2022 •

edited

Loading

wkloucek left a comment •

edited

Loading

wkloucek Mar 14, 2022 •

edited

Loading