next branch tracker for oCIS latest 5.0 compatiblity

[wkloucek]

• digest pin latest tag (so that one has reproducible deploys) ??? -> Renovate + CI!? -> digest pin done in improve next #410
• pullPolicy -> reverted in improve next #410

Chart changes

• bannedPasswordsList / FRONTEND_PASSWORD_POLICY_BANNED_PASSWORDS_LIST -> this is a path!? where comes the file from!? Probably we should hardcode FRONTEND_PASSWORD_POLICY_BANNED_PASSWORDS_LIST and only allow to set a ConfigMap reference to enable this list -> fixed in improve next #410
• why has the authservice /tmp volume? (from what I know this is only required for reva service to create pid files) -> it's a reve service and stores a pid file there
• prepare documentation PR for next branch changes
• clientlog service keeps restarting with exit code 2 -> fixed in fix clientlog restart loop #411
• why has the authservice environment variable AUTH_SERVICE_API_KEY that is not known to the documentation -> fix in remove non existing AUTH_SERVICE_API_KEY #413

Adapt to latest latest master

• fix USERLOG_DEBUG_ADDR in clientlog -> fixed in fix clientlog restart loop #411
• check if we can remove xxx_MACHINE_AUTH_API_KEY ( at least userlog seems to have this replaced by service accounts) -> fix in remove auth machine secret where no longer needed #412
• remove STORAGE_USERS_PURGE_TRASH_BIN_USER_ID use service accounts on next branch #390 (comment) -> see PR Remove unused STORAGE_USERS_PURGE_TRASH_BIN_USER_ID #405

Code Changes

• why has the authservice a environment variable AUTH_MACHINE_JWT_SECRET -> code fix in fix auth service env variable ocis#7523, fix in switch to AUTH_SERVICE_JWT_SECRET #415
• autoAcceptShares: true -> oCIS product defaults will also switch to true -> done in [full-ci] ApiTests. enable auto accepting in api tests ocis#7477

[wkloucek]

@kobergj maybe we can have a look at this together tomorrow?

[wkloucek]

@micbar @kobergj looks like the next branch is in a pretty good shape now. Do you have something that should be added / changed?

[wkloucek]

From my POV, only #414 is currently open for next. There seems to be something fishy.

[micbar]

autoAcceptShares: true -> oCIS product defaults will also switch to true

This would need service accounts.

[wkloucek]

autoAcceptShares: true -> oCIS product defaults will also switch to true

This would need service accounts.

We have them already in the oCIS Helm Chart on next branch. This is just a tracker to check if oCIS product goes to auto accept share defaulting to true (is currently not the case for the oCIS product, but for the Helm Chart). I'd prefer them to be in sync.

[kobergj]

Auto accept is already true on default in ocis since owncloud/ocis#7477

[wkloucek]

Auto accept is already true on default in ocis since owncloud/ocis#7477

Looks like https://github.com/owncloud/ocis/blob/master/changelog/unreleased/auto-accept-shares.md should be updated and changed to Change instead of Enhancement

[kobergj]

Quick change: owncloud/ocis#7569

[wkloucek]

Currently missing no next branch:

• switch to AUTH_SERVICE_JWT_SECRET #415
• mail config #414

[wkloucek]

Some more configuration options + more recent oCIS: #419

[wkloucek]

We should expose settings for:

• Allow configuring custom proxy routes ocis#7741

[wkloucek]

@micbar I lost a little track what we need to do in the oCIS Chart for a good oCIS 5 support. These are things that come to my mind:

• support nats-js-kv store / cache (might be forbidden right now)
• we have now different new / change cli commands
  • what changes are needed for the uploads cleanup?
  • do we need new cron jobs? eg. restarting postprocessing?

Do you have more things that come to your mind?

[micbar]

Nats ja cache will come with 5.0.0-beta.2.

@butonic can you provide the info about the new cleanup cli?

[micbar]

we have now different new / change cli commands

One change still pending owncloud/ocis#7933

what changes are needed for the uploads cleanup?

We basically have the new ocis storage-users uploads sessions command which should replace the cis storage-users uploads list

NAME:
   ocis storage-users uploads sessions - Print a list of upload sessions

USAGE:
   ocis storage-users uploads sessions [command options] [arguments...]

OPTIONS:
   --id value      filter sessions by upload session id
   --processing    filter sessions by processing status
   --expired       filter sessions by expired status
   --output value  output format to use (can be 'plain' or 'json',  experimental) (default: plain)
   --help, -h      show help

@butonic please provide more info for @wkloucek

Will that be available also on the stable-4.0?

do we need new cron jobs? eg. restarting postprocessing?

Custom Postprocessing Steps

By using the envvar POSTPROCESSING_STEPS, custom postprocessing steps can be added. Any word can be used as step name but be careful not to conflict with exising keywords like virusscan and delay. In addition, if a keyword is misspelled or the corresponding service does either not exist or does not follow the necessary event communication, the postprocessing service will wait forever getting the required response to proceed and does not continue any other processing.

Prerequisites

For using custom postprocessing steps you need a custom service listening to the configured event system (see General Prerequisites)

Workflow

When defining a custom postprocessing step (eg. "customstep"), the postprocessing service will eventually send an event during postprocessing. The event will be of type StartPostprocessingStep with its field StepToStart set to "customstep". When the service defined as custom step receives this event, it can safely execute its actions. The postprocessing service will wait until it has finished its work. The event contains further information (filename, executing user, size, ...) and also requires tokens and URLs to download the file in case byte inspection is necessary.

Once the service defined as custom step has finished its work, it should send an event of type PostprocessingFinished via the configured events system back to the postprocessing service. This event needs to contain a FinishedStep field set to "customstep". It also must contain the outcome of the step, which can be one of the following:

• delete: Abort postprocessing, delete the file.
• abort: Abort postprocessing, keep the file.
• retry: There was a problem that was most likely temporary and may be solved by trying again after some backoff duration. Retry runs automatically and is defined by the backoff behavior as described below.
• continue: Continue postprocessing, this is the success case.

The backoff behavior as mentioned in the retry outcome can be configured using the POSTPROCESSING_RETRY_BACKOFF_DURATION and POSTPROCESSING_MAX_RETRIES environment variables. The backoff duration is calculated using the following formula after each failure: backoff_duration = POSTPROCESSING_RETRY_BACKOFF_DURATION * 2^(number of failures - 1). This means that the time between the next round grows exponentially limited by the number of retries. Steps that still don't succeed after the maximum number of retries will be automatically moved to the abort state.

See the cs3 org for up-to-date information of reserved step names and event definitions.

[butonic]

The ocis storage-users uploads list and ocis storage-users uploads clean commands have been fixed. They can be used as is and now work as they were originally intended.

• list will show upload sessions that have not yet expired, including uploads that are in postprocessing
• clean will now only remove expired uploads that are not in postprocessing.

So any jobs periodically cleaning expired uploads using clean should now work as expected.

To give admins even more control over upload sessions we also added the ocis storage-users uploads sessions command. It can be used to filter upload sessions by id, processing state or expired status. The options reflect what is possible with the current interface. We can certainly add more filters or e.g. a --purge option that could then be used to kill a specific upload ... or multiple. I don't know if deleting all uploads that will expire before/after a given timestamp is a valuable thing.
For more robust integration with scripts the sessions command also allows rendering the list of upload sessions as JSON.

[wkloucek]

So any jobs periodically cleaning expired uploads using clean should now work as expected.

That's great, so we don't need to change the current one 🥳

[wkloucek]

@micbar I didn't get your part about postprocessing. Where can retry be configured? Is this supposed to go to ANTIVIRUS_INFECTED_FILE_HANDLING!? How does that help restarting the postprocessing pipelines?

[wkloucek]

Something seems to be broken for next #458

[wkloucek]

@micbar I didn't get your part about postprocessing. Where can retry be configured? Is this supposed to go to ANTIVIRUS_INFECTED_FILE_HANDLING!? How does that help restarting the postprocessing pipelines?

@micbar could you please clarify?

[kobergj]

No additional configuration needed. It will automatically retry when it hits an error. Compare https://github.com/owncloud/ocis/blob/master/services/antivirus/pkg/service/service.go#L178-L188

ANTIVIRUS_INFECTED_FILE_HANDLING only configures what to do with an infected file.