From fde614db5c157e51b0b5f15a2c4c454977bf0ccf Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Fri, 25 Oct 2024 11:23:43 +0200 Subject: [PATCH 01/28] Migration and Update Guide Changes --- .../deployment/services/env-var-changes.adoc | 4 +- .../ROOT/pages/migration/upgrading-ocis.adoc | 14 ++++++ .../migration/upgrading_5.0.x_7.0.0.adoc | 50 +++++++++++++++++++ 3 files changed, 66 insertions(+), 2 deletions(-) create mode 100644 modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc diff --git a/modules/ROOT/pages/deployment/services/env-var-changes.adoc b/modules/ROOT/pages/deployment/services/env-var-changes.adoc index c71766d26..71d4fbc8e 100644 --- a/modules/ROOT/pages/deployment/services/env-var-changes.adoc +++ b/modules/ROOT/pages/deployment/services/env-var-changes.adoc @@ -1,9 +1,9 @@ # Changed Environment Variables in Versions :toc: right -:description: This page contains tables with added and removed environment variables between Infinite Scale version 4.0.0 and 5.0.0. +:description: This page contains tables with added and removed environment variables between Infinite Scale version 5.0.x and 7.0.0. :source_path: {ocis_services_raw_url}{service_url_component}{ocis_services_final_path}adoc/env-var-deltas/ -// https://raw.githubusercontent.com/owncloud/ocis/docs-stable-5.0/services/_includes/adoc/env-var-deltas/ +// https://raw.githubusercontent.com/owncloud/ocis/docs-stable-7.0/services/_includes/adoc/env-var-deltas/ == Introduction diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index f52a4610b..9aae5b890 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -19,6 +19,20 @@ IMPORTANT: When upgrading from an older release to the desired one, *ALL* upgrad IMPORTANT: When upgrading from an older release to the desired one, mandatory configuration settings may have been added or removed. To see the changes required, you can run `ocis init --diff` after upgrading but before finally starting. For more details, see the xref:deployment/general/ocis-init.adoc[ocis init command] description. +== Version 5.0.x to 7.0.0 + +=== Notable Changes Requiring Manual Intervention + +* lets see what will come here + +=== Breaking Changes Requiring Manual Intervention + +* All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. + +=== Upgrade Steps + +For a detailed description of the steps to upgrade, see the xref:migration/upgrading_5.0.x_7.0.0.adoc[Upgrading from 5.0.x to 7.0.0] documentation. + == Version 4.0.0 to 5.0.0 === Notable Changes Requiring Manual Intervention diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc new file mode 100644 index 000000000..ad1a762fe --- /dev/null +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -0,0 +1,50 @@ += Upgrading from 5.0.x to 7.0.0 +:toc: right +:description: This document describes the necessary steps when upgrading Infinite Scale from release 5.0.x to 7.0.0. + +include::partial$multi-location/compose-version.adoc[] + +== Introduction + +{description} + +IMPORTANT: Read the important notes in the xref:migration/upgrading-ocis.adoc#introduction[Upgrading Infinite Scale] documentation before you start. + +IMPORTANT: Check below, if you are affected by breaking changes and prepare all steps mentioned before you start the upgrade. + +== Upgrade Steps + +. Shut down the Infinite Scale instance. +. Update Infinite Scale via: ++ +-- +* Docker ++ +[source,bash,subs="attributes+"] +---- +docker pull owncloud/ocis:7.0.0 +---- + +* or get the binary from {ocis-downloadpage-url}/stable/?sort=time&order=desc[download.owncloud.com,window=_blank]. +-- + +== Manage Breaking Changes + +* All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. + +=== How to Identify if You Are Affected + +* If you are using deprecated environment variables in your config. +** See the xref:deployment/services/env-var-changes.adoc[Added and Removed Environment Variables] documentation. + +In all other cases, there is nothing that needs to be done as the new defaults for client pool selectors will be used automatically. + +=== How to Manage the Change + +* Manage deprecated variables +** Environment variables that have been deprecated without successor can safely be removed from the configuration. These envvars do not harm as they are not used anymore. +** Environment variables that have been deprecated with a successor need to be updated accordingly. + +=== Deprecations + +* lets see what will come here From c848ba3422b253f15fc70eb3d4a57319e418108e Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Sun, 27 Oct 2024 16:32:07 +0100 Subject: [PATCH 02/28] overview update --- modules/ROOT/pages/migration/upgrading-ocis.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index 9aae5b890..df5e24c11 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -23,11 +23,13 @@ IMPORTANT: When upgrading from an older release to the desired one, mandatory co === Notable Changes Requiring Manual Intervention -* lets see what will come here +* Resharing has finally been removed. +* The maintenance command `ocis storage-users uploads list` has been removed. === Breaking Changes Requiring Manual Intervention * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. +* A new mandatory Infinite Scale config setting for the `activitylog` service named `service_account` has been added. If not adding this config, `activities` will not get handled. === Upgrade Steps From 44953e4a76d70418f1e2d2ff7e454f1cdc9f4c12 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Sun, 27 Oct 2024 16:32:29 +0100 Subject: [PATCH 03/28] ocis init --diff (not finished) --- .../migration/upgrading_5.0.x_7.0.0.adoc | 64 ++++++++++++++++++- 1 file changed, 61 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index ad1a762fe..386dc0011 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -14,30 +14,39 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all == Upgrade Steps -. Shut down the Infinite Scale instance. +. Shut down the Infinite Scale instance, *do not start it after upgrading*! . Update Infinite Scale via: + -- -* Docker +* Docker and docker compose + [source,bash,subs="attributes+"] ---- docker pull owncloud/ocis:7.0.0 ---- +** When using Docker Compose + +*** Update every compose file where the `ocis image` is referenced accordingly. + +*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:https://doc.owncloud.com/ocis/next/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section carefully. + * or get the binary from {ocis-downloadpage-url}/stable/?sort=time&order=desc[download.owncloud.com,window=_blank]. -- == Manage Breaking Changes * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. +* A new mandatory Infinite Scale config setting for the `activitylog` service named `service_account` has been added. === How to Identify if You Are Affected * If you are using deprecated environment variables in your config. ** See the xref:deployment/services/env-var-changes.adoc[Added and Removed Environment Variables] documentation. -In all other cases, there is nothing that needs to be done as the new defaults for client pool selectors will be used automatically. +* Because the `activitylog` service has been added, everyone who upgrades is affected. + +In all other cases, there is nothing that needs to be done. === How to Manage the Change @@ -45,6 +54,55 @@ In all other cases, there is nothing that needs to be done as the new defaults f ** Environment variables that have been deprecated without successor can safely be removed from the configuration. These envvars do not harm as they are not used anymore. ** Environment variables that have been deprecated with a successor need to be updated accordingly. +* Add new Infnite Scale config settings. + +To do so, the following steps need to be taken, here is the overview: ++ +-- +. Get the new binary/container, but do not start Infinite Scale +. Get the config diff via command +. Apply the diff created +-- ++ +See the detailed steps below. + +{empty} + +Update Config Settings:: +* If not already done, get the new Infinite Scale version as described in xref:upgrade-steps[Upgrade Steps] above. +* Run the `ocis init --diff` command: + +Because this will now run the updated ocis code, it will create in case of changes found two files: ++ +-- +. `ocis.config.patch` which contains all the changes that can be applied and +. `ocis.yaml..backup` containing a backup of the original config. +-- ++ +Depending on the deployment, you need to run this command differently: + +** *Binary deployment*: ++ +-- +[source,bash] +---- +ocis init --diff +---- +Change into the Infinite Scale config directory and apply the changes with: +[source,bash] +---- +patch < ocis.config.patch +---- +You can now start infinite Scale as usual. +-- + +** *Docker deployment*: ++ +-- +asasd +-- + + + + === Deprecations * lets see what will come here From bf615822b964ccd72ba412bc6d56c86a9ee667d2 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Sun, 27 Oct 2024 18:29:18 +0100 Subject: [PATCH 04/28] update details (breaking changes) --- .../migration/upgrading_5.0.x_7.0.0.adoc | 101 ++++++++++++++---- 1 file changed, 81 insertions(+), 20 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 386dc0011..83e295595 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -14,25 +14,35 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all == Upgrade Steps -. Shut down the Infinite Scale instance, *do not start it after upgrading*! -. Update Infinite Scale via: -+ --- -* Docker and docker compose +. Shut down the Infinite Scale instance and + +*do not start it after downloading the binary or image*! +. Download and install Infinite Scale +. Manage Breaking Changes +. Start Infinite Scale + +== Download and Install Infinite Scale + +Download and install Infinite Scale via: + +* *Binary* + +Follow the xref:depl-examples/minimal-bare-metal.adoc#installation[Installation] section of the bare metal deployment example. + +* *Docker and Docker Compose* + [source,bash,subs="attributes+"] ---- docker pull owncloud/ocis:7.0.0 ---- -** When using Docker Compose +** When using `docker run` -*** Update every compose file where the `ocis image` is referenced accordingly. +*** Update the image version used in the docker run command. -*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:https://doc.owncloud.com/ocis/next/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section carefully. +** When using `docker compose` -* or get the binary from {ocis-downloadpage-url}/stable/?sort=time&order=desc[download.owncloud.com,window=_blank]. --- +*** Update _every_ compose file where the `ocis image` is referenced accordingly. + +*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of that pages carefully. == Manage Breaking Changes @@ -58,51 +68,102 @@ In all other cases, there is nothing that needs to be done. To do so, the following steps need to be taken, here is the overview: + -- -. Get the new binary/container, but do not start Infinite Scale +. Get the new binary/image, but do not start Infinite Scale . Get the config diff via command . Apply the diff created -- + See the detailed steps below. +* You can now start Infinite Scale as usual + {empty} Update Config Settings:: -* If not already done, get the new Infinite Scale version as described in xref:upgrade-steps[Upgrade Steps] above. +* If not already done, get the new Infinite Scale version as described in xref:download-and-install-infinite-scale[Download and Install Infinite Scale]. + * Run the `ocis init --diff` command: + -Because this will now run the updated ocis code, it will create in case of changes found two files: +Because this will now run the new ocis code, it will create in case of changes found two files: + -- . `ocis.config.patch` which contains all the changes that can be applied and . `ocis.yaml..backup` containing a backup of the original config. -- + -Depending on the deployment, you need to run this command differently: +The diff option will NOT overwrite anything though questions will be asked. Consider using the correct setting for `certificate checking` as this will influence the respective diff output. ++ +Depending on the deployment chosen, you need to prepare for this command differently, assuming the location of the config directory is the default: -** *Binary deployment*: +** *Binary Deployment*: + -- [source,bash] ---- ocis init --diff ---- -Change into the Infinite Scale config directory and apply the changes with: +Change into the Infinite Scale config directory located at `$HOME/.ocis/config/` and apply the changes with: + [source,bash] ---- patch < ocis.config.patch ---- -You can now start infinite Scale as usual. -- -** *Docker deployment*: +** *Docker Deployment*: + -- -asasd +This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the mount/volume definition where the configuration settings are stored to make the `ocis init` command work. The bind mount from below is just an example and fails if the local directory does not exist! + +[source,bash] +---- +sudo docker run --rm -it \ + owncloud/ocis:7.0.0 \ + --mount type=bind,source=$PWD/ocis/ocis-config,target=/etc/ocis \ + sh +---- + +Then, when the containers shell is available, type: + +[source,bash] +---- +ocis init --diff +---- +Change into the containers config directory `/etc/ocis` and apply the changes with: + +[source,bash] +---- +patch < ocis.config.patch +---- +-- + +** *Docker Compose Deployment*: ++ -- +This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the volume definition where the configuration settings are stored defined in the compose file to make the `ocis init` command work. The named volume from below is just an example and uses the volume name from our docker compose deployment examples. + +[source,bash] +---- +sudo docker run --rm -it \ + owncloud/ocis:7.0.0 \ + -v ocis-config:/etc/ocis \ + sh +---- + +Then, when the containers shell is available, type: +[source,bash] +---- +ocis init --diff +---- +Change into the containers config directory `/etc/ocis` and apply the changes with: +[source,bash] +---- +patch < ocis.config.patch +---- +-- -=== Deprecations +== Deprecations * lets see what will come here From 7c6c884cda51eed67069aedd63f19fd9e8f88176 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 28 Oct 2024 10:21:10 +0100 Subject: [PATCH 05/28] updates --- .../migration/upgrading_5.0.x_7.0.0.adoc | 32 +++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 83e295595..d057a325a 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -14,9 +14,9 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all == Upgrade Steps -. Shut down the Infinite Scale instance and + -*do not start it after downloading the binary or image*! -. Download and install Infinite Scale +. Shut down the Infinite Scale instance +. Download and install Infinite Scale + +*Do not start it after downloading the binary or image*! . Manage Breaking Changes . Start Infinite Scale @@ -27,7 +27,7 @@ Download and install Infinite Scale via: * *Binary* + Follow the xref:depl-examples/minimal-bare-metal.adoc#installation[Installation] section of the bare metal deployment example. -* *Docker and Docker Compose* +* *Image Based Deployments* + [source,bash,subs="attributes+"] ---- @@ -35,15 +35,15 @@ docker pull owncloud/ocis:7.0.0 ---- ** When using `docker run` - *** Update the image version used in the docker run command. ** When using `docker compose` - *** Update _every_ compose file where the `ocis image` is referenced accordingly. - *** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of that pages carefully. +** When using `Kubernetes - Helm` +*** Update `{{ .Values.image.tag }}` and/or `{{ .Chart.AppVersion }}` accordingly. + == Manage Breaking Changes * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. @@ -54,9 +54,7 @@ docker pull owncloud/ocis:7.0.0 * If you are using deprecated environment variables in your config. ** See the xref:deployment/services/env-var-changes.adoc[Added and Removed Environment Variables] documentation. -* Because the `activitylog` service has been added, everyone who upgrades is affected. - -In all other cases, there is nothing that needs to be done. +* Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, everyone who upgrades is affected. === How to Manage the Change @@ -90,9 +88,9 @@ Because this will now run the new ocis code, it will create in case of changes f . `ocis.yaml..backup` containing a backup of the original config. -- + -The diff option will NOT overwrite anything though questions will be asked. Consider using the correct setting for `certificate checking` as this will influence the respective diff output. +The diff option will NOT overwrite anything though questions will be asked. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. + -Depending on the deployment chosen, you need to prepare for this command differently, assuming the location of the config directory is the default: +Depending on the deployment selected, you need to prepare for this command differently, assuming the location of the config directory is the default: ** *Binary Deployment*: + @@ -107,6 +105,7 @@ Change into the Infinite Scale config directory located at `$HOME/.ocis/config/` ---- patch < ocis.config.patch ---- +{empty} -- ** *Docker Deployment*: @@ -118,7 +117,7 @@ This will open a shell in a temporary container using the new Infinite Scale ima ---- sudo docker run --rm -it \ owncloud/ocis:7.0.0 \ - --mount type=bind,source=$PWD/ocis/ocis-config,target=/etc/ocis \ + --mount type=bind,source=$HOME/ocis/ocis-config,target=/etc/ocis \ sh ---- @@ -134,6 +133,7 @@ Change into the containers config directory `/etc/ocis` and apply the changes wi ---- patch < ocis.config.patch ---- +{empty} -- ** *Docker Compose Deployment*: @@ -161,8 +161,14 @@ Change into the containers config directory `/etc/ocis` and apply the changes wi ---- patch < ocis.config.patch ---- +{empty} -- +** *Any Other Image Based Deployment*: ++ +-- +Use one of the Docker based examples from above and derive from. +-- == Deprecations From 4c0da85119e076f871476f33d40ac1521c17e4ff Mon Sep 17 00:00:00 2001 From: Martin Date: Mon, 28 Oct 2024 10:28:21 +0100 Subject: [PATCH 06/28] Apply suggestions from code review Co-authored-by: Phil Davis --- modules/ROOT/pages/migration/upgrading-ocis.adoc | 2 +- modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index df5e24c11..9d8ae82e9 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -29,7 +29,7 @@ IMPORTANT: When upgrading from an older release to the desired one, mandatory co === Breaking Changes Requiring Manual Intervention * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. -* A new mandatory Infinite Scale config setting for the `activitylog` service named `service_account` has been added. If not adding this config, `activities` will not get handled. +* A new mandatory Infinite Scale config setting for the `activitylog` service named `service_account` has been added. If this config is not set, `activities` will not be recorded. === Upgrade Steps diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index d057a325a..18f1a73c9 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -39,7 +39,7 @@ docker pull owncloud/ocis:7.0.0 ** When using `docker compose` *** Update _every_ compose file where the `ocis image` is referenced accordingly. -*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of that pages carefully. +*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of those pages carefully. ** When using `Kubernetes - Helm` *** Update `{{ .Values.image.tag }}` and/or `{{ .Chart.AppVersion }}` accordingly. @@ -81,7 +81,7 @@ Update Config Settings:: * If not already done, get the new Infinite Scale version as described in xref:download-and-install-infinite-scale[Download and Install Infinite Scale]. * Run the `ocis init --diff` command: + -Because this will now run the new ocis code, it will create in case of changes found two files: +Because this will now run the new ocis code, it will create two files to help manage the needed changes: + -- . `ocis.config.patch` which contains all the changes that can be applied and From 6d68ea8ad7cf97c18432f6d2162efae685989675 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 28 Oct 2024 16:24:25 +0100 Subject: [PATCH 07/28] add changed cli commands --- .../deployment/general/general-info.adoc | 10 ++++++---- .../pages/maintenance/commands/commands.adoc | 4 ++-- .../migration/upgrading_5.0.x_7.0.0.adoc | 20 +++++++++++++++++++ 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/pages/deployment/general/general-info.adoc b/modules/ROOT/pages/deployment/general/general-info.adoc index 01aef11af..035487210 100644 --- a/modules/ROOT/pages/deployment/general/general-info.adoc +++ b/modules/ROOT/pages/deployment/general/general-info.adoc @@ -622,13 +622,15 @@ An admin user will be created when running the `ocis init` command with the foll Login to the webinterface with this admin user and change relevant data according your needs or create new users. As an example to reach out the webinterface use `\https://localhost:9200`. -=== Password Reset for the Admin User +=== Password Reset for IDM Users The admin password can be reset via command line if it has been forgotten and the admin can't enter the webUI anymore. Note that the admin user must already exist which happens if you have xref:deployment/general/ocis-init.adoc[initialized Infinite Scale]. -After running the respective command and entering a new password, the admin can relogin using the new password. +See the command below for details and options. + +After running the respective command and entering a new password, the admin or user can relogin using the new password. -Note that when Infinite Scale gets initialized with xref:deployment/general/ocis-init.adoc[The ocis init Command], an admin password is created and stored in the `ocis.yaml` file. The lifespan of this admin password is up to the point when it either gets changed in the webUI or via the resetpassword command. Any admin password changes are NOT written back to the `ocis.yaml` file nor manual changes to the `admin_password` are considered as a new password. +Note that when Infinite Scale gets initialized with xref:deployment/general/ocis-init.adoc[The ocis init Command], an admin password is created and stored in the `ocis.yaml` file. The lifespan of this admin password is up to the point when it either gets changed in the webUI or via the resetpassword command. Any admin password changes are *NOT* written back to the `ocis.yaml` file nor manual changes to the `admin_password` are considered as a new password. [source,yaml] ---- @@ -657,7 +659,7 @@ When the prerequisite from the note above is fulfilled, you can reset the admin ==== Binary Setup -Run the following command to reset the admin password: +Run the following command to reset the admin password, use `--user-name ` for ordinary users: [source,bash] ---- diff --git a/modules/ROOT/pages/maintenance/commands/commands.adoc b/modules/ROOT/pages/maintenance/commands/commands.adoc index efa90662e..fbbc7b4fd 100644 --- a/modules/ROOT/pages/maintenance/commands/commands.adoc +++ b/modules/ROOT/pages/maintenance/commands/commands.adoc @@ -48,9 +48,9 @@ Infinite Scale provides a xref:maintenance/commands/revisions-cleanup.adoc[CLI c Infinite Scale provides a xref:maintenance/commands/trash.adoc[CLI command] with which you can remove empty folders from the trashbin. -=== Reset the Admin Password +=== Reset Password for IDM Users -If you need to reset the admin password from the command line, follow the xref:deployment/general/general-info.adoc#password-reset-for-the-admin-user[Password Reset for the Admin User]. +If you need to reset the password for the admin or a user in the xref:{s-path}/idm.adoc[idm] service from the command line, follow the xref:deployment/general/general-info.adoc#password-reset-for-idm-users[Password Reset for IDM Users]. == Online Only Commands diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 18f1a73c9..839202e87 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -170,6 +170,26 @@ patch < ocis.config.patch Use one of the Docker based examples from above and derive from. -- +== Changed or Added CLI Commands + +See the link for a detailed description of the respective CLI command if available. If a command has been backported to version 5, always use the latest production version of that release. + +* xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + +The `ocis storage-users uploads` got restructured. + +The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. + +* xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + +The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. + +* A `health` command was added to each service: `ocis health`. + +* xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + +The `ocis trash purge-empty-dirs` command has been added. + +* xref:maintenance/commands/commands.adoc#revisions-cleanup[Revisions Cleanup] + +The `ocis revisions purge` command allows removing revisions of files in the storage. + +Note that this command has also been backported to version 5. + == Deprecations * lets see what will come here From 190cf52c84f43ba1f3188b709d39fcb6a26ce8f5 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 28 Oct 2024 16:57:53 +0100 Subject: [PATCH 08/28] more cli command updates --- .../ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 839202e87..786043fd7 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -174,6 +174,8 @@ Use one of the Docker based examples from above and derive from. See the link for a detailed description of the respective CLI command if available. If a command has been backported to version 5, always use the latest production version of that release. +* A `health` command was added to each service: `ocis health`. + * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. @@ -181,14 +183,13 @@ The deprecated `list` option is now removed, the `clean` option is now part of t * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. -* A `health` command was added to each service: `ocis health`. - * xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + -The `ocis trash purge-empty-dirs` command has been added. +The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. * xref:maintenance/commands/commands.adoc#revisions-cleanup[Revisions Cleanup] + -The `ocis revisions purge` command allows removing revisions of files in the storage. + -Note that this command has also been backported to version 5. +The `ocis revisions purge` command allows removing revisions of files in the storage. Note that this command has also been backported to version 5 available with its latest release. + +* The `ocis graph list-unified-roles` command simplifies the process of finding out which UID belongs to which role. Note that this command is described in the https://github.com/owncloud/ocis/tree/master/ocis#list-unified-roles[ocis repository, window=_blank] and has been added for completeness only. == Deprecations From afe3064e104a172701dbbe22e2591b446aba5396 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Tue, 29 Oct 2024 08:57:18 +0100 Subject: [PATCH 09/28] add searvice health --- .../pages/maintenance/commands/commands.adoc | 42 ++++++++++--------- .../maintenance/commands/service-health.adoc | 32 ++++++++++++++ .../pages/maintenance/commands/trash.adoc | 2 +- 3 files changed, 56 insertions(+), 20 deletions(-) create mode 100644 modules/ROOT/pages/maintenance/commands/service-health.adoc diff --git a/modules/ROOT/pages/maintenance/commands/commands.adoc b/modules/ROOT/pages/maintenance/commands/commands.adoc index fbbc7b4fd..204a955d6 100644 --- a/modules/ROOT/pages/maintenance/commands/commands.adoc +++ b/modules/ROOT/pages/maintenance/commands/commands.adoc @@ -40,22 +40,37 @@ Infinite Scale provides a xref:maintenance/commands/backup-consistency.adoc[CLI NOTE: This command MUST only be issued when Infinite Scale is offline and no services are running. This command will report false positives when issued during normal operation. +=== Reset Password for IDM Users + +If you need to reset the password for the admin or a user in the xref:{s-path}/idm.adoc[idm] service from the command line, follow the xref:deployment/general/general-info.adoc#password-reset-for-idm-users[Password Reset for IDM Users]. + === Revisions Cleanup Infinite Scale provides a xref:maintenance/commands/revisions-cleanup.adoc[CLI command] which allows removing revisions of files in the storage. -=== Trash Purge +=== Service Health -Infinite Scale provides a xref:maintenance/commands/trash.adoc[CLI command] with which you can remove empty folders from the trashbin. +Infinite Scale provides a xref:maintenance/commands/service-health.adoc[CLI command] checking the health of a service. -=== Reset Password for IDM Users +=== Trash Purge -If you need to reset the password for the admin or a user in the xref:{s-path}/idm.adoc[idm] service from the command line, follow the xref:deployment/general/general-info.adoc#password-reset-for-idm-users[Password Reset for IDM Users]. +Infinite Scale provides a xref:maintenance/commands/trash.adoc[CLI command] with which you can remove empty folders from the trashbin. == Online Only Commands Compared to offline commands that can be issued when Infinite Scale is shut down, online commands must be issued when Infinite Scale and the respective service(es) are running. +=== Inspect and Manipulate Node Metadata + +Infinite Scale provides a xref:maintenance/commands/node-metadata.adoc[CLI command] with which you can inspect and manipulate node metadata. + +WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. + +=== Inspect and Repair Node Tree Sizes + +Infinite Scale provides a xref:maintenance/commands/node-tree-size.adoc[CLI command] with which you can inspect and repair node tree sizes. This command can be necessary in very rare cases where spaces or files are shown in the Web UI with a size not matching reality. For the repair option Infinite Scale must be shut down. + +WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. === Manage Expired Uploads Compared to unfinished uploads which are handled by the system, managing expired uploads can be a necessity as those files can pile up, blocking storage resources and need to be removed by command regularly. See the xref:{s-path}/storage-users.adoc#manage-unfinished-uploads[Manage Unfinished Uploads] section at the _Storage Users_ service for details. @@ -68,30 +83,19 @@ This command is about purging old trash-bin items of `project` spaces (spaces th In rare circumstances, it can be necessary to initiate indexing manually for a given space a user has access to or all spaces. Though the search service handles exception cases automatically, it can happen that the search service was not able to complete indexing due to a dirty shut-down of the service or because of a bug. Re-indexing should *only* be run on explicit request and supervision by ownCloud support. See the xref:{s-path}/search.adoc#manually-trigger-re-indexing-spaces[Manually Trigger Re-Indexing Spaces] section at the _Search_ service for details. -=== Resume Post-Processing - -If post-processing fails in one step due to an unforeseen error, current uploads will not be retried automatically. A system administrator can instead run a xref:{s-path}/postprocessing.adoc#cli-commands[CLI Commands] to retry the failed upload. - -=== Inspect and Manipulate Node Metadata +=== Repair and Migrate jsoncs3 Indexes -Infinite Scale provides a xref:maintenance/commands/node-metadata.adoc[CLI command] with which you can inspect and manipulate node metadata. +A xref:maintenance/commands/rebuild-jsoncs3-indexes.adoc[CLI command] is provided to repair and migrate jsoncs3 indexes. In rare circumstances the data for shares from the "Shared with others" and "Shared with me" index can be corrupted though no data is lost. When using this command, you can recreate that index and migrate it to a new layout which fixes the issue. WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. -=== Inspect and Repair Node Tree Sizes +=== Resume Post-Processing -Infinite Scale provides a xref:maintenance/commands/node-tree-size.adoc[CLI command] with which you can inspect and repair node tree sizes. This command can be necessary in very rare cases where spaces or files are shown in the Web UI with a size not matching reality. For the repair option Infinite Scale must be shut down. +If post-processing fails in one step due to an unforeseen error, current uploads will not be retried automatically. A system administrator can instead run a xref:{s-path}/postprocessing.adoc#cli-commands[CLI Commands] to retry the failed upload. -WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. === Roll Back / Roll Forward Decomposedfs Migrations A xref:maintenance/commands/rolling-back-and-forward.adoc[CLI command] is provided to roll back or roll forward a decomposedfs migration. WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. - -=== Repair and Migrate jsoncs3 Indexes - -A xref:maintenance/commands/rebuild-jsoncs3-indexes.adoc[CLI command] is provided to repair and migrate jsoncs3 indexes. In rare circumstances the data for shares from the "Shared with others" and "Shared with me" index can be corrupted though no data is lost. When using this command, you can recreate that index and migrate it to a new layout which fixes the issue. - -WARNING: Use this command with absolute care. It is not intended to play around and should only be used on request and under supervision of ownCloud support. diff --git a/modules/ROOT/pages/maintenance/commands/service-health.adoc b/modules/ROOT/pages/maintenance/commands/service-health.adoc new file mode 100644 index 000000000..d53b02c88 --- /dev/null +++ b/modules/ROOT/pages/maintenance/commands/service-health.adoc @@ -0,0 +1,32 @@ += Service Health + +The service health CLI command allows checking the health status of a service. If there are no issues found, nothing health related will get printed. + +[source,bash] +---- +ocis health +---- + +Examples:: ++ +-- +* The `collaboration` service has been started but not configured and is therefore not in a healthy state: ++ +[source,bash] +---- +ocis collaboration health + +The WOPI secret has not been set properly in your config for collaboration. Make sure your /root/.ocis/config config contains the proper values (e.g. by running ocis init or setting it manually in the config/corresponding environment variable). +---- + +* The `antivirus` service has not been started, the health check responds accordingly. ++ +[source,bash] +---- +ocis antivirus health + +{"level":"fatal","service":"antivirus","error":"Get \"http://127.0.0.1:9277/healthz\": dial tcp 127.0.0.1:9277: connect: connection refused","time":"2024-10-28T17:47:54+01:00","message":"Failed to request health check"} +---- +-- + + diff --git a/modules/ROOT/pages/maintenance/commands/trash.adoc b/modules/ROOT/pages/maintenance/commands/trash.adoc index 9d0bd5cd8..565f43f9b 100644 --- a/modules/ROOT/pages/maintenance/commands/trash.adoc +++ b/modules/ROOT/pages/maintenance/commands/trash.adoc @@ -1,6 +1,6 @@ = Trash CLI -The trash cli command allows removing empty folders from the trashbin. This should be used to speed up trash bin operations. +The trash CLI command allows removing empty folders from the trashbin. This should be used to speed up trash bin operations. [source,bash] ---- From c84dd92d1e18920b7b518d7ad9137cc3387ef9d8 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Tue, 29 Oct 2024 09:00:53 +0100 Subject: [PATCH 10/28] link service health --- .../ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 786043fd7..2ae76144f 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -174,8 +174,6 @@ Use one of the Docker based examples from above and derive from. See the link for a detailed description of the respective CLI command if available. If a command has been backported to version 5, always use the latest production version of that release. -* A `health` command was added to each service: `ocis health`. - * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. @@ -183,12 +181,15 @@ The deprecated `list` option is now removed, the `clean` option is now part of t * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. -* xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + -The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. - * xref:maintenance/commands/commands.adoc#revisions-cleanup[Revisions Cleanup] + The `ocis revisions purge` command allows removing revisions of files in the storage. Note that this command has also been backported to version 5 available with its latest release. +* xref:maintenance/commands/commands.adoc#service-health[Service Health] + +A `health` command has been added to each service: `ocis health`. + +* xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + +The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. + * The `ocis graph list-unified-roles` command simplifies the process of finding out which UID belongs to which role. Note that this command is described in the https://github.com/owncloud/ocis/tree/master/ocis#list-unified-roles[ocis repository, window=_blank] and has been added for completeness only. == Deprecations From aa927dbda91fd757db279561455db7c52685bb1d Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Tue, 29 Oct 2024 13:23:10 +0100 Subject: [PATCH 11/28] text fixes --- modules/ROOT/pages/migration/upgrading-ocis.adoc | 2 +- modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index 9d8ae82e9..1d7534034 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -24,7 +24,7 @@ IMPORTANT: When upgrading from an older release to the desired one, mandatory co === Notable Changes Requiring Manual Intervention * Resharing has finally been removed. -* The maintenance command `ocis storage-users uploads list` has been removed. +* The maintenance command `ocis storage-users uploads list` has been removed. Its functionality is now part of other options. === Breaking Changes Requiring Manual Intervention diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 2ae76144f..334808485 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -172,7 +172,7 @@ Use one of the Docker based examples from above and derive from. == Changed or Added CLI Commands -See the link for a detailed description of the respective CLI command if available. If a command has been backported to version 5, always use the latest production version of that release. +See the link for a detailed description of the respective CLI command if available. * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + From 1ae7fc60fcb79896766bb40bf4e7c9f0023d88a7 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Wed, 30 Oct 2024 10:02:08 +0100 Subject: [PATCH 12/28] docker run changes --- .../migration/upgrading_5.0.x_7.0.0.adoc | 20 +++++++++++-------- .../ubuntu-compose/shared-setup.adoc | 2 +- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 334808485..5659c60ea 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -81,7 +81,7 @@ Update Config Settings:: * If not already done, get the new Infinite Scale version as described in xref:download-and-install-infinite-scale[Download and Install Infinite Scale]. * Run the `ocis init --diff` command: + -Because this will now run the new ocis code, it will create two files to help manage the needed changes: +Because this will now run the new ocis code, it will create two files to help manage the needed changes. Note that the patch file is only created if changes are found: + -- . `ocis.config.patch` which contains all the changes that can be applied and @@ -90,6 +90,8 @@ Because this will now run the new ocis code, it will create two files to help ma + The diff option will NOT overwrite anything though questions will be asked. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. + +If a patch file is created, do a review before applying it. ++ Depending on the deployment selected, you need to prepare for this command differently, assuming the location of the config directory is the default: ** *Binary Deployment*: @@ -115,10 +117,11 @@ This will open a shell in a temporary container using the new Infinite Scale ima [source,bash] ---- -sudo docker run --rm -it \ - owncloud/ocis:7.0.0 \ +sudo docker run \ + --rm -it \ + --entrypoint sh \ --mount type=bind,source=$HOME/ocis/ocis-config,target=/etc/ocis \ - sh + owncloud/ocis:7.0.0 ---- Then, when the containers shell is available, type: @@ -139,14 +142,15 @@ patch < ocis.config.patch ** *Docker Compose Deployment*: + -- -This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the volume definition where the configuration settings are stored defined in the compose file to make the `ocis init` command work. The named volume from below is just an example and uses the volume name from our docker compose deployment examples. +This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the volume definition (either the volume name or the local path) where the configuration settings are stored defined in the compose file to make the `ocis init` command work. The named volume `ocis-config` from below is just an example and derives from our docker compose deployment examples. [source,bash] ---- -sudo docker run --rm -it \ - owncloud/ocis:7.0.0 \ +sudo docker run \ + --rm -it \ + --entrypoint sh \ -v ocis-config:/etc/ocis \ - sh + owncloud/ocis:7.0.0 ---- Then, when the containers shell is available, type: diff --git a/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc b/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc index 59730b4d8..e078a1427 100644 --- a/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc +++ b/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc @@ -508,7 +508,7 @@ If no password can be identified, you must reset the admin password via the comm === Command Line Password Reset -To change the admin password from the command line, which you can do at any time, follow the guide described in xref:deployment/general/general-info.adoc#password-reset-for-the-admin-user[Password Reset for the Admin User]. +To change the admin password from the command line, which you can do at any time, follow the guide described in xref:deployment/general/general-info.adoc#password-reset-for-idm-users[Password Reset for IDM Users]. == Volume Migration From a3ed485e9ef48d3753214fb454cb0f5c910438f2 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Wed, 30 Oct 2024 12:06:41 +0100 Subject: [PATCH 13/28] move cli changes --- .../maintenance/commands/changed-cli.adoc | 29 +++++++++++++++++++ .../pages/maintenance/commands/commands.adoc | 4 +++ .../ROOT/pages/migration/upgrading-ocis.adoc | 2 +- .../migration/upgrading_5.0.x_7.0.0.adoc | 24 +-------------- 4 files changed, 35 insertions(+), 24 deletions(-) create mode 100644 modules/ROOT/pages/maintenance/commands/changed-cli.adoc diff --git a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc new file mode 100644 index 000000000..30d69ecba --- /dev/null +++ b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc @@ -0,0 +1,29 @@ += Changed or Added CLI Commands +:toc: right +:description: This page contains a list with added, changed or removed CLI commands between Infinite Scale version 5.0.x and 7.0.0. + +== Introduction + +{description} + +== Affected CLI Commands + +See the link for a detailed description of the respective CLI command if available. + +* xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + +The `ocis storage-users uploads` got restructured. + +The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. + +* xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + +The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. + +* xref:maintenance/commands/commands.adoc#revisions-cleanup[Revisions Cleanup] + +The `ocis revisions purge` command allows removing revisions of files in the storage. Note that this command has also been backported to version 5 available with its latest release. + +* xref:maintenance/commands/commands.adoc#service-health[Service Health] + +A `health` command has been added to each service: `ocis health`. + +* xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + +The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. + +* The `ocis graph list-unified-roles` command simplifies the process of finding out which UID belongs to which role. Note that this command is described in the https://github.com/owncloud/ocis/tree/master/ocis#list-unified-roles[ocis repository, window=_blank] and has been added for completeness only. diff --git a/modules/ROOT/pages/maintenance/commands/commands.adoc b/modules/ROOT/pages/maintenance/commands/commands.adoc index 204a955d6..be42fd085 100644 --- a/modules/ROOT/pages/maintenance/commands/commands.adoc +++ b/modules/ROOT/pages/maintenance/commands/commands.adoc @@ -16,6 +16,10 @@ This could happen if services are developed which are not shipped as part of the single binary. ==== +== Changed CLI Commands + +CLI commands can get added, changed or removed between versions. See the xref:maintenance/commands/changed-cli.adoc[Changed or Added CLI Commands] document for details. + == Offline Commands Compared to online commands that need to be issued during operation of Infinite Scale, offline commands can, and somtimes must only be issued if Infinite Scale is not running. diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index 1d7534034..8b0ba62b3 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -33,7 +33,7 @@ IMPORTANT: When upgrading from an older release to the desired one, mandatory co === Upgrade Steps -For a detailed description of the steps to upgrade, see the xref:migration/upgrading_5.0.x_7.0.0.adoc[Upgrading from 5.0.x to 7.0.0] documentation. +For a detailed description of the steps to upgrade, see the xref:migration/upgrading_5.0.x_7.0.0.adoc[Upgrading from 5.0.x to 7.0.0] documentation. Note that this document also contains references to added/changed/removed CLI commands and environment variables. == Version 4.0.0 to 5.0.0 diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 5659c60ea..d5658f904 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -176,26 +176,4 @@ Use one of the Docker based examples from above and derive from. == Changed or Added CLI Commands -See the link for a detailed description of the respective CLI command if available. - -* xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + -The `ocis storage-users uploads` got restructured. + -The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. - -* xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + -The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. - -* xref:maintenance/commands/commands.adoc#revisions-cleanup[Revisions Cleanup] + -The `ocis revisions purge` command allows removing revisions of files in the storage. Note that this command has also been backported to version 5 available with its latest release. - -* xref:maintenance/commands/commands.adoc#service-health[Service Health] + -A `health` command has been added to each service: `ocis health`. - -* xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + -The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. - -* The `ocis graph list-unified-roles` command simplifies the process of finding out which UID belongs to which role. Note that this command is described in the https://github.com/owncloud/ocis/tree/master/ocis#list-unified-roles[ocis repository, window=_blank] and has been added for completeness only. - -== Deprecations - -* lets see what will come here +See the xref:maintenance/commands/changed-cli.adoc[Changed or Added CLI Commands] document for details. From a273ca854d836e861272e549dc2ed86135ef80c1 Mon Sep 17 00:00:00 2001 From: Martin Date: Wed, 30 Oct 2024 14:14:40 +0100 Subject: [PATCH 14/28] Apply suggestions from code review Co-authored-by: Phil Davis --- modules/ROOT/pages/maintenance/commands/changed-cli.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc index 30d69ecba..7a39c78e5 100644 --- a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc +++ b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc @@ -12,7 +12,7 @@ See the link for a detailed description of the respective CLI command if availab * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + -The deprecated `list` option is now removed, the `clean` option is now part of the `sessons command`. +The deprecated `list` option is now removed, the `clean` option is now part of the `sessions command`. * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. @@ -24,6 +24,6 @@ The `ocis revisions purge` command allows removing revisions of files in the sto A `health` command has been added to each service: `ocis health`. * xref:maintenance/commands/commands.adoc#trash-purge[Trash Purge]. + -The `ocis trash purge-empty-dirs` command allows to remove empty folders from the trashbin. +The `ocis trash purge-empty-dirs` command allows removing empty folders from the trashbin. * The `ocis graph list-unified-roles` command simplifies the process of finding out which UID belongs to which role. Note that this command is described in the https://github.com/owncloud/ocis/tree/master/ocis#list-unified-roles[ocis repository, window=_blank] and has been added for completeness only. From e1e6e217ca75a55d56d724e6246e9c3448f417f9 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 4 Nov 2024 15:43:31 +0100 Subject: [PATCH 15/28] cli restart and resume --- .../services/s-list/postprocessing.adoc | 54 +++++++++++++++---- .../services/s-list/storage-users.adoc | 25 ++++++--- .../maintenance/commands/changed-cli.adoc | 8 +++ 3 files changed, 69 insertions(+), 18 deletions(-) diff --git a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc index c7923c52a..5af913c4b 100644 --- a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc @@ -63,39 +63,73 @@ The backoff behavior as mentioned in the `retry` outcome can be configured using === Resume Post-Processing -If post-processing fails in one step due to an unforeseen error, current uploads will not be retried automatically. A system administrator can instead run CLI commands to retry the failed upload which is a two step process. For details on the `storage-users` command see the xref:{s-path}/storage-users.adoc#manage-unfinished-uploads[Manage Unfinished Uploads] documentation. +[NOTE] +==== +If not noted otherwise, `commands with the `restart` option can also use the `resume` option. This changes behaviour slightly. -* First list ongoing upload sessions: +* `restart` + +When restarting an upload, all steps for open items will be restarted, except if otherwise defined. +* `resume` + +When resuming an upload, processing will continue for unfinished items from their last completed step. +==== + +If post-processing fails in one step due to an unforeseen error, current uploads will not be resumed automatically. A system administrator can instead run CLI commands to resume the failed upload manually which is at minimum a two step process. + +NOTE: For details on the `storage-users` command which provides many options see the xref:{s-path}/storage-users.adoc#manage-unfinished-uploads[Manage Unfinished Uploads] documentation. + +Depending if you want to restart/resume all or defined failed uploads, different commands are used. + +* First, list ongoing upload sessions to identify possibly failed ones. + +Note that there never can be a clear identification of a failed upload session due to various reasons causing them. You need to apply more critera like free space on disk, a failed service like antivirus etc. to declare an upload as failed. + [source,bash] ---- ocis storage-users uploads sessions ---- -* If you want to restart *all* uploads, just rerun the command with the `--restart` flag: +{empty} + +* *All failed uploads* + +-- +If you want to restart/resume all failed uploads, just rerun the command with the relevant flag. Note that this is the preferred command to handle failed processing steps: + [source,bash] ---- -ocis storage-users uploads sessions --restart +ocis storage-users uploads sessions --resume ---- +{empty} +-- -* If you want to restart *only one* upload, use the postprocessing restart command with the ID selected: +* *Particular failed uploads* + +Use the `postprocessing` command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, `resume` is an alias for `restart` to keep old functionality. `restart` is subject of change and will most likely be removed in a later version. + +** *Defined by ID* + +-- +If you want to resume only a specific upload, use the postprocessing restart command with the ID selected: + [source,bash] ---- -ocis postprocessing restart -u +ocis postprocessing resume -u ---- +{empty} +-- -Alternatively, instead of starting one specific upload, a system admin can also restart all uploads that are currently in a specific step. +** *Defined by step* ++ +-- +Alternatively, instead of restarting one specific upload, a system admin can also resume all uploads that are currently in a specific step. Examples: [source,bash] ---- -ocis postprocessing restart # Restarts all uploads where postprocessing is finished, but upload is not finished. -ocis postprocessing restart -s "finished" # Equivalent to the above. -ocis postprocessing restart -s "virusscan" # Restart all uploads currently in virusscan step. +ocis postprocessing resume # Restarts all uploads where postprocessing is finished, but upload is not finished. +ocis postprocessing resume -s "finished" # Equivalent to the above. +ocis postprocessing resume -s "virusscan" # Restart all uploads currently in virusscan step. ---- +-- == Storing diff --git a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc index f89163784..dee3c38ba 100644 --- a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc @@ -69,22 +69,30 @@ ocis storage-users uploads ---- COMMANDS: sessions Print a list of upload sessions - clean Clean up leftovers from expired uploads (deprecated, do not use it) - list Print a list of all incomplete uploads (deprecated, do not use it) ---- -- ==== Sessions command -The `sessions` command is the entry point for listing, restarting and cleaning unfinished uploads. +The `sessions` command is the entry point for listing, restarting/resuming and cleaning unfinished uploads. + +NOTE: There can never be a clear identification of a failed upload session due to various reasons causing them. You need to apply more critera like free space on disk, a failed service like antivirus etc. to declare an upload as failed. + +[NOTE] +==== +If not noted otherwise, `the `restart` option can also be replaced the `resume` option. This changes behaviour slightly. + +* `restart` + +When restarting an upload, all steps for open items will be restarted, except if otherwise defined. +* `resume` + +When resuming an upload, processing will continue for unfinished items from their last completed step. +==== [source,bash] ---- ocis storage-users uploads sessions ---- -Note, though possible, do not use the deprecated upload commands listed above. Their function has been embedded in the session command as command options. - [source,plaintext] ---- NAME: @@ -100,6 +108,7 @@ OPTIONS: --has-virus filter sessions by virus scan result (default: unset) --json output as json (default: false) --restart send restart event for all listed sessions (default: false) + --resume send resume event for all listed sessions (default: false) --clean remove uploads (default: false) --help, -h show help ---- @@ -165,7 +174,7 @@ ocis storage-users uploads sessions --expired=false --processing=false | |=== -The sessions command can also clear and restart uploads. The output is the same as if run without the `--clean` or `--restart` flag. Note that it is recommended to run the command first without the `--clean` (`--processing`) flag to double check which uploads get cleaned (restarted). +The sessions command can also clear and restart/resume uploads. The output is the same as if run without the `--clean` or `--restart/--resume` option. Note that it is recommended to run the command first without the `--clean` (`--processing`) option to double check which uploads would get cleaned (restarted/resumed). .Cleans all expired uploads regardless of processing and virus state. [source,bash] @@ -175,13 +184,13 @@ ocis storage-users uploads sessions \ --clean ---- -.Restarts all uploads that are processing and are not virus infected +.Resume all uploads that are currently processing and are not virus infected [source,bash] ---- ocis storage-users uploads sessions \ --processing=false \ --has-virus=false \ - --restart + --resume ---- === Manage Trash-Bin Items diff --git a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc index 7a39c78e5..2add33f6b 100644 --- a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc +++ b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc @@ -10,10 +10,18 @@ See the link for a detailed description of the respective CLI command if available. +* xref:maintenance/commands/commands.adoc#manage-expired-uploads[Manage Expired Uploads] + +The `ocis storage-users uploads sessions --restart` command got an alternative `--resume` option. + +Resume can be used in the same way as `restart` with a slighly different behavior change. + * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + The deprecated `list` option is now removed, the `clean` option is now part of the `sessions command`. +* xref:maintenance/commands/commands.adoc#resume-post-processing[Resume Post-Processing] + +The `ocis postprocessing restart` command got an alternative `resume` option. + +Resume can be used in the same way as `restart` with a slighly different behavior change. + * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. From 23702cc20f7f9bbb6dc3e6df4e2c41d80a82abac Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 4 Nov 2024 16:53:47 +0100 Subject: [PATCH 16/28] small fixes --- .../deployment/services/s-list/postprocessing.adoc | 12 ++++++------ .../deployment/services/s-list/storage-users.adoc | 4 ++-- .../ROOT/pages/maintenance/commands/changed-cli.adoc | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc index 5af913c4b..494bb95c8 100644 --- a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc @@ -65,12 +65,12 @@ The backoff behavior as mentioned in the `retry` outcome can be configured using [NOTE] ==== -If not noted otherwise, `commands with the `restart` option can also use the `resume` option. This changes behaviour slightly. +If not noted otherwise, commands with the `restart` option can also use the `resume` option. This changes behaviour slightly. * `restart` + When restarting an upload, all steps for open items will be restarted, except if otherwise defined. * `resume` + -When resuming an upload, processing will continue for unfinished items from their last completed step. +When resuming an upload, processing will continue unfinished items from their last completed step. ==== If post-processing fails in one step due to an unforeseen error, current uploads will not be resumed automatically. A system administrator can instead run CLI commands to resume the failed upload manually which is at minimum a two step process. @@ -102,12 +102,12 @@ ocis storage-users uploads sessions --resume -- * *Particular failed uploads* + -Use the `postprocessing` command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, `resume` is an alias for `restart` to keep old functionality. `restart` is subject of change and will most likely be removed in a later version. +Use the `postprocessing` command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, `resume` is an alias for `restart` to keep old functionality. `restart` is subject of change and will most likely be removed in a later version. ** *Defined by ID* + -- -If you want to resume only a specific upload, use the postprocessing restart command with the ID selected: +If you want to resume only a specific upload, use the postprocessing resume command with the ID selected: [source,bash] ---- @@ -125,9 +125,9 @@ Examples: [source,bash] ---- -ocis postprocessing resume # Restarts all uploads where postprocessing is finished, but upload is not finished. +ocis postprocessing resume # Resumes all uploads where postprocessing is finished, but upload is not finished. ocis postprocessing resume -s "finished" # Equivalent to the above. -ocis postprocessing resume -s "virusscan" # Restart all uploads currently in virusscan step. +ocis postprocessing resume -s "virusscan" # Resume all uploads currently in virusscan step. ---- -- diff --git a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc index dee3c38ba..3f8648e0b 100644 --- a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc @@ -80,12 +80,12 @@ NOTE: There can never be a clear identification of a failed upload session due t [NOTE] ==== -If not noted otherwise, `the `restart` option can also be replaced the `resume` option. This changes behaviour slightly. +If not noted otherwise, the `restart` option can also be replaced the `resume` option. This changes behaviour slightly. * `restart` + When restarting an upload, all steps for open items will be restarted, except if otherwise defined. * `resume` + -When resuming an upload, processing will continue for unfinished items from their last completed step. +When resuming an upload, processing will continue unfinished items from their last completed step. ==== [source,bash] diff --git a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc index 2add33f6b..fcbfadfe8 100644 --- a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc +++ b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc @@ -12,7 +12,7 @@ See the link for a detailed description of the respective CLI command if availab * xref:maintenance/commands/commands.adoc#manage-expired-uploads[Manage Expired Uploads] + The `ocis storage-users uploads sessions --restart` command got an alternative `--resume` option. + -Resume can be used in the same way as `restart` with a slighly different behavior change. +Resume can be used in the same way as restart with a slighly different behavior change. * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + @@ -20,7 +20,7 @@ The deprecated `list` option is now removed, the `clean` option is now part of t * xref:maintenance/commands/commands.adoc#resume-post-processing[Resume Post-Processing] + The `ocis postprocessing restart` command got an alternative `resume` option. + -Resume can be used in the same way as `restart` with a slighly different behavior change. +Resume can be used in the same way as restart with a slighly different behavior change. * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. From 42c9cbbccf0207b179a91d8041153c09257c1974 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Thu, 7 Nov 2024 14:34:25 +0100 Subject: [PATCH 17/28] update envvars changes --- antora.yml | 2 +- modules/ROOT/pages/deployment/services/env-var-changes.adoc | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/antora.yml b/antora.yml index 7da477b48..63a9ee1a5 100644 --- a/antora.yml +++ b/antora.yml @@ -48,7 +48,7 @@ asciidoc: # this is the first part of the name for envvars between major versions that will be added or removed # example for full name: 4.0.0-5.0.0-added.adoc or 4.0.0-5.0.0-removed.adoc - env_var_delta_name: '4.0.0-5.0.0' + env_var_delta_name: '5.0.0-7.0.0' # helm_tab_x will be used to assemble the url (tag) accessing the raw content for helm charts (tag) # note that tab 2 always contains the actual release and tab 3 the former diff --git a/modules/ROOT/pages/deployment/services/env-var-changes.adoc b/modules/ROOT/pages/deployment/services/env-var-changes.adoc index 71d4fbc8e..4298abaca 100644 --- a/modules/ROOT/pages/deployment/services/env-var-changes.adoc +++ b/modules/ROOT/pages/deployment/services/env-var-changes.adoc @@ -23,4 +23,9 @@ Removed:: -- include::{source_path}{env_var_delta_name}-removed.adoc[] -- +Deprecated:: ++ +-- +include::{source_path}{env_var_delta_name}-deprecated.adoc[] +-- ==== From 210f6906ec3ee04327b0b08d6dd8d46bd0a13d46 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Fri, 8 Nov 2024 12:00:43 +0100 Subject: [PATCH 18/28] numbered sections (unfinished) --- .../migration/upgrading_5.0.x_7.0.0.adoc | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index d5658f904..f7683aa8c 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -14,12 +14,14 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all == Upgrade Steps -. Shut down the Infinite Scale instance . Download and install Infinite Scale + *Do not start it after downloading the binary or image*! +. Shut down the Infinite Scale instance . Manage Breaking Changes . Start Infinite Scale +:sectnums: + == Download and Install Infinite Scale Download and install Infinite Scale via: @@ -44,6 +46,19 @@ docker pull owncloud/ocis:7.0.0 ** When using `Kubernetes - Helm` *** Update `{{ .Values.image.tag }}` and/or `{{ .Chart.AppVersion }}` accordingly. +== Shut Down the Infinite Scale Instance + +Depending how you deployed Infinite Scale, you need to shut it down differently. + +* For binary deployments, do a graceful shutdown as described in xref:depl-examples/minimal-bare-metal.adoc#stopping-infinite-scale[Stopping Infinite Scale]. + +* For depoyments using `docker run` do a graceful shutdown as described in xref:depl-examples/container-setup.adoc#stop-a-running-container[Stop a Running Container]. + +* For deployments using `docker compose` do a graceful shutdown as described in xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#stop-the-deployment[Stop the Deployment]. + +* For any other image based deployment, shut down Infinite Scale according the vendors deployment description. + + == Manage Breaking Changes * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. @@ -174,6 +189,10 @@ patch < ocis.config.patch Use one of the Docker based examples from above and derive from. -- +== Start Infinite Scale + +:sectnums!: + == Changed or Added CLI Commands See the xref:maintenance/commands/changed-cli.adoc[Changed or Added CLI Commands] document for details. From e12938c019cc139a690db1a8993e6bbab888bd8e Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Fri, 8 Nov 2024 15:36:14 +0100 Subject: [PATCH 19/28] numbered sections (finished) --- .../migration/upgrading_5.0.x_7.0.0.adoc | 112 +++++++++++------- 1 file changed, 66 insertions(+), 46 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index f7683aa8c..07c9d2e26 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -17,6 +17,8 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all . Download and install Infinite Scale + *Do not start it after downloading the binary or image*! . Shut down the Infinite Scale instance +. We strongly recommend doing a backup +. Reconfigure the deployment . Manage Breaking Changes . Start Infinite Scale @@ -26,38 +28,54 @@ IMPORTANT: Check below, if you are affected by breaking changes and prepare all Download and install Infinite Scale via: -* *Binary* + +=== Binary + Follow the xref:depl-examples/minimal-bare-metal.adoc#installation[Installation] section of the bare metal deployment example. -* *Image Based Deployments* +=== Image Based Deployments + +* Issue the following command to download the new image: + [source,bash,subs="attributes+"] ---- docker pull owncloud/ocis:7.0.0 ---- -** When using `docker run` -*** Update the image version used in the docker run command. +== Shut Down the Infinite Scale Instance + +Depending how you deployed Infinite Scale, you need to shut it down differently. -** When using `docker compose` -*** Update _every_ compose file where the `ocis image` is referenced accordingly. -*** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of those pages carefully. +* *Binary* + +For binary deployments, do a graceful shutdown as described in xref:depl-examples/minimal-bare-metal.adoc#stopping-infinite-scale[Stopping Infinite Scale]. -** When using `Kubernetes - Helm` -*** Update `{{ .Values.image.tag }}` and/or `{{ .Chart.AppVersion }}` accordingly. +* *docker run* + +For depoyments using `docker run` do a graceful shutdown as described in xref:depl-examples/container-setup.adoc#stop-a-running-container[Stop a Running Container]. -== Shut Down the Infinite Scale Instance +* *docker compose* + +For deployments using `docker compose` do a graceful shutdown as described in xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#stop-the-deployment[Stop the Deployment]. -Depending how you deployed Infinite Scale, you need to shut it down differently. +* *Any other image based deployment* + +For any other image based deployment, shut down Infinite Scale according the vendors deployment description. + +== Backup of Infinite Scale -* For binary deployments, do a graceful shutdown as described in xref:depl-examples/minimal-bare-metal.adoc#stopping-infinite-scale[Stopping Infinite Scale]. +See the xref:maintenance/b-r/backup_considerations.adoc[Backup Considerations] and the xref:maintenance/b-r/backup.adoc[Backup] documentation for more details. -* For depoyments using `docker run` do a graceful shutdown as described in xref:depl-examples/container-setup.adoc#stop-a-running-container[Stop a Running Container]. +== Reconfigure the Deployment -* For deployments using `docker compose` do a graceful shutdown as described in xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#stop-the-deployment[Stop the Deployment]. +Reconfigure the deployment to use the new image: -* For any other image based deployment, shut down Infinite Scale according the vendors deployment description. +* For binary, nothing extra needs to be done +* When using `docker run` +** Update the image version used in the docker run command. + +* When using `docker compose` +** Update _every_ compose file where the `ocis image` is referenced accordingly. +** If you have used the deployment examples based on _rolling releases version 6.x_ either for xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner], read the *Updating* section of those pages carefully. + +* When using `Kubernetes - Helm` +** Update `{{ .Values.image.tag }}` and/or `{{ .Chart.AppVersion }}` accordingly. == Manage Breaking Changes @@ -66,50 +84,47 @@ Depending how you deployed Infinite Scale, you need to shut it down differently. === How to Identify if You Are Affected -* If you are using deprecated environment variables in your config. +* If you are using removed or deprecated environment variables in your config. ** See the xref:deployment/services/env-var-changes.adoc[Added and Removed Environment Variables] documentation. -* Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, everyone who upgrades is affected. +* Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, _everyone_ who upgrades is affected. === How to Manage the Change -* Manage deprecated variables -** Environment variables that have been deprecated without successor can safely be removed from the configuration. These envvars do not harm as they are not used anymore. -** Environment variables that have been deprecated with a successor need to be updated accordingly. +==== Manage deprecated variables + +* Environment variables that have been deprecated without successor can safely be removed from the configuration. These envvars do not harm as they are not used anymore. +* Environment variables that have been deprecated with a successor need to be updated accordingly. +* Environment variables that habe been removed from Infinite Scale can safely be removed from the config if set manually. + +==== Add new Infnite Scale config settings + +To do so, the following steps need to be taken, here is the *overview*: -* Add new Infnite Scale config settings. + -To do so, the following steps need to be taken, here is the overview: -+ --- -. Get the new binary/image, but do not start Infinite Scale . Get the config diff via command . Apply the diff created --- -+ -See the detailed steps below. +. Start Infinite Scale as usual -* You can now start Infinite Scale as usual +See the next section for *details* how to update the config settings. -{empty} +=== Update Config Settings -Update Config Settings:: -* If not already done, get the new Infinite Scale version as described in xref:download-and-install-infinite-scale[Download and Install Infinite Scale]. +Run the `ocis init --diff` command: -* Run the `ocis init --diff` command: + Because this will now run the new ocis code, it will create two files to help manage the needed changes. Note that the patch file is only created if changes are found: -+ --- + . `ocis.config.patch` which contains all the changes that can be applied and . `ocis.yaml..backup` containing a backup of the original config. --- -+ + +Note that after applying the patch file, the created files by the `--diff` option must be deleted manually. + The diff option will NOT overwrite anything though questions will be asked. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. -+ + If a patch file is created, do a review before applying it. -+ + Depending on the deployment selected, you need to prepare for this command differently, assuming the location of the config directory is the default: -** *Binary Deployment*: +* *Binary Deployment*: + -- [source,bash] @@ -122,10 +137,11 @@ Change into the Infinite Scale config directory located at `$HOME/.ocis/config/` ---- patch < ocis.config.patch ---- -{empty} -- -** *Docker Deployment*: +{empty} + +* *Docker Deployment*: + -- This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the mount/volume definition where the configuration settings are stored to make the `ocis init` command work. The bind mount from below is just an example and fails if the local directory does not exist! @@ -151,10 +167,11 @@ Change into the containers config directory `/etc/ocis` and apply the changes wi ---- patch < ocis.config.patch ---- -{empty} -- -** *Docker Compose Deployment*: +{empty} + +* *Docker Compose Deployment*: + -- This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the volume definition (either the volume name or the local path) where the configuration settings are stored defined in the compose file to make the `ocis init` command work. The named volume `ocis-config` from below is just an example and derives from our docker compose deployment examples. @@ -180,10 +197,11 @@ Change into the containers config directory `/etc/ocis` and apply the changes wi ---- patch < ocis.config.patch ---- -{empty} -- -** *Any Other Image Based Deployment*: +{empty} + +* *Any Other Image Based Deployment*: + -- Use one of the Docker based examples from above and derive from. @@ -191,6 +209,8 @@ Use one of the Docker based examples from above and derive from. == Start Infinite Scale +When you have finished upgrading, you can now start infinite Scale as ususal. + :sectnums!: == Changed or Added CLI Commands From 47a56e1e9d9227eef341573d44c6bb866f560af4 Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 11 Nov 2024 17:30:12 +0100 Subject: [PATCH 20/28] add note if patch command is missing --- modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 07c9d2e26..7686f8dd7 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -122,6 +122,8 @@ The diff option will NOT overwrite anything though questions will be asked. Cons If a patch file is created, do a review before applying it. +NOTE: If the `patch` command is not present in your deployment, you must apply the changes provided by the patch file manually to the `ocis.yaml` file. + Depending on the deployment selected, you need to prepare for this command differently, assuming the location of the config directory is the default: * *Binary Deployment*: From dc8aa17090db7e6fa1388b735e2e133aa7d491df Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Wed, 13 Nov 2024 12:43:23 +0100 Subject: [PATCH 21/28] add collaboration related stuff --- .../migration/upgrading_5.0.x_7.0.0.adoc | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 7686f8dd7..309fe949a 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -81,6 +81,8 @@ Reconfigure the deployment to use the new image: * All environment variables that were marked for deprecation in Infinite Scale release 5.0.x have finally been removed. * A new mandatory Infinite Scale config setting for the `activitylog` service named `service_account` has been added. +* A new mandatory Infinite Scale config setting for the `collaboration` service named `secret` has been added. +* Deployment examples based on one of the `Ubuntu with Docker Compose` deployments have been updated/upgraded. === How to Identify if You Are Affected @@ -89,6 +91,12 @@ Reconfigure the deployment to use the new image: * Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, _everyone_ who upgrades is affected. +* Because the xref:{s-path}/collaboration.adoc[collaboration] service has been added and is started automatically, _everyone_ who upgrades is affected. + +** If you have used web office document servers such as Collabora, ONLYOFFICE or Microsoft using the WOPI protocol with the external https://github.com/cs3org/wopiserver[cs3org/wopi] server, we highly recommend switching to the xref:{s-path}/collaboration.adoc[collaboration] service which requires deployment reconfiguration which is a breaking change. Though the environment variables from the xref:{s-path}/app-provider.adoc[app-provider] service formerly used for web office document deployments are currently not deprecated, the are a candidate for future deprecations. + +* If you have used a rolling release and deployed web office document servers like the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or the xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner] deployment example, you should follow the reconfiguration section below because the deployment files have been updated/upgraded. + === How to Manage the Change ==== Manage deprecated variables @@ -209,9 +217,24 @@ patch < ocis.config.patch Use one of the Docker based examples from above and derive from. -- +=== Reconfigure web Office Document Deployments + +With the introduction of the xref:{s-path}/collaboration.adoc[collaboration] service, the external https://github.com/cs3org/wopiserver[cs3org/wopi] server is no longer required. Deployments using the external cs3org/wopi server should upgrade. This upgrade is a breaking change, but if you have used one of the `Ubuntu with Docker Compose` deployment examples, a re-deployment is quite easy. If you have setup your own deployment not based on the examples, you can derive from the changes based on the deployment files. + +The following steps are based on the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] deployment example. The steps are identical for the xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner]. + +* Backup the the base folder containing the existing deployment example by renaming it. + +You will need your configuration details with the new example. + +* Follow the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#download-and-transfer-the-example[Download and Transfer the Example] to get the new deployment and extract it as described in the following section of the guide. + +* Reconfigure the new `.env` file based on settings made in the `.env` file of the backup. + == Start Infinite Scale -When you have finished upgrading, you can now start infinite Scale as ususal. +When you have finished upgrading, you now can start Infinite Scale as ususal. + +For any deployment used, you now can delete/remove old binaries or images/containers. :sectnums!: From 65c8f38b37ffa8f82b13c5f55a2b6fa8e3c808ea Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Wed, 13 Nov 2024 20:40:33 +0100 Subject: [PATCH 22/28] fix collaboration reason --- modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 309fe949a..62c1e786f 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -91,7 +91,7 @@ Reconfigure the deployment to use the new image: * Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, _everyone_ who upgrades is affected. -* Because the xref:{s-path}/collaboration.adoc[collaboration] service has been added and is started automatically, _everyone_ who upgrades is affected. +* The xref:{s-path}/collaboration.adoc[collaboration] service has been added. Though this service needs to be started manually, it requires a secret (preferred a random UUID string) when started. For safety reasons on a fresh install, this secret is added when running the `ocis init` command and should be added on upgrades. The used upgrade procedure provides a secret if not present in the config. Note that the secret can be overwritten in the config or with an environment variable if necessary. ** If you have used web office document servers such as Collabora, ONLYOFFICE or Microsoft using the WOPI protocol with the external https://github.com/cs3org/wopiserver[cs3org/wopi] server, we highly recommend switching to the xref:{s-path}/collaboration.adoc[collaboration] service which requires deployment reconfiguration which is a breaking change. Though the environment variables from the xref:{s-path}/app-provider.adoc[app-provider] service formerly used for web office document deployments are currently not deprecated, the are a candidate for future deprecations. From 5dcbb20bbc2921bfd073367e963f89b30e99195b Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Thu, 14 Nov 2024 14:38:55 +0100 Subject: [PATCH 23/28] fixes service health text --- modules/ROOT/pages/maintenance/commands/service-health.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/maintenance/commands/service-health.adoc b/modules/ROOT/pages/maintenance/commands/service-health.adoc index d53b02c88..bad425eda 100644 --- a/modules/ROOT/pages/maintenance/commands/service-health.adoc +++ b/modules/ROOT/pages/maintenance/commands/service-health.adoc @@ -16,7 +16,7 @@ Examples:: ---- ocis collaboration health -The WOPI secret has not been set properly in your config for collaboration. Make sure your /root/.ocis/config config contains the proper values (e.g. by running ocis init or setting it manually in the config/corresponding environment variable). +The WOPI secret has not been set properly in your config for collaboration. Make sure your /root/.ocis/config config contains the proper values (e.g. by using 'ocis init --diff' and applying the patch or setting a value manually in the config/corresponding environment variable). ---- * The `antivirus` service has not been started, the health check responds accordingly. From 7a8ad0343d8a7240812e98c2d868d0bb9744bed8 Mon Sep 17 00:00:00 2001 From: Juergen Weigert Date: Thu, 14 Nov 2024 16:47:51 +0100 Subject: [PATCH 24/28] docs: avoid sloppy language. Throu -> trough; dangling adverb 'first' moved to its verb. --- .../pages/deployment/services/s-list/storage-users.adoc | 2 +- modules/ROOT/pages/migration/upgrading-ocis.adoc | 6 +++--- .../partials/depl-examples/ubuntu-compose/shared-setup.adoc | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc index 3f8648e0b..76cb63ad1 100644 --- a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc @@ -23,7 +23,7 @@ IMPORTANT: The graceful shutdown period is only applicable if the `storage-users When hard-stopping Infinite Scale, for example with the `kill ` command (SIGKILL), it is possible and likely that not all data from the decomposedfs (metadata) has been written to the storage which may result in an inconsistent decomposedfs. When gracefully shutting down Infinite Scale, using a command like SIGTERM, the process will no longer accept any write requests from _other_ services and will try to write the internal open requests which can take an undefined duration based on many factors. To mitigate that situation, the following things have been implemented: -* With the value of the environment variable `STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT`, the `storage-users` service will delay its shutdown giving it time to finalize writing necessary data. This delay can be necessary if there is a lot of data to be saved and/or if storage access/thruput is slow. In such a case you would receive an error log entry informing you that not all data could be saved in time. To prevent such occurrences, you must increase the default value. +* With the value of the environment variable `STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT`, the `storage-users` service will delay its shutdown giving it time to finalize writing necessary data. This delay can be necessary if there is a lot of data to be saved and/or if storage access/throughput is slow. In such a case you would receive an error log entry informing you that not all data could be saved in time. To prevent such occurrences, you must increase the default value. * If a shutdown error has been logged, the command-line maintenance tool xref:maintenance/commands/commands.adoc#inspect-and-manipulate-node-metadata[Inspect and Manipulate Node Metadata] can help to fix the issue. Please contact support for details. diff --git a/modules/ROOT/pages/migration/upgrading-ocis.adoc b/modules/ROOT/pages/migration/upgrading-ocis.adoc index 8b0ba62b3..300c0adaa 100644 --- a/modules/ROOT/pages/migration/upgrading-ocis.adoc +++ b/modules/ROOT/pages/migration/upgrading-ocis.adoc @@ -1,7 +1,7 @@ = Upgrading Infinite Scale :toc: right :toclevels: 2 -:description: When upgrading Infinite Scale, migration steps may be required. This document guides thru the necessary steps based on the versions used. +:description: When upgrading Infinite Scale, migration steps may be required. This document guides you through the necessary steps based on the versions used. include::partial$multi-location/compose-version.adoc[] @@ -11,9 +11,9 @@ include::partial$multi-location/compose-version.adoc[] NOTE: For any Infinite Scale version to upgrade to, see {ocis-downloadpage-url}/stable/?sort=time&order=desc[download.owncloud.com,window=_blank] or {docker_ocis_prod_url}[Infinite Scale image] to get the right version. -IMPORTANT: Before starting any upgrade, make a xref:maintenance/b-r/backup.adoc[backup] first. +IMPORTANT: Before starting any upgrade, first make a xref:maintenance/b-r/backup.adoc[backup]. -IMPORTANT: If not explicitly mentioned, any upgrade is forward only and a backstep is neither supported nor recommended and will most likely break the instance. +IMPORTANT: If not explicitly mentioned, any upgrade is forward only and a step back is neither supported nor recommended and will most likely break the instance. IMPORTANT: When upgrading from an older release to the desired one, *ALL* upgrade steps from each release must be taken in order and none can be skipped. diff --git a/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc b/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc index e078a1427..2cc1d6269 100644 --- a/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc +++ b/modules/ROOT/partials/depl-examples/ubuntu-compose/shared-setup.adoc @@ -502,7 +502,7 @@ Use the container ID identified in the following command to read the Infinite Sc docker logs 2>&1 | less ---- -The output prints the log from the beginning. As first entry, the initial admin password set during first startup is shown. You can scroll thru the log using the keyboard, see the https://wiki.ubuntuusers.de/less/[less description] for more details. +The output prints the log from the beginning. As first entry, the initial admin password set during first startup is shown. You can scroll through the log using the keyboard, see the https://wiki.ubuntuusers.de/less/[less description] for more details. If no password can be identified, you must reset the admin password via the command line as described below. From 33fab71112643e4bd770edb716a469028f4c683d Mon Sep 17 00:00:00 2001 From: Martin Date: Fri, 15 Nov 2024 15:53:06 +0100 Subject: [PATCH 25/28] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Jürgen Weigert --- modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 62c1e786f..8b10574a1 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -119,14 +119,14 @@ See the next section for *details* how to update the config settings. Run the `ocis init --diff` command: -Because this will now run the new ocis code, it will create two files to help manage the needed changes. Note that the patch file is only created if changes are found: +Because this will now run the new ocis code, it will create two files in the config directory to help manage the needed changes. Note that the patch file is only created if changes are found: . `ocis.config.patch` which contains all the changes that can be applied and . `ocis.yaml..backup` containing a backup of the original config. Note that after applying the patch file, the created files by the `--diff` option must be deleted manually. -The diff option will NOT overwrite anything though questions will be asked. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. +The diff option will NOT overwrite anything (except an earlier `ocis.config.patch`). It may ask questions though. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. If a patch file is created, do a review before applying it. From fb23547556be4e17c45e4da3e890dbc0795b5fd2 Mon Sep 17 00:00:00 2001 From: Martin Date: Thu, 21 Nov 2024 08:17:10 +0100 Subject: [PATCH 26/28] Apply suggestions from code review Co-authored-by: Phil Davis --- .../deployment/services/s-list/postprocessing.adoc | 2 +- .../deployment/services/s-list/storage-users.adoc | 2 +- .../ROOT/pages/maintenance/commands/changed-cli.adoc | 4 ++-- .../ROOT/pages/maintenance/commands/commands.adoc | 2 +- .../ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc | 12 ++++++------ 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc index 494bb95c8..12fa35a1b 100644 --- a/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc @@ -102,7 +102,7 @@ ocis storage-users uploads sessions --resume -- * *Particular failed uploads* + -Use the `postprocessing` command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, `resume` is an alias for `restart` to keep old functionality. `restart` is subject of change and will most likely be removed in a later version. +Use the `postprocessing` command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, `resume` is an alias for `restart` to keep old functionality. `restart` is subject to change and will most likely be removed in a later version. ** *Defined by ID* + diff --git a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc index 76cb63ad1..5c54ca198 100644 --- a/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/storage-users.adoc @@ -80,7 +80,7 @@ NOTE: There can never be a clear identification of a failed upload session due t [NOTE] ==== -If not noted otherwise, the `restart` option can also be replaced the `resume` option. This changes behaviour slightly. +If not noted otherwise, the `restart` option can also be replaced with the `resume` option. This changes behaviour slightly. * `restart` + When restarting an upload, all steps for open items will be restarted, except if otherwise defined. diff --git a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc index fcbfadfe8..d9de9361e 100644 --- a/modules/ROOT/pages/maintenance/commands/changed-cli.adoc +++ b/modules/ROOT/pages/maintenance/commands/changed-cli.adoc @@ -12,7 +12,7 @@ See the link for a detailed description of the respective CLI command if availab * xref:maintenance/commands/commands.adoc#manage-expired-uploads[Manage Expired Uploads] + The `ocis storage-users uploads sessions --restart` command got an alternative `--resume` option. + -Resume can be used in the same way as restart with a slighly different behavior change. +Resume can be used in the same way as restart with a slightly different behavior. * xref:maintenance/commands/commands.adoc#purge-expired-space-trash-bin-items[Purge Expired Space Trash-Bin Items] + The `ocis storage-users uploads` got restructured. + @@ -20,7 +20,7 @@ The deprecated `list` option is now removed, the `clean` option is now part of t * xref:maintenance/commands/commands.adoc#resume-post-processing[Resume Post-Processing] + The `ocis postprocessing restart` command got an alternative `resume` option. + -Resume can be used in the same way as restart with a slighly different behavior change. +Resume can be used in the same way as restart with a slightly different behavior. * xref:maintenance/commands/commands.adoc#reset-password-for-idm-users[Reset Password for IDM Users] + The `ocis idm resetpassword` can now specify the user name via the `--user-name` (`-u`) flag. diff --git a/modules/ROOT/pages/maintenance/commands/commands.adoc b/modules/ROOT/pages/maintenance/commands/commands.adoc index be42fd085..52ebaccf9 100644 --- a/modules/ROOT/pages/maintenance/commands/commands.adoc +++ b/modules/ROOT/pages/maintenance/commands/commands.adoc @@ -95,7 +95,7 @@ WARNING: Use this command with absolute care. It is not intended to play around === Resume Post-Processing -If post-processing fails in one step due to an unforeseen error, current uploads will not be retried automatically. A system administrator can instead run a xref:{s-path}/postprocessing.adoc#cli-commands[CLI Commands] to retry the failed upload. +If post-processing fails in one step due to an unforeseen error, current uploads will not be retried automatically. A system administrator can instead run a xref:{s-path}/postprocessing.adoc#cli-commands[CLI Command] to retry the failed upload. === Roll Back / Roll Forward Decomposedfs Migrations diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 8b10574a1..41eba5473 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -93,7 +93,7 @@ Reconfigure the deployment to use the new image: * The xref:{s-path}/collaboration.adoc[collaboration] service has been added. Though this service needs to be started manually, it requires a secret (preferred a random UUID string) when started. For safety reasons on a fresh install, this secret is added when running the `ocis init` command and should be added on upgrades. The used upgrade procedure provides a secret if not present in the config. Note that the secret can be overwritten in the config or with an environment variable if necessary. -** If you have used web office document servers such as Collabora, ONLYOFFICE or Microsoft using the WOPI protocol with the external https://github.com/cs3org/wopiserver[cs3org/wopi] server, we highly recommend switching to the xref:{s-path}/collaboration.adoc[collaboration] service which requires deployment reconfiguration which is a breaking change. Though the environment variables from the xref:{s-path}/app-provider.adoc[app-provider] service formerly used for web office document deployments are currently not deprecated, the are a candidate for future deprecations. +** If you have used web office document servers such as Collabora, ONLYOFFICE or Microsoft using the WOPI protocol with the external https://github.com/cs3org/wopiserver[cs3org/wopi] server, we highly recommend switching to the xref:{s-path}/collaboration.adoc[collaboration] service which requires deployment reconfiguration which is a breaking change. Though the environment variables from the xref:{s-path}/app-provider.adoc[app-provider] service formerly used for web office document deployments are currently not deprecated, they are a candidate for future deprecations. * If you have used a rolling release and deployed web office document servers like the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] or the xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner] deployment example, you should follow the reconfiguration section below because the deployment files have been updated/upgraded. @@ -103,9 +103,9 @@ Reconfigure the deployment to use the new image: * Environment variables that have been deprecated without successor can safely be removed from the configuration. These envvars do not harm as they are not used anymore. * Environment variables that have been deprecated with a successor need to be updated accordingly. -* Environment variables that habe been removed from Infinite Scale can safely be removed from the config if set manually. +* Environment variables that have been removed from Infinite Scale can safely be removed from the config if set manually. -==== Add new Infnite Scale config settings +==== Add new Infinite Scale config settings To do so, the following steps need to be taken, here is the *overview*: @@ -124,7 +124,7 @@ Because this will now run the new ocis code, it will create two files in the con . `ocis.config.patch` which contains all the changes that can be applied and . `ocis.yaml..backup` containing a backup of the original config. -Note that after applying the patch file, the created files by the `--diff` option must be deleted manually. +Note that after applying the patch file, the files created by the `--diff` option must be deleted manually. The diff option will NOT overwrite anything (except an earlier `ocis.config.patch`). It may ask questions though. Consider using the correct answer for `certificate checking` as this will influence the respective diff output. @@ -214,7 +214,7 @@ patch < ocis.config.patch * *Any Other Image Based Deployment*: + -- -Use one of the Docker based examples from above and derive from. +Use one of the Docker based examples from above and adjust for your deployment. -- === Reconfigure web Office Document Deployments @@ -226,7 +226,7 @@ The following steps are based on the xref:depl-examples/ubuntu-compose/ubuntu-co * Backup the the base folder containing the existing deployment example by renaming it. + You will need your configuration details with the new example. -* Follow the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#download-and-transfer-the-example[Download and Transfer the Example] to get the new deployment and extract it as described in the following section of the guide. +* Follow the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc#download-and-transfer-the-example[Download and Transfer Example] to get the new deployment and extract it as described in the following section of the guide. * Reconfigure the new `.env` file based on settings made in the `.env` file of the backup. From 70ca1b1c110a6c7c29d0fff22c7902a45ce7f24d Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Fri, 22 Nov 2024 16:43:58 +0100 Subject: [PATCH 27/28] add csp to migrations --- antora.yml | 3 ++ .../ubuntu-compose-hetzner.adoc | 3 -- .../ubuntu-compose/ubuntu-compose-prod.adoc | 3 -- .../deployment/services/s-list/proxy.adoc | 18 ++++++++++++ .../migration/upgrading_5.0.x_7.0.0.adoc | 28 +++++++++++++------ 5 files changed, 41 insertions(+), 14 deletions(-) diff --git a/antora.yml b/antora.yml index 63a9ee1a5..bae4ea0e3 100644 --- a/antora.yml +++ b/antora.yml @@ -22,6 +22,9 @@ asciidoc: # https://github.com/owncloud/ocis-charts/tree/master/charts/ocis/docs s-path: 'deployment/services/s-list' + # the name of the sub directory that defines the sources used in the compose deployment examples + ocis_wopi: ocis_full + # note that service_url_component is used for services ONLY # service_url_component will be used to assemble the url for services to include content (tables) sourced from the ocis repo. service_url_component: 'docs' # docs for master or for stable, like docs-stable-5.0 diff --git a/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc b/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc index 16b1a7483..4802da601 100644 --- a/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc +++ b/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc @@ -4,9 +4,6 @@ :keywords: docker compose, raspberry pi, install, ocis, infinite scale, letsencrypt :description: Install Infinite Scale using Docker Compose on the Hetzner Cloud for production use. -// the folder to use for the example -:ocis_wopi: ocis_full - include::partial$multi-location/compose-version.adoc[] image:depl-examples/ubuntu-compose/ubuntu-basic-teaser-image.png[Teaser Image, width=650] diff --git a/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc b/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc index 211f55a62..2b660b7b0 100644 --- a/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc +++ b/modules/ROOT/pages/depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc @@ -4,9 +4,6 @@ :keywords: docker compose, raspberry pi, install, ocis, infinite scale, letsencrypt :description: Install Infinite Scale using Docker Compose on a server for production use. -// the folder to use for the example -:ocis_wopi: ocis_full - include::partial$multi-location/compose-version.adoc[] image:depl-examples/ubuntu-compose/ubuntu-basic-teaser-image.png[Teaser Image, width=650] diff --git a/modules/ROOT/pages/deployment/services/s-list/proxy.adoc b/modules/ROOT/pages/deployment/services/s-list/proxy.adoc index d5e3bab8e..f674cc4f9 100644 --- a/modules/ROOT/pages/deployment/services/s-list/proxy.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/proxy.adoc @@ -5,6 +5,8 @@ :service_name: proxy +include::partial$multi-location/compose-version.adoc[] + == Introduction {description} Every HTTP request goes through this service. Authentication, logging and other preprocessing of requests also happens here. Mechanisms like request rate limiting or intrusion prevention are **not** included in the proxy service and must be set up in front of an external reverse proxy. @@ -174,6 +176,22 @@ The default `role_claim` (or `PROXY_ROLE_ASSIGNMENT_OIDC_CLAIM`) is `roles`. The * In a production deployment, you want to have basic authentication (`PROXY_ENABLE_BASIC_AUTH`) _disabled_ which is the default state. You should also set up a firewall to only allow requests to the proxy service or the reverse proxy if you have one. Requests to the other services should be blocked by the firewall. +=== Content Security Policy + +What is a Content Security Policy (CSP) and why is it used in Infinite Scale:: ++ +-- +A Content Security Policy (CSP) is a feature that helps to prevent or minimize the risk of certain types of security threats. It consists of a series of instructions from a website to a browser, which instruct the browser to place restrictions on the things that the code comprising site is allowed to do. It is mainly used as a defense against cross-site scripting (XSS) attacks, in which an attacker is able to inject malicious code into the victim's site and includes defending against clickjacking, and helping to ensure that a site's pages will be loaded over HTTPS. + +For Infinite Scale, external resources like an IDP (e.g. Keycloak) or when using web office documents or web apps, require defining a CSP. If not defined, the referenced services will not work. +-- + +To create a Content Security Policy (CSP), you need to create a yaml file containing the CSP definitions. To activate the settings, reference the file as value in the `PROXY_CSP_CONFIG_FILE_LOCATION` environment variable. For each change, a restart of the Infinite Scale deployment or the proxy service is required. + +A working example for a CSP can be found in a sub path of the `config` directory of the {compose_url}v{compose_version}{compose_final_path}/{ocis_wopi}[{ocis_wopi},window=_blank] deployment example which is the base for our xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] and the xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner]. + +See the https://content-security-policy.com[Content Security Policy (CSP) Quick Reference Guide,window=_blank] for a description of directives. + == Presigned Urls Important, also see section xref:caching[caching] above. diff --git a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc index 41eba5473..91a6a1beb 100644 --- a/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc +++ b/modules/ROOT/pages/migration/upgrading_5.0.x_7.0.0.adoc @@ -2,6 +2,8 @@ :toc: right :description: This document describes the necessary steps when upgrading Infinite Scale from release 5.0.x to 7.0.0. +:actual_seven_version: 7.0.0 + include::partial$multi-location/compose-version.adoc[] == Introduction @@ -38,7 +40,7 @@ Follow the xref:depl-examples/minimal-bare-metal.adoc#installation[Installation] + [source,bash,subs="attributes+"] ---- -docker pull owncloud/ocis:7.0.0 +docker pull owncloud/ocis:{actual_seven_version} ---- == Shut Down the Infinite Scale Instance @@ -89,9 +91,11 @@ Reconfigure the deployment to use the new image: * If you are using removed or deprecated environment variables in your config. ** See the xref:deployment/services/env-var-changes.adoc[Added and Removed Environment Variables] documentation. -* Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, _everyone_ who upgrades is affected. +* Because the xref:{s-path}/activitylog.adoc[activitylog] service has been added and is started automatically, *everyone* is affected. + +* Because of the implementation of a Content Security Policy (CSP), *everyone* is affected *if* any external services like an IDP (e.g. Keycloak, Authentik, etc.) or web office servers have been used. -* The xref:{s-path}/collaboration.adoc[collaboration] service has been added. Though this service needs to be started manually, it requires a secret (preferred a random UUID string) when started. For safety reasons on a fresh install, this secret is added when running the `ocis init` command and should be added on upgrades. The used upgrade procedure provides a secret if not present in the config. Note that the secret can be overwritten in the config or with an environment variable if necessary. +* The xref:{s-path}/collaboration.adoc[collaboration] service has been added which replaces the external https://github.com/cs3org/wopiserver[cs3org/wopi] server used before. Though the collaboration service needs to be started manually, it requires a secret (preferred a random UUID string) when started. For safety reasons on a fresh install, this secret is added automatically when running the `ocis init` command and should be added on upgrades. The used upgrade procedure provides a secret if not present in the config. Note that the secret can be overwritten in the config or with an environment variable if necessary at any time. ** If you have used web office document servers such as Collabora, ONLYOFFICE or Microsoft using the WOPI protocol with the external https://github.com/cs3org/wopiserver[cs3org/wopi] server, we highly recommend switching to the xref:{s-path}/collaboration.adoc[collaboration] service which requires deployment reconfiguration which is a breaking change. Though the environment variables from the xref:{s-path}/app-provider.adoc[app-provider] service formerly used for web office document deployments are currently not deprecated, they are a candidate for future deprecations. @@ -156,13 +160,13 @@ patch < ocis.config.patch -- This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the mount/volume definition where the configuration settings are stored to make the `ocis init` command work. The bind mount from below is just an example and fails if the local directory does not exist! -[source,bash] +[source,bash,subs="attributes+"]] ---- sudo docker run \ --rm -it \ --entrypoint sh \ --mount type=bind,source=$HOME/ocis/ocis-config,target=/etc/ocis \ - owncloud/ocis:7.0.0 + owncloud/ocis:{actual_seven_version} ---- Then, when the containers shell is available, type: @@ -186,13 +190,13 @@ patch < ocis.config.patch -- This will open a shell in a temporary container using the new Infinite Scale image. Note that you *MUST* provide the volume definition (either the volume name or the local path) where the configuration settings are stored defined in the compose file to make the `ocis init` command work. The named volume `ocis-config` from below is just an example and derives from our docker compose deployment examples. -[source,bash] +[source,bash,subs="attributes+"]] ---- sudo docker run \ --rm -it \ --entrypoint sh \ -v ocis-config:/etc/ocis \ - owncloud/ocis:7.0.0 + owncloud/ocis:{actual_seven_version} ---- Then, when the containers shell is available, type: @@ -217,9 +221,17 @@ patch < ocis.config.patch Use one of the Docker based examples from above and adjust for your deployment. -- +=== Define a Content Security Policy + +To set a Content Security Policy (CSP), you need to create a yaml file containing the definitions. To activate the settings, reference the file as value in the `PROXY_CSP_CONFIG_FILE_LOCATION` environment variable. For each change, a restart of the Infinite Scale deployment or the proxy service is required. + +For details how to define a CSP, see the xref:{s-path}/proxy.adoc#content-security-policy[Content Security Policy, window=_blank] section in the proxy service documentation. + +The reconfiguration of web office document deployments as described in the next section already contain a preconfigured CSP. + === Reconfigure web Office Document Deployments -With the introduction of the xref:{s-path}/collaboration.adoc[collaboration] service, the external https://github.com/cs3org/wopiserver[cs3org/wopi] server is no longer required. Deployments using the external cs3org/wopi server should upgrade. This upgrade is a breaking change, but if you have used one of the `Ubuntu with Docker Compose` deployment examples, a re-deployment is quite easy. If you have setup your own deployment not based on the examples, you can derive from the changes based on the deployment files. +With the introduction of the xref:{s-path}/collaboration.adoc[collaboration] service, the external https://github.com/cs3org/wopiserver[cs3org/wopi] server is no longer required. Deployments using the external cs3org/wopi server must upgrade. This upgrade is a breaking change because of the need to use a CSP which is provided. If you have used one of the `Ubuntu with Docker Compose` deployment examples, a re-deployment is quite easy. If you have setup your own deployment not based on the examples, you can derive from the changes based on the deployment files. The following steps are based on the xref:depl-examples/ubuntu-compose/ubuntu-compose-prod.adoc[Local Production Setup] deployment example. The steps are identical for the xref:depl-examples/ubuntu-compose/ubuntu-compose-hetzner.adoc[Deployment on Hetzner]. From c96d6d0055cc0bec70aec720b4434d406cc44b3d Mon Sep 17 00:00:00 2001 From: Martin Mattel Date: Mon, 25 Nov 2024 11:00:38 +0100 Subject: [PATCH 28/28] FRONTEND_OCS_xxx envvars are deprecated --- modules/ROOT/pages/deployment/services/s-list/frontend.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/deployment/services/s-list/frontend.adoc b/modules/ROOT/pages/deployment/services/s-list/frontend.adoc index 241c4b84d..1a0ded01a 100644 --- a/modules/ROOT/pages/deployment/services/s-list/frontend.adoc +++ b/modules/ROOT/pages/deployment/services/s-list/frontend.adoc @@ -103,11 +103,11 @@ Aggregating share information is one of the most time consuming operations in OC To save network trips the sharing implementation can cache the stat requests with an in memory cache or in redis. It will shorten the response time by the network round-trip overhead at the cost of the API only eventually being updated. -Setting `FRONTEND_OCS_RESOURCE_INFO_CACHE_TTL=60` would cache the stat info for 60 seconds. Increasing this value makes sense for large deployments with thousands of active users that keep the cache up to date. Low frequency usage scenarios should not expect a noticeable improvement. +Setting `FRONTEND_OCS_RESOURCE_INFO_CACHE_TTL=60` (deprecated) would cache the stat info for 60 seconds. Increasing this value makes sense for large deployments with thousands of active users that keep the cache up to date. Low frequency usage scenarios should not expect a noticeable improvement. == Scalability -While the frontend service does not persist any data, it does cache information about files and filesystem (`Stat()`) responses and user information. Therefore, multiple instances of this service can be spawned in a bigger deployment like when using container orchestration with Kubernetes, when configuring `FRONTEND_OCS_RESOURCE_INFO_CACHE_STORE` and the related config options. +While the frontend service does not persist any data, it does cache information about files and filesystem (`Stat()`) responses and user information. Therefore, multiple instances of this service can be spawned in a bigger deployment like when using container orchestration with Kubernetes, when configuring `FRONTEND_OCS_RESOURCE_INFO_CACHE_STORE` (deprecated) and the related config options. == Define Read-Only Attributes