diff --git a/src/dev/ce-migration/index.md b/src/dev/ce-migration/index.md index 2426a4bd..0a1628b3 100644 --- a/src/dev/ce-migration/index.md +++ b/src/dev/ce-migration/index.md @@ -1,17 +1,53 @@ # Migration Guides + +Migration guides are here to help you migrate to the latest version of our [ Community Edition](../mergince/). + +::: warning +Migrations must be performed one by one and cannot be skipped. + +Make sure to always back up your database data before doing a migration. +::: + [[toc]] -Migration guides are here to help you migrate to the latest version of [ Community Edition](../mergince/). +## Migration guide from 2023.2.0+ to 2023.6.1 + +⚠️ If you are on a server version lower than `2023.2.0`, it is important to first [upgrade to `2023.2.0`](#migration-guide-from-any-previous-version-to-2023-2-0) before continuing with this migration. + +----- + +Get the latest or update docker images manually to version `2023.6.1`. +Perform the migration: + +1. Start up your docker containers + ```bash + $ docker-compose -f docker-compose.yml up # or similarly, based on your deployment + ``` + +2. Check that you are on a correct version (`b6cb0a98ce20`) + ```bash + $ docker exec merginmaps-server flask db current + INFO [alembic.runtime.migration] Context impl PostgresqlImpl. + INFO [alembic.runtime.migration] Will assume transactional DDL. + b6cb0a98ce20 # <--- important + ``` + + - If you do not see the version number on the last line, run the following command: + ```bash + $ docker exec merginmaps-server flask db stamp b6cb0a98ce20 + ``` -## 2023.2.0 -> 2023.3.0 -Since there is no db migration or new config settings needed it is enough just to use new docker image tag. +3. Run the database migration + ```bash + $ docker exec merginmaps-server flask db upgrade 3a77058a2fd7 + ``` -## Migration guide from any previous version to 2023.2 +## Migration guide from any previous version to 2023.2.0 Besides various fixes, enhancements and performance improvements the most notable change recently introduced is the concept of workspaces. For Community Edition it means there is a **common shared workspace (global workspace)** for all users where all projects are stored, instead of having a personal or organisational namespace for projects. :::tip -In case you do not need your previous data, we advise to start with [clean deployment](../mergince/#how-to-deploy-mergin-maps-ce) without the need to follow this migration guide. +In case you do not need your previous data, we advise to start with [clean deployment](../mergince/#deployment) without the need to follow this migration guide. ::: **Upgrading to 2023.2** @@ -44,7 +80,7 @@ Specify its name with the following environment variable: - `GLOBAL_WORKSPACE=ShinyWorkspace` - name of your workspace. A good fit is a name of your company or team. This value *should not be changed* later. :::tip -You can find all available environment variables [here](../mergince/#how-to-deploy-mergin-maps-ce) together with a tutorial how to set them up. +You can find all available environment variables [here](../mergince/#deployment) together with a tutorial how to set them up. ::: Further, you need to set a default role for people in your workspace. **Pick one** of these options @@ -63,7 +99,7 @@ You can specify the maximum storage for your shiny new workspace 🌟 with the f New users can be created from administration panel, by navigating to `/admin`. ::: - 1. Make sure projects volume mounts in `docker-compose` file still match (You can set up new volumes by following the [quick start guide](../mergince/#how-to-deploy-mergin-maps-ce)). Switch to new server version and PostgreSQL to at least version 12 (14 recommended) by running new docker containers: + 1. Make sure projects volume mounts in `docker-compose` file still match (You can set up new volumes by following the [quick start guide](../mergince/#deployment)). Switch to new server version and PostgreSQL to at least version 12 (14 recommended) by running new docker containers: ```bash $ docker-compose -f docker-compose.yml up ``` diff --git a/src/dev/mergince.md b/src/dev/mergince.md index 8ce36e25..66fe86be 100644 --- a/src/dev/mergince.md +++ b/src/dev/mergince.md @@ -13,11 +13,11 @@ More information about and its features can be found o Need more functionality than offers? Explore our [subscription plans](https://merginmaps.com/pricing). ::: -## How to deploy Mergin Maps CE +## Deployment Follow these steps to run local instance. ### Start docker containers -Provided that `docker` and `docker-compose` are installed on your host, running Mergin Maps stack should be as simple as running `docker-compose`. However, before doing that you would need to [configure](#configure) your server setup via environment variables in file. +Provided that `docker` and `docker-compose` are installed on your host, running stack should be as simple as running `docker-compose`. However, before doing that you would need to [configure](#configure-environment) your server setup via environment variables in file. Once configured, you can run: ```shell @@ -27,92 +27,99 @@ $ sudo chmod g+s ./projects/ $ docker-compose -f docker-compose.yml up ``` ​​ -### Initialise +### Initialise database If server is started for the first time, database needs to be initialised and super-user created (set admin username, password and email): ```shell $ docker exec merginmaps-server flask init-db $ docker exec merginmaps-server flask user create --is-admin --email ``` ​ -### Configure +### Configure environment There are several application settings which can be changed via . Some of them have defaults and some of them needs to be modified, particularly those with `fixme` placeholder (marked with asterisks below). ​ #### Server basics -Settings about your deployment. Variables marked with star ⭐️ need to be modified. +Variables marked with star ⭐️ need to be modified for production use. | Variable name | Type | Default | Description | |--------------------------|-----------|-----------|-------------| | `CONTACT_EMAIL`⭐️ | string | | Email contact for application administrator. | -| `MERGIN_BASE_URL`⭐️ | string | | Deployment URL where is hosted. | | `COLLECT_STATISTICS` | Boolean | `true` | Whether to send usage statistics for application improvements. | +| `MERGIN_BASE_URL`⭐️ | string | | Deployment URL where is hosted. | | `SERVICE_ID` | string | | Deployment UUID. Auto-generated on first run. | ​ - -#### Security settings -Security settings. Variables marked with star ⭐️ need to be modified. +#### Security settings (important for production use)🛡️ +Security settings are important for production use. | Variable name | Type | Default | Description | |--------------------------|-----------|-----------|-------------| +| `BEARER_TOKEN_EXPIRATION`| integer | `43200` | Lifetime of authorisation bearer token in seconds. When expired, user needs to log in again. | | `SECRET_KEY`⭐️ | string | | Secret key for authorisation, should be generated strong string. | | `SECURITY_PASSWORD_SALT`⭐️| string | | Password salt for hashing, should be generated strong string. | -| `BEARER_TOKEN_EXPIRATION`| integer | `43200` | Lifetime of authorisation bearer token in seconds. When expired, user needs to log in again. | | `WTF_CSRF_ENABLED` | Boolean | `true` | Enable CSRF protection. It is strongly recommended to have this on. | | `WTF_CSRF_TIME_LIMIT` | integer | `86400` | Lifetime of CSRF token in seconds. When expired, user needs to refresh it. | ​ #### Database settings -Mergin Maps uses PostgreSQL database to store its data. Variables marked with star ⭐️ need to be modified. +Mergin Maps uses PostgreSQL database to store its data. ​ | Variable name | Type | Default | Description | |---------------------------|-----------|-----------|-------------| | `DB_APPLICATION_NAME` | string | `mergin` | Comment in database connection string to better identify connection source. | -| `DB_DATABASE` | string |`postgres` | Database to store tables. | +| `DB_DATABASE` ⭐️ | string |`postgres` | Database to store tables. | | `DB_HOST` | string | `db` | Database host. Mapped to docker-compose service name. | +| `DB_PASSWORD` ⭐️ | string |`postgres` | PostgreSQL user password. | | `DB_PORT` | integer | `5432` | Database port. If non-default should match port exposed in docker-compose file. | -| `DB_USER`⭐️ | string |`postgres` | PostgreSQL user to connect to database. | -| `DB_PASSWORD`⭐️ | string |`postgres` | PostgreSQL user password. | | `DB_POOL_MAX_OVERFLOW=10` | integer | `10` | Database `max_overflow` limit for [SQLAlchemy](https://docs.sqlalchemy.org/en/14/core/engines.html). | | `DB_POOL_SIZE` | integer | `2` | Database pool size for SQLAlchemy. With overflow determines maximum of concurrent connections to database. | | `DB_POOL_TIMEOUT` | integer | `300` | Database pool timeout for SQLAlchemy. | +| `DB_USER` ⭐️ | string |`postgres` | PostgreSQL user to connect to database. | ​ +#### Permission management +To ease the process of permission (user) management, you can set the following global variables that apply to all registered users. + +| Variable name | Type | Default | Description | +|---------------------------|---------|-------------|---------------------------| +| `GLOBAL_ADMIN` | Boolean | `false` | All registered users can create/delete projects. | +| `GLOBAL_READ` | Boolean | `false` | All registered users have read access to all projects. If false, application admin would need to grant project access to users manually. | +| `GLOBAL_WRITE` | Boolean | `false` | All registered users have write access (can sync) to all projects. | + + #### Sending Emails -​Mergin Maps can connect to SMTP server to send notifications. Also required for password recovery to work. Variables marked with star ⭐️ need to be modified. +​Mergin Maps can connect to SMTP server to send notifications and password recovery emails. ​ | Variable name | Type | Default | Description | |---------------------------|-----------|-----------|-------------| -| `MAIL_SUPPRESS_SEND`⭐️ | Boolean | `true` | Whether to suppress email sending. | -| `MAIL_BCC`⭐️ | string | | Email address to send copies of all emails sent. Should be system/application administrator. | -| `MAIL_DEFAULT_SENDER`⭐️ | string | | Sender of emails. Best to have some no-reply address. | -| `MAIL_USERNAME`⭐️ | string | | Connection to SMTP server. | -| `MAIL_PASSWORD`⭐️ | string | | Password for user connecting to SMTP server. | -| `MAIL_PORT`⭐️ | integer | `587` | SMTP mail server port. | -| `MAIL_SERVER`⭐️ | string |`localhost`| SMTP mail server host. | -| `MERGIN_LOGO_URL`⭐️ | string | `null` | Link to logo in emails. | - +| `MAIL_SUPPRESS_SEND` | Boolean | `true` | Whether to suppress email sending. If set to true, you should define the following variables. | +| `MAIL_BCC` | string | | Email address to send copies of all emails sent. Should be system/application administrator. | +| `MAIL_DEFAULT_SENDER` | string | | Sender of emails. Best to have some no-reply address. | +| `MAIL_USERNAME` | string | | Connection to SMTP server. | +| `MAIL_PASSWORD` | string | | Password for user connecting to SMTP server. | +| `MAIL_PORT` | integer | `587` | SMTP mail server port. | +| `MAIL_SERVER` | string |`localhost`| SMTP mail server host. | +| `MERGIN_LOGO_URL` | string | `null` | Link to logo in emails. | + + +#### Data synchronisation and management +Other settings related to data management. + +| Variable name | Type | Default | Description | +|------------------------------|---------|-------------|---------------------------| +| `GLOBAL_WORKSPACE` ⭐️ | string | `mergin` | Namespace (part of URL) for all projects. All projects belong to this single workspace with certain permissions (see below). | +| `GLOBAL_STORAGE` ⭐️ | integer |`10737418240`| Storage limit can use to store projects (last version) in bytes (default is 10 GB). Should be reasonably large. | +| `LOCAL_PROJECTS` | string | `./projects` | Directory to store projects on container. Please refer to volume mapping in docker-compose file. | +| `TEMP_DIR` | string | Result of `gettempdir()` call | Trash directory for temp files being cleaned regularly. Please refer to volume mapping in docker-compose file. | +| `MAINTENANCE_FILE` | string |`/data/MAINTENANCE`| File to indicate server is in maintenance - read only mode. Please refer to volume mapping in docker-compose file. | +| `BLACKLIST` | string | `.mergin/`, `.DS_Store`, `.directory` | Pattern to ignore when syncing files. | +| `FILE_EXPIRATION` | integer | `172800` | When GeoPackage file was updated with "" change, original data are being removed (as they can be reconstructed on demand) to save disk space. File lifetime in seconds. | +| `LOCKFILE_EXPIRATION` | integer | `300` | Time in seconds for project being locked while updated. If no change happens to project in such time, lockfile is removed. | +| `MAX_CHUNK_SIZE` | integer | `10485760` | Maximum size of file chunk to be uploaded (and received by server) in bytes. | +| `MAX_DOWNLOAD_ARCHIVE_SIZE` | integer | `1073741824`| Maximum size of project zip archive in bytes for direct download. Too large projects may take too long to download or never complete in one request. | +| `USE_X_ACCEL` ⭐️ | Boolean | `false` | Whether to use nginx do serve files. Should be enabled if used with nginx proxy for performance reasons. Read more [here](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/). | +| `CLOSED_ACCOUNT_EXPIRATION` | integer | `1` | Time in days after user closed his account to all projects and files are permanently deleted. Please note than until user is removed username/email is occupied. | +| `DELETED_PROJECT_EXPIRATION` | integer| `7` | Lifetime in days for deleted project. Expired projects are removed permanently without possibility to restore afterwards. | +| `PROJECT_ACCESS_REQUEST` | integer | `604800` | Lifetime of active project access request in seconds. | +| `TEMP_EXPIRATION` | integer | `7` | Time in days after files in temporary folder are permanently deleted. | ​ -#### Data synchronisation and user management -​Variables marked with star ⭐️ need to be modified. -| Variable name | Type | Default | Description | -|---------------------------|---------|-------------|---------------------------| -| `GLOBAL_WORKSPACE`⭐️ | string | `mergin` | Namespace (part of URL) for all projects. All projects belong to this single workspace with certain permissions (see below). | -| `GLOBAL_READ` | Boolean | `true` | All registered users have read access to all projects. If false, application admin would need to grant project access to users manually. | -| `GLOBAL_WRITE` | Boolean | `false` | All registered users have write access (can sync) to all projects. | -| `GLOBAL_ADMIN` | Boolean | `false` | All registered users can create/delete projects. | -| `GLOBAL_STORAGE`⭐️ | integer |`10737418240`| Storage limit Mergin Maps can use to store projects (last version) in bytes (default is 10 GB). Should be reasonably large. | -| `LOCAL_PROJECTS` | string |`/data/live` | Directory to store projects on container. Please refer to volume mapping in docker-compose file. | -| `TEMP_DIR` | string | `/data/tmp` | Trash directory for temp files being cleaned regularly. Please refer to volume mapping in docker-compose file. | -| `MAINTENANCE_FILE` | string |`/data/MAINTENANCE`| File to indicate server is in maintenance - read only mode. Please refer to volume mapping in docker-compose file. | -| `BLACKLIST` | string | `.mergin/`, `.DS_Store`, `.directory` | Pattern to ignore when syncing files. | -| `FILE_EXPIRATION` | integer | `172800` | When GeoPackage file was updated with "" change, original data are being removed (as they can be reconstructed on demand) to save disk space. File lifetime in seconds. | -| `LOCKFILE_EXPIRATION` | integer | `300` | Time in seconds for project being locked while updated. If no change happens to project in such time, lockfile is removed. | -| `MAX_CHUNK_SIZE` | integer | `10485760` | Maximum size of file chunk to be uploaded (and received by server) in bytes. | -|`MAX_DOWNLOAD_ARCHIVE_SIZE`| integer | `1073741824`| Maximum size of project zip archive in bytes for direct download. Too large projects may take too long to download or never complete in one request. | -| `USE_X_ACCEL`⭐️ | Boolean | `false` | Whether to use nginx do serve files. Should be enabled if used with nginx proxy for performance reasons. Read more [here](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/). | -|`CLOSED_ACCOUNT_EXPIRATION`| integer | `1` | Time in days after user closed his account to all projects and files are permanently deleted. Please note than until user is removed username/email is occupied. | -| `DELETED_PROJECT_EXPIRATION`| integer| `7` | Lifetime in days for deleted project. Expired projects are removed permanently without possibility to restore afterwards. | -| `PROJECT_ACCESS_REQUEST` | integer | `604800` | Lifetime of active project access request in seconds. | -| `TEMP_EXPIRATION` | integer | `7` | Time in days after files in temporary folder are permanently deleted. | -​ #### Celery asynchronous tasks ​Mergin Maps is using Celery and Redis to perform asynchronous tasks or doing regular jobs. ​​ @@ -121,24 +128,17 @@ Mergin Maps uses PostgreSQL database to store its data. Variables marked with st |`BROKER_URL` |string|`redis://merginmaps-redis:6379/0`| Connection details to celery message broker. If non-default, it should match definition in `docker-compose` file. | |`CELERY_RESULT_BACKEND`|string|`redis://merginmaps-redis:6379/0`| Connection details to celery result back-end broker. If non-default, it should match definition in `docker-compose` file. | ​ -​ -## How to opt out of sending statistics - -By default, collects anonymous usage information to make the service better. There is a variable named `COLLECT_STATISTICS` that controls if statistics are collected and sent to . -If you do not want to provide these data, you can opt-out any time by setting this variable to *false* -``` -COLLECT_STATISTICS=false -``` +## Migration guides -## Server migration guides +Follow our [**Migration Guides**](./ce-migration/) to migrate from older server versions to the latest version of .​ -Follow our [**Migration Guides**](./ce-migration/) to migrate from older versions to the latest version of . +## Troubleshooting -## Mergin Maps CE troubleshooting +Find the common server deployment issues here. -### Mergin Maps CE server is not properly configured +### Server is not properly configured Did you get an error that the server is not properly configured? ![Mergin Maps CE server not configured error](./ce-server-not-configured.jpg "Mergin Maps CE server not configured error") @@ -147,3 +147,15 @@ Did you get an error that the server is not properly configured? 2. Restart the container with the `MERGIN_BASE_URL` variable + +## Opt out of sending statistics + + +By default, collects anonymous usage information to make the service better. There is a variable named `COLLECT_STATISTICS` that controls if statistics are collected and sent. + +If you do not want to provide these data, you can opt-out any time by setting this variable to *false* +``` +COLLECT_STATISTICS=false +``` + +