diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 595ffe4ef..706315d9f 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,5 +1,5 @@ --- -name: Bug report +name: Bug Report about: Create a report to help us improve title: '' labels: '' diff --git a/.github/ISSUE_TEMPLATE/release.md b/.github/ISSUE_TEMPLATE/release.md new file mode 100644 index 000000000..5a5f580a6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/release.md @@ -0,0 +1,19 @@ +--- +name: [INTERNAL] Release +about: Release a version +title: 'Release Version v$version' +labels: '' +assignees: '' + +--- + +# Release Workflow + +- [ ] Check that the `.mailmap` and `AUTHORS` files are up to date +- [ ] Update `internal/version.go` +- [ ] Update `CHANGELOG.md` +- [ ] Create and push a signed tag for the version +- [ ] Build packages +- [ ] Create release on GitHub +- [ ] Update public docs +- [ ] Announce release diff --git a/.github/workflows/version.yml b/.github/workflows/version.yml new file mode 100644 index 000000000..98d28c0e7 --- /dev/null +++ b/.github/workflows/version.yml @@ -0,0 +1,25 @@ +name: Version +on: + push: + branches: + - master + pull_request: {} + +jobs: + version: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + fetch-depth: 0 + - uses: actions/setup-go@v2 + with: + go-version: 1.18.x + - name: Check version + # TODO(elippmann): Needs adjustments as soon as we release from (support) branches too. + run: | + actual="$(go run cmd/icingadb/main.go --version | head -n1 | sed 's/Icinga DB version: \(.*\)/v\1/')" + expected="$(git describe --tags $(git rev-list --tags --max-count=1))" + echo "Actual version: $actual" + echo "Expected version: $expected" + test "$actual" = "$expected" || (echo "Versions do not match"; (exit 1)) diff --git a/CHANGELOG.md b/CHANGELOG.md index 37de654db..6cf72aa5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ -## 1.0.0 RC1 (2020-03-13) +# Icinga DB Changelog + +## 1.0.0 RC2 (2021-11-12) -### Notes +Second release candidate + +## 1.0.0 RC1 (2020-03-13) -First official release \ No newline at end of file +Initial release diff --git a/README.md b/README.md index b57623ff8..8cc7c16ce 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,23 @@ # Icinga DB -![Icinga Logo](https://icinga.com/wp-content/uploads/2014/06/icinga_logo.png) +Icinga DB is a set of components for publishing, synchronizing and +visualizing monitoring data in the Icinga ecosystem, consisting of: -#### Table of Contents +* The Icinga DB daemon, which synchronizes monitoring data between a Redis server and a database +* Icinga 2 with its [Icinga DB feature](https://icinga.com/docs/icinga-2/latest/doc/14-features/#icinga-db) enabled, + responsible for publishing the data to the Redis server, i.e. configuration and its runtime updates, check results, + state changes, downtimes, acknowledgements, notifications, and other events such as flapping +* And Icinga Web with the + [Icinga DB Web](https://icinga.com/docs/icinga-db-web) module enabled, + which connects to both Redis and the database to display and work with the most up-to-date data -- [About](#about) -- [License](#license) -- [Installation](#installation) -- [Documentation](#documentation) -- [Support](#support) -- [Contributing](#contributing) - -## About - -Icinga DB serves as a synchronisation daemon between Icinga 2 (Redis) and Icinga Web 2 (MySQL/MariaDB/PostgreSQL database). -It synchronises configuration, state and history of an Icinga 2 environment using checksums. - -Icinga DB also supports reading from multiple environments and writing into a single database. - -## License - -Icinga DB and the Icinga DB documentation are licensed under the terms of the GNU -General Public License Version 2, you will find a copy of this license in the -LICENSE file included in the source package. - -## Installation - -For installing Icinga DB please check the [installation chapter](https://icinga.com/docs/icingadb/latest/doc/02-Installation/) -in the documentation. +![Icinga DB Architecture](doc/images/icingadb-architecture.png) ## Documentation -The documentation is located in the [doc/](doc/) directory and also available -on [icinga.com/docs](https://icinga.com/docs/icingadb/latest/). - -## Support +Icinga DB documentation is available at [icinga.com/docs](https://icinga.com/docs/icinga-db). -Check the [project website](https://icinga.com) for status updates. Join the -[community channels](https://icinga.com/community/) for questions -or ask an Icinga partner for [professional support](https://icinga.com/support/). - -## Contributing - -There are many ways to contribute to Icinga -- whether it be sending patches, -testing, reporting bugs, or reviewing and updating the documentation. Every -contribution is appreciated! +## License -Please continue reading in the [contributing chapter](CONTRIBUTING.md). +Icinga DB and the Icinga DB documentation are licensed under the terms of the +[GNU General Public License Version 2](LICENSE). diff --git a/RELEASE.md b/RELEASE.md deleted file mode 100644 index ce0013eac..000000000 --- a/RELEASE.md +++ /dev/null @@ -1,293 +0,0 @@ -# Release Workflow - -#### Table of Content - -- [1. Preparations](#preparations) - - [1.1. Issues](#issues) - - [1.2. Backport Commits](#backport-commits) - - [1.3. Authors](#authors) -- [2. Version](#version) -- [3. Changelog](#changelog) -- [4. Git Tag](#git-tag) -- [5. Package Builds](#package-builds) - - [5.1. RPM Packages](#rpm-packages) - - [5.2. DEB Packages](#deb-packages) -- [6. Build Server](#build-infrastructure) -- [7. Release Tests](#release-tests) -- [8. GitHub Release](#github-release) -- [9. Chocolatey](#chocolatey) -- [10. Post Release](#post-release) - - [10.1. Online Documentation](#online-documentation) - - [10.2. Announcement](#announcement) - - [10.3. Project Management](#project-management) - -## Preparations - -Specify the release version. - -``` -VERSION=1.0.0 -``` - -Add your signing key to your Git configuration file, if not already there. - -``` -vim $HOME/.gitconfig - -[user] - email = noah.hilverling@icinga.com - name = Noah Hilverling - signingKey = "CE23E835630F89D9" -``` - -### Issues - -Check issues at https://github.com/icinga/icingadb - -### Backport Commits - -For minor versions you need to manually backports any and all commits from the -master branch which should be part of this release. - -### Authors - -Update the [.mailmap](.mailmap) and [AUTHORS](AUTHORS) files: - -``` -git checkout master -git log --use-mailmap | grep '^Author:' | cut -f2- -d' ' | sort | uniq > AUTHORS -``` - -## Version - -Update the version in `version.go` - -## Changelog - -Edit `CHANGELOG.md`, choose the most important issues and summarize them in multiple groups/paragraphs. Provide links to the mentioned -issues/PRs. At the start include a link to the milestone's closed issues. - -## Git Tag - -``` -git commit -v -a -m "Release version $VERSION" -``` - -Create a signed tag (tags/v) on the `master` branch (for major -releases) or the `support` branch (for minor releases). - -``` -git tag -s -m "Version $VERSION" v$VERSION -``` - -Push the tag: - -``` -git push origin tags/v$VERSION -``` - -**For major releases:** Create a new `support` branch: - -``` -git checkout master -git push - -git checkout -b support/1.0 -git push -u origin support/1.0 -``` - - -## Package Builds - -``` -mkdir $HOME/projects/icinga/packaging -cd $HOME/projects/icinga/packaging -``` - -### RPM Packages - -``` -git clone git@git.icinga.com:icinga/rpm-icingadb.git && cd rpm-icingadb -``` - -### DEB Packages - -``` -git clone git@git.icinga.com:packaging/deb-icingadb.git && cd deb-icingadb -``` - - -### Branch Workflow - -Checkout `master` and create a new branch. - -* For releases use x.x as branch name (e.g. 1.0) -* For releases with revision use x.x.x-n (e.g. 1.0.0-1) - - -### Switch Build Type - -Edit file `.gitlab-ci.yml` and comment variable `ICINGA_BUILD_TYPE` out. - -```yaml -variables: - ... - #ICINGA_BUILD_TYPE: snapshot - ... -``` - -Commit the change. - -``` -git commit -av -m "Switch build type for $VERSION-1" -``` - -#### RPM Release Preparations - -Set the `Version`, `revision` and `%changelog` inside the spec file: - -``` -sed -i "s/Version:.*/Version: $VERSION/g" icingadb.spec - -vim icingadb.spec - -%changelog -* Fri Mar 13 2020 Noah Hilverling 1.0.0-1 -- Release 1.0.1 -``` - -#### DEB and Raspbian Release Preparations - -Update file `debian/changelog` and add at the beginning: - -``` -icingadb (1.0.0-1) icinga; urgency=medium - - * Release 1.0.0 - - -- Noah Hilverling Fri, 13 Mar 2020 10:50:31 +0100 -``` - - -### Release Commit - -Commit the changes and push the branch. - -``` -git commit -av -m "Release $VERSION-1" -git push origin 1.0 -``` - -GitLab will now build snapshot packages based on the tag `v1.0.0` of Icinga DB. - -### Package Tests - -In order to test the created packages you can download a job's artifacts: - -Visit [git.icinga.com](https://git.icinga.com/packaging/rpm-icingadb) -and navigate to the respective pipeline under `CI / CD -> Pipelines`. - -There click on the job you want to download packages from. - -The job's output appears. On the right-hand sidebar you can browse its artifacts. - -Once there, navigate to `build/RPMS/noarch` where you'll find the packages. - -### Release Packages - -To build release packages and upload them to [packages.icinga.com](https://packages.icinga.com) -tag the release commit and push it. - -RPM/DEB: - -``` -git tag -s $VERSION-1 -m "Release v$VERSION-1" -git push origin $VERSION-1 -``` - -Now cherry pick the release commit to `master` so that the changes are transferred back to it. - -**Attention**: Only the release commit. *NOT* the one switching the build type! - - -## Build Infrastructure - -https://git.icinga.com/packaging/rpm-icingadb/pipelines -https://git.icinga.com/packaging/deb-icingadb/pipelines - -* Verify package build changes for this version. -* Test the snapshot packages for all distributions beforehand. - -Once the release repository tags are pushed, release builds -are triggered and automatically published to packages.icinga.com - - -## GitHub Release - -Create a new release for the newly created Git tag: https://github.com/icinga/icingadb/releases - -> Hint: Choose [tags](https://github.com/icinga/icingadb/tags), pick one to edit and -> make this a release. You can also create a draft release. - -The release body should contain a short changelog, with links -into the roadmap, changelog and blogpost. - - -## Post Release - -### Online Documentation - -> Only required for major releases. - -Navigate to `puppet-customer/icinga.git` and do the following steps: - -#### Testing - -``` -git checkout testing && git pull -vim files/var/www/docs/config/icingadb-latest.yml - -git commit -av -m "icinga-web: Update docs for Icinga DB" - -git push -``` - -SSH into the webserver and do a manual Puppet dry run with the testing environment. - -``` -puppet agent -t --environment testing --noop -``` - -Once succeeded, continue with production deployment. - -#### Production - -``` -git checkout master && git pull -git merge testing -git push -``` - -SSH into the webserver and do a manual Puppet run from the production environment (default). - -``` -puppet agent -t -``` - -#### Manual Generation - -SSH into the webserver or ask @bobapple. - -``` -cd /usr/local/icinga-docs-tools && ./build-docs.rb -c /var/www/docs/config/icingadb-latest.yml -``` - -### Announcement - -* Create a new blog post on [icinga.com/blog](https://icinga.com/blog) including a featured image -* Create a release topic on [community.icinga.com](https://community.icinga.com) -* Release email to net-tech & team - -### Project Management - -* Add new minor version on [GitHub](https://github.com/icinga/icingadb/milestones). diff --git a/config.example.yml b/config.example.yml index c6cdbc5a8..61d39339a 100644 --- a/config.example.yml +++ b/config.example.yml @@ -1,25 +1,46 @@ # This is the configuration file for Icinga DB. +# Connection configuration for the database to which Icinga DB synchronizes monitoring data. +# This is also the database used in Icinga DB Web to view and work with the data. +# In high availability setups, all Icinga DB instances must write to the same database. database: - type: mysql + # Database type. Either 'mysql' for MySQL or 'pgsql' for PostgreSQL. + # Defaults to 'mysql'. +# type: mysql + + # Database host or absolute Unix socket path. host: localhost - port: 3306 + + # Database port. By default, the MySQL or PostgreSQL port, depending on the database type. +# port: + + # Database name. database: icingadb + + # Database user. user: icingadb - password: icingadb -# Connection configuration for the Redis where Icinga writes its configuration, state and history. -# This is the same connection as configured in the 'icingadb' feature of the corresponding Icinga node. -# High availability setups require a dedicated Redis server per Icinga node and + # Database password. + password: CHANGEME + +# Connection configuration for the Redis server where Icinga 2 writes its configuration, state and history items. +# This is the same connection as configured in the 'icingadb' feature of the corresponding Icinga 2 node. +# High availability setups require a dedicated Redis server per Icinga 2 node and # therefore a dedicated Icinga DB instance that connects to it. redis: # Redis host or absolute Unix socket path. host: localhost - # Redis port. The Redis server provided by the icingadb-redis package listens on port '6380'. - # If not specified, the standard Redis port '6379' is used instead. - port: 6380 + # Redis port. + # Defaults to '6380' since the Redis server provided by the 'icingadb-redis' package listens on that port. +# port: 6380 + + # Redis password. +# password: +# Icinga DB logs its activities at various severity levels and any errors that occur either +# on the console or in systemd's journal. The latter is used automatically when running under systemd. +# In any case, the default log level is 'info'. logging: # Default logging level. Can be set to 'fatal', 'error', 'warn', 'info' or 'debug'. # If not set, defaults to 'info'. @@ -49,13 +70,18 @@ logging: # runtime-updates: # telemetry: +# Retention is an optional feature to limit the number of days that historical data is available, +# as no historical data is deleted by default. retention: # Number of days to retain full historical data. By default, historical data is retained forever. # history-days: + # Number of days to retain historical data for SLA reporting. By default, it is retained forever. # sla-days: + # Map of history category to number of days to retain its data in order to - # enable retention only for specific categories or to override the number that has been configured in days. + # enable retention only for specific categories or to + # override the number that has been configured in history-days. options: # acknowledgement: # comment: diff --git a/doc/01-About.md b/doc/01-About.md index 3c3e8903c..8211e57c9 100644 --- a/doc/01-About.md +++ b/doc/01-About.md @@ -1,8 +1,23 @@ # Icinga DB -![Icinga DB Context](images/about/icinga-db-in-icinga-context.png) +Icinga DB is a set of components for publishing, synchronizing and +visualizing monitoring data in the Icinga ecosystem, consisting of: -Icinga DB serves as a synchronisation daemon between Icinga 2 (Redis) and Icinga Web 2 (MySQL/MariaDB/PostgreSQL database). -It synchronises configuration, volatile states and history of an Icinga 2 environment using checksums. +* The Icinga DB daemon, which synchronizes monitoring data between a Redis server and a database +* Icinga 2 with its [Icinga DB feature](https://icinga.com/docs/icinga-2/latest/doc/14-features/#icinga-db) enabled, + responsible for publishing the data to the Redis server, i.e. configuration and its runtime updates, check results, + state changes, downtimes, acknowledgements, notifications, and other events such as flapping +* And Icinga Web with the + [Icinga DB Web](https://icinga.com/docs/icinga-db-web) module enabled, + which connects to both Redis and the database to display and work with the most up-to-date data -Icinga DB also supports reading from multiple environments and writing into a single database. +![Icinga DB Architecture](images/icingadb-architecture.png) + +## Installation + +To install Icinga DB see [Installation](02-Installation.md). + +## License + +Icinga DB and the Icinga DB documentation are licensed under the terms of the +GNU General Public License Version 2. diff --git a/doc/02-Installation.md b/doc/02-Installation.md index 83ec8fc97..b4f50c543 100644 --- a/doc/02-Installation.md +++ b/doc/02-Installation.md @@ -1,145 +1,219 @@ -# Installation + +# Installing Icinga DB -## Requirements +The recommended way to install Icinga DB is to use prebuilt packages for +all supported platforms from our official release repository. +Please follow the steps listed for your target operating system, +which guide you through setting up the repository and installing Icinga DB. -* Local Redis (≥6.2) instance (Will be installed during this documentation) -* MySQL (≥5.5), MariaDB (≥10.1), or PostgreSQL (≥9.6): database, user and schema imports (Will be set up during this documentation) +To upgrade an existing Icinga DB installation to a newer version, +see the [Upgrading](04-Upgrading.md) documentation for the necessary steps. -## Setting up Icinga DB +![Icinga DB Daemon](images/icingadb-daemon.png) -### Package Repositories +Before installing Icinga DB, make sure you have installed [Icinga 2](https://icinga.com/docs/icinga-2), +set up a Redis server, and enabled the `icingadb` feature. +The Icinga 2 installation documentation covers all the necessary steps. +Additionally, Icinga offers the `icingadb-redis` package for all supported operating systems, +which ships an up-to-date Redis server version and is pre-configured for the Icinga DB components. -In order to install the latest release candidate, you have to add our `testing` repository as shown below. We assume -that you have our `release` repository already activated. The following commands must be executed with root permissions -unless noted otherwise. +!!! tip -#### RHEL/CentOS/Fedora Repositories + Although Icinga DB can run anywhere in an Icinga environment, + we recommend to install it where the corresponding Icinga 2 node and Redis server is running to + keep latency between the components low. -Make sure you have wget installed. + +## Adding Icinga Package Repository -``` -rpm --import https://packages.icinga.com/icinga.key +The recommended way to install Icinga DB is to use prebuilt packages from our official release repository. -wget https://packages.icinga.com/epel/ICINGA-testing.repo -O /etc/yum.repos.d/ICINGA-testing.repo -``` +!!! tip -#### SLES/OpenSUSE Repositories + If you install Icinga DB on a node that has Icinga 2 or Icinga Web installed via packages, + proceed to [installing the Icinga DB package](#installing-icinga-db-web-package) as + the repository is already configured. -``` -rpm --import https://packages.icinga.com/icinga.key +Here's how to add the official release repository: -zypper ar https://packages.icinga.com/SUSE/ICINGA-testing.repo -zypper ref -``` + + +### Amazon Linux 2 Repository + +!!! info + + A paid repository subscription is required for Amazon Linux 2 repositories. Get more information on + [icinga.com/subscription](https://icinga.com/subscription). -#### Debian/Ubuntu Repositories + Don't forget to fill in the username and password section with appropriate credentials in the local .repo file. -Debian: +```bash +rpm --import https://packages.icinga.com/icinga.key +wget https://packages.icinga.com/subscription/amazon/ICINGA-release.repo -O /etc/yum.repos.d/ICINGA-release.repo +``` + + + +### CentOS Repository + +```bash +rpm --import https://packages.icinga.com/icinga.key +wget https://packages.icinga.com/centos/ICINGA-release.repo -O /etc/yum.repos.d/ICINGA-release.repo ``` + + + + +### Debian Repository + + +```bash apt-get update apt-get -y install apt-transport-https wget gnupg wget -O - https://packages.icinga.com/icinga.key | apt-key add - DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \ - echo "deb https://packages.icinga.com/debian icinga-${DIST}-testing main" > \ - /etc/apt/sources.list.d/${DIST}-icinga-testing.list - echo "deb-src https://packages.icinga.com/debian icinga-${DIST}-testing main" >> \ - /etc/apt/sources.list.d/${DIST}-icinga-testing.list + echo "deb https://packages.icinga.com/debian icinga-${DIST} main" > \ + /etc/apt/sources.list.d/${DIST}-icinga.list + echo "deb-src https://packages.icinga.com/debian icinga-${DIST} main" >> \ + /etc/apt/sources.list.d/${DIST}-icinga.list apt-get update ``` + + + + +### RHEL Repository + +!!! info + + A paid repository subscription is required for RHEL repositories. Get more information on + [icinga.com/subscription](https://icinga.com/subscription). + + Don't forget to fill in the username and password section with appropriate credentials in the local .repo file. + +```bash +rpm --import https://packages.icinga.com/icinga.key +wget https://packages.icinga.com/subscription/rhel/ICINGA-release.repo -O /etc/yum.repos.d/ICINGA-release.repo +``` + + + + +### SLES Repository + +!!! info -Ubuntu: + A paid repository subscription is required for SLES repositories. Get more information on + [icinga.com/subscription](https://icinga.com/subscription). + Don't forget to fill in the username and password section with appropriate credentials in the local .repo file. + +```bash +rpm --import https://packages.icinga.com/icinga.key + +zypper ar https://packages.icinga.com/subscription/sles/ICINGA-release.repo +zypper ref ``` + + + + +### Ubuntu Repository + + +```bash apt-get update apt-get -y install apt-transport-https wget gnupg wget -O - https://packages.icinga.com/icinga.key | apt-key add - . /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \ - echo "deb https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" > \ - /etc/apt/sources.list.d/${DIST}-icinga-testing.list - echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" >> \ - /etc/apt/sources.list.d/${DIST}-icinga-testing.list + echo "deb https://packages.icinga.com/ubuntu icinga-${DIST} main" > \ + /etc/apt/sources.list.d/${DIST}-icinga.list + echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST} main" >> \ + /etc/apt/sources.list.d/${DIST}-icinga.list apt-get update ``` + -### Installing Icinga DB - -RHEL/CentOS 8/Fedora: - -``` -dnf install icingadb -systemctl enable icingadb -systemctl start icingadb -``` +## Installing Icinga DB Package -RHEL/CentOS 7: +Use your distribution's package manager to install the `icingadb` package as follows: -``` + + +#### Amazon Linux 2 + +```bash yum install icingadb -systemctl enable icingadb -systemctl start icingadb ``` + -SUSE: + + +#### CentOS + +!!! info -``` -zypper install icingadb -``` - -Debian/Ubuntu: + Note that installing Icinga DB is only supported on CentOS 7 as CentOS 8 is EOL. +```bash +yum install icingadb ``` + + + + +#### Debian / Ubuntu + +```bash apt-get install icingadb ``` + -### Installing Icinga DB Redis - -RHEL/CentOS 8/Fedora: - -``` -dnf install icingadb-redis + +#### RHEL 8 or Later -systemctl enable icingadb-redis -systemctl start icingadb-redis +```bash +dnf install icingadb ``` -RHEL/CentOS 7: +#### RHEL 7 +```bash +yum install icingadb ``` -yum install icingadb-redis + -systemctl enable icingadb-redis -systemctl start icingadb-redis + + +#### SLES + +```bash +zypper install icingadb ``` + -SUSE: +## Setting up the Database -``` -zypper install icingadb-redis +A MySQL (≥5.5), MariaDB (≥10.1), or PostgreSQL (≥9.6) database is required to run Icinga DB. +Please follow the steps listed for your target database, +which guide you through setting up the database and user and importing the schema. -systemctl enable icingadb-redis -systemctl start icingadb-redis -``` +![Icinga DB Database](images/icingadb-database.png) -Debian/Ubuntu: +!!! info -``` -apt-get install icingadb-redis -``` + In high availability setups, all Icinga DB instances must write to the same database. -### Setting up the Database +### Setting up a MySQL or MariaDB Database -A MySQL/MariaDB or PostgreSQL database is required. - -#### MySQL/MariaDB - -Note that if you're using a version of MySQL < 5.7 or MariaDB < 10.2, the following server options must be set: +If you use a version of MySQL < 5.7 or MariaDB < 10.2, the following server options must be set: ``` innodb_file_format=barracuda @@ -153,17 +227,17 @@ Set up a MySQL database for Icinga DB: # mysql -u root -p CREATE DATABASE icingadb; -CREATE USER 'icingadb'@'localhost' IDENTIFIED BY 'icingadb'; +CREATE USER 'icingadb'@'localhost' IDENTIFIED BY 'CHANGEME'; GRANT ALL ON icingadb.* TO 'icingadb'@'localhost'; ``` -After creating the database, you can import the Icinga DB schema using the following command: +After creating the database, import the Icinga DB schema using the following command: ``` mysql -u root -p icingadb +### Setting up a PostgreSQL Database Set up a PostgreSQL database for Icinga DB: @@ -175,7 +249,7 @@ createdb -E UTF8 --locale en_US.UTF-8 -T template0 -O icingadb icingadb psql icingadb <<<'CREATE EXTENSION IF NOT EXISTS citext;' ``` -The CREATE EXTENSION command requires the postgresql-contrib package. +The `CREATE EXTENSION` command requires the `postgresql-contrib` package. Edit `pg_hba.conf`, insert the following before everything else: @@ -185,41 +259,61 @@ host all icingadb 0.0.0.0/0 md5 host all icingadb ::/0 md5 ``` -To apply those changes, run `systemctl reload postgresql`. +To apply these changes, run `systemctl reload postgresql`. -After creating the database you can import the Icinga DB schema using the -following command. Enter the password when asked. +After creating the database, import the Icinga DB schema using the following command: ``` psql -U icingadb icingadb < /usr/share/icingadb/schema/pgsql/schema.sql ``` -### Running Icinga DB - -Foreground: - -``` -icingadb --config /etc/icingadb/config.yml -``` - -Systemd service: - -``` -systemctl enable icingadb -systemctl start icingadb -``` - -### Enable remote Redis connections - -By default `icingadb-redis` listens only on `127.0.0.1`. If you want to change that, e.g. for Icinga Web 2 or Icinga 2 -running on another node, just change `bind 127.0.0.1 ::1` and `protected-mode yes` -in `/etc/icingadb-redis/icingadb-redis.conf` to the interface you want to use and to `protected-mode no`. - -> WARNING: By default, Redis does not have any authentication that prevents others from accessing it. -> If you open Redis to an external interface, make sure that you set up appropriate firewall rules or configure TLS -> with certificate authentication on Redis and its consumers, i.e. Icinga 2, Icinga DB and Icinga Web 2. - -### Icinga DB Web - -Consult the [Icinga DB Web documentation](https://icinga.com/docs/icingadb/latest/icingadb-web/doc/02-Installation/) on -how to connect Icinga Web 2 with Icinga DB. +## Configuring Icinga DB + +Icinga DB installs its configuration file to `/etc/icingadb/config.yml`, +pre-populating most of the settings for a local setup. Before running Icinga DB, +adjust the Redis and database credentials and, if necessary, the connection configuration. +The configuration file explains general settings. +All available settings can be found under [Configuration](03-Configuration.md). + +## Running Icinga DB + +The `icingadb` package automatically installs the necessary systemd unit files to run Icinga DB. +Please run the following command to enable and start its service: + +```bash +systemctl enable --now icingadb +``` + +## Installing Icinga DB Web + +With Icinga 2, Redis, Icinga DB and the database fully set up, it is now time to install Icinga DB Web, +which connects to both Redis and the database to display and work with the monitoring data. + +![Icinga DB Web](images/icingadb-web.png) + +The Icinga DB Web package is also included in the Icinga repository, and since it is already set up, +you have completed the instructions here and can proceed to + +[installing Icinga DB Web on Amazon Linux](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/01-Amazon-Linux/#installing-icinga-db-web-package), + + +[installing Icinga DB Web on CentOS](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/02-CentOS/#installing-icinga-db-web-package), + + +[installing Icinga DB Web on Debian](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/03-Debian/#installing-icinga-db-web-package), + + +[installing Icinga DB Web on RHEL](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/04-RHEL/#installing-icinga-db-web-package), + + +[installing Icinga DB Web on SLES](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/05-SLES/#installing-icinga-db-web-package), + + +[installing Icinga DB Web on Ubuntu](https://icinga.com/docs/icinga-db-web/latest/doc/02-Installation/06-Ubuntu/#installing-icinga-db-web-package), + +which will also guide you through the setup of the Icinga Web PHP framework, +which is required to run the Icinga DB web module. +Below is a preview of how the interface visualizes monitoring data and also supports dark and light mode: + +![Icinga DB Web](images/icingadb-dashboard.png) + diff --git a/doc/02-Installation.md.d/01-Amazon-Linux.md b/doc/02-Installation.md.d/01-Amazon-Linux.md new file mode 100644 index 000000000..cb5e161c8 --- /dev/null +++ b/doc/02-Installation.md.d/01-Amazon-Linux.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on Amazon Linux + + diff --git a/doc/02-Installation.md.d/02-CentOS.md b/doc/02-Installation.md.d/02-CentOS.md new file mode 100644 index 000000000..343d64062 --- /dev/null +++ b/doc/02-Installation.md.d/02-CentOS.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on CentOS + + diff --git a/doc/02-Installation.md.d/03-Debian.md b/doc/02-Installation.md.d/03-Debian.md new file mode 100644 index 000000000..2cea527dd --- /dev/null +++ b/doc/02-Installation.md.d/03-Debian.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on Debian + + diff --git a/doc/02-Installation.md.d/04-RHEL.md b/doc/02-Installation.md.d/04-RHEL.md new file mode 100644 index 000000000..0dd059afb --- /dev/null +++ b/doc/02-Installation.md.d/04-RHEL.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on RHEL + + diff --git a/doc/02-Installation.md.d/05-SLES.md b/doc/02-Installation.md.d/05-SLES.md new file mode 100644 index 000000000..9af0e4869 --- /dev/null +++ b/doc/02-Installation.md.d/05-SLES.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on SLES + + diff --git a/doc/02-Installation.md.d/06-Ubuntu.md b/doc/02-Installation.md.d/06-Ubuntu.md new file mode 100644 index 000000000..0ce05e011 --- /dev/null +++ b/doc/02-Installation.md.d/06-Ubuntu.md @@ -0,0 +1,3 @@ +# Installing Icinga DB on Ubuntu + + diff --git a/doc/03-Configuration.md b/doc/03-Configuration.md index 14f58e594..af9278338 100644 --- a/doc/03-Configuration.md +++ b/doc/03-Configuration.md @@ -1,84 +1,93 @@ -# Configuration - -## Overview +# Configuration The configuration is stored in `/etc/icingadb/config.yml`. See [config.example.yml](../config.example.yml) for an example configuration. -## Redis Configuration - -Configuration of the Redis that Icinga writes to. - -Option | Description --------------------------|----------------------------------------------- -host | **Required.** Redis host or absolute Unix socket path. -port | **Optional.** Redis port. Defaults to `6379`. Specify `6380` if using the `icingadb-redis` package. -password | **Optional.** The password to use. -tls | **Optional.** Whether to use TLS. -cert | **Optional.** Path to TLS client certificate. -key | **Optional.** Path to TLS private key. -ca | **Optional.** Path to TLS CA certificate. -insecure | **Optional.** Whether not to verify the peer. - -## Database Configuration - -Configuration of the database used by Icinga DB. - -Option | Description --------------------------|----------------------------------------------- -type | **Optional.** Either `mysql` (default) or `pgsql`. -host | **Required.** Database host or absolute Unix socket path. -port | **Required.** Database port. -database | **Required.** Database database. -user | **Required.** Database username. -password | **Required.** Database password. -tls | **Optional.** Whether to use TLS. -cert | **Optional.** Path to TLS client certificate. -key | **Optional.** Path to TLS private key. -ca | **Optional.** Path to TLS CA certificate. -insecure | **Optional.** Whether not to verify the peer. - -## Logging Configuration +## Redis Configuration + +Connection configuration for the Redis server where Icinga 2 writes its configuration, state and history items. +This is the same connection as configured in the +[Icinga DB feature](https://icinga.com/docs/icinga-2/latest/doc/14-features/#icinga-db) of +the corresponding Icinga 2 node. High availability setups require a dedicated Redis server per Icinga 2 node and +therefore a dedicated Icinga DB instance that connects to it. + +| Option | Description | +|----------|------------------------------------------------------------------------------------------------------------------------------------| +| host | **Required.** Redis host or absolute Unix socket path. | +| port | **Optional.** Redis port. Defaults to `6380` since the Redis server provided by the `icingadb-redis` package listens on that port. | +| password | **Optional.** The password to use. | +| tls | **Optional.** Whether to use TLS. | +| cert | **Optional.** Path to TLS client certificate. | +| key | **Optional.** Path to TLS private key. | +| ca | **Optional.** Path to TLS CA certificate. | +| insecure | **Optional.** Whether not to verify the peer. | + +## Database Configuration + +Connection configuration for the database to which Icinga DB synchronizes monitoring data. +This is also the database used in +[Icinga DB Web](https://icinga.com/docs/icinga-db/latest/icinga-db-web/doc/01-About/) to view and work with the data. +In high availability setups, all Icinga DB instances must write to the same database. + +| Option | Description | +|----------|--------------------------------------------------------------------------------------------------------| +| type | **Optional.** Either `mysql` (default) or `pgsql`. | +| host | **Required.** Database host or absolute Unix socket path. | +| port | **Optional.** Database port. By default, the MySQL or PostgreSQL port, depending on the database type. | +| database | **Required.** Database name. | +| user | **Required.** Database username. | +| password | **Optional.** Database password. | +| tls | **Optional.** Whether to use TLS. | +| cert | **Optional.** Path to TLS client certificate. | +| key | **Optional.** Path to TLS private key. | +| ca | **Optional.** Path to TLS CA certificate. | +| insecure | **Optional.** Whether not to verify the peer. | + +## Logging Configuration Configuration of the logging component used by Icinga DB. -Option | Description --------------------------|----------------------------------------------- -level | **Optional.** Specifies the default logging level. Can be set to `fatal`, `error`, `warn`, `info` or `debug`. Defaults to `info`. -output | **Optional.** Configures the logging output. Can be set to `console` (stderr) or `systemd-journald`. If not set, logs to systemd-journald when running under systemd, otherwise stderr. -interval | **Optional.** Interval for periodic logging defined as [duration string](#duration-string). Defaults to `"20s"`. -options | **Optional.** Map of component name to logging level in order to set a different logging level for each component instead of the default one. See [logging components](#logging-components) for details. - -### Logging Components - -Component | Description --------------------------|----------------------------------------------- -config-sync | Config object synchronization between Redis and MySQL. -database | Database connection status and queries. -dump-signals | Dump signals received from Icinga. -heartbeat | Icinga heartbeats received through Redis. -high-availability | Manages responsibility of Icinga DB instances. -history-sync | Synchronization of history entries from Redis to MySQL. -overdue-sync | Calculation and synchronization of the overdue status of checkables. -redis | Redis connection status and queries. -retention | Deletes historical data that exceed their configured retention period. -runtime-updates | Runtime updates of config objects after the initial config synchronization. -telemetry | Reporting of Icinga DB status to Icinga 2 via Redis (for monitoring purposes). - -### Duration String - -A duration string is a sequence of decimal numbers and a unit suffix, such as `"20s"`. -Valid units are `"ms"`, `"s"`, `"m"` and `"h"`. - -## Retention - -By default, no historical data is deleted, which means that the longer the data is retained, the more disk space is required to store it. -History retention is an optional feature that allows you to limit the number of days that historical data is available for each history category. +| Option | Description | +|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| level | **Optional.** Specifies the default logging level. Can be set to `fatal`, `error`, `warn`, `info` or `debug`. Defaults to `info`. | +| output | **Optional.** Configures the logging output. Can be set to `console` (stderr) or `systemd-journald`. If not set, logs to systemd-journald when running under systemd, otherwise stderr. | +| interval | **Optional.** Interval for periodic logging defined as [duration string](#duration-string). Defaults to `"20s"`. | +| options | **Optional.** Map of component name to logging level in order to set a different logging level for each component instead of the default one. See [logging components](#logging-components) for details. | + +### Logging Components + +| Component | Description | +|-------------------|--------------------------------------------------------------------------------| +| config-sync | Config object synchronization between Redis and MySQL. | +| database | Database connection status and queries. | +| dump-signals | Dump signals received from Icinga. | +| heartbeat | Icinga heartbeats received through Redis. | +| high-availability | Manages responsibility of Icinga DB instances. | +| history-sync | Synchronization of history entries from Redis to MySQL. | +| overdue-sync | Calculation and synchronization of the overdue status of checkables. | +| redis | Redis connection status and queries. | +| retention | Deletes historical data that exceed their configured retention period. | +| runtime-updates | Runtime updates of config objects after the initial config synchronization. | +| telemetry | Reporting of Icinga DB status to Icinga 2 via Redis (for monitoring purposes). | + +## Retention + +By default, no historical data is deleted, which means that the longer the data is retained, +the more disk space is required to store it. History retention is an optional feature that allows to +limit the number of days that historical data is available for each history category. There are separate options for the full history tables used to display history information in the web interface and -SLA tables which store the minimal information required for SLA reporting, allowing to keep this information for longer with a smaller storage footprint. +SLA tables which store the minimal information required for SLA reporting, +allowing to keep this information for longer with a smaller storage footprint. | Option | Description | |--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | history-days | **Optional.** Number of days to retain historical data for all history categories. Use `options` in order to enable retention only for specific categories or to override the retention days configured here. | | sla-days | **Optional.** Number of days to retain historical data for SLA reporting. | | options | **Optional.** Map of history category to number of days to retain its data. Available categories are `acknowledgement`, `comment`, `downtime`, `flapping`, `notification`, `sla` and `state`. | + +## Appendix + +### Duration String + +A duration string is a sequence of decimal numbers and a unit suffix, such as `"20s"`. +Valid units are `"ms"`, `"s"`, `"m"` and `"h"`. diff --git a/doc/04-Upgrading.md b/doc/04-Upgrading.md index 37f40844e..ebe9c2271 100644 --- a/doc/04-Upgrading.md +++ b/doc/04-Upgrading.md @@ -1,4 +1,18 @@ -# Upgrading Icinga DB +# Upgrading Icinga DB + +Specific version upgrades are described below. Please note that version upgrades are incremental. +If you are upgrading across multiple versions, make sure to follow the steps for each of them. + +## Upgrading to Icinga DB v1.0 + +**Requirements** + +* You need at least Icinga 2 version 2.13.4 to run Icinga DB v1.0.0. + +**Database Schema** + +* For MySQL databases, please apply the `1.0.0.sql` upgrade script. + For package installations, you can find this file at `/usr/share/doc/icingadb/schema/mysql/upgrades/`. ## Upgrading to Icinga DB RC2 diff --git a/doc/05-Distributed-Setups.md b/doc/05-Distributed-Setups.md new file mode 100644 index 000000000..a324a65ee --- /dev/null +++ b/doc/05-Distributed-Setups.md @@ -0,0 +1,85 @@ +# Distributed Setups + +## High Availability + +High availability functionality is built directly into Icinga DB and +can be deployed without additional third-party components. + +![Icinga DB HA](images/icingadb-ha.png) + +First, you need an Icinga 2 high availability setup with two master nodes, such as described +[here](https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring#high-availability-master-with-agents). + +Each of the master nodes must have the Icinga DB feature enabled and +have their own dedicated Redis server set up for it, so that each node writes the monitoring data separately. +The setup steps per node are no different from a single node setup and can be found in the +[Icinga 2 installation documentation](https://icinga.com/docs/icinga-2/latest/doc/02-installation). +Each Redis server will always have the complete data available as long as +its corresponding Icinga 2 master is running and writing to its Redis. +This is because the Icinga 2 master nodes synchronize their data and events with each other as long as +they are connected, +and each takes over the entire configuration in case the other node or their connection to each other fails. + +For each Redis server you need to set up its own dedicated Icinga DB instance that connects to it, +but the Icinga DB instances must write to the same database, which of course can be replicated or a cluster. +So the steps from the standard +[Icinga DB installation documentation](https://icinga.com/docs/icinga-db/latest/doc/02-installation) +can be followed. However, as mentioned, the database only needs to be set up once. + +All in all, an Icinga DB HA environment involves setting up two Icinga 2 master nodes, two Redis servers, +two Icinga DB instances and one database. + +Please read the note about the [environment ID](#environment-id), +which is common to all Icinga DB components and generated by Icinga 2's Icinga DB feature. + +There is only one Icinga DB instance at a time, +which is responsible for performing database operations in the following areas: + +* Synchronizing configuration, also across Icinga 2 restarts. +* Performing configuration runtime updates made via the Icinga 2 API. +* Updating recent host and service states. +* Flagging hosts and services that are overdue for updates. +* Deleting history items that have exceeded their configured retention time. + +However, both Icinga DB instances write all events relevant to the history of hosts and services to the database. +This way, no data is lost if one of the Icinga 2 masters is unavailable or if their connection to each other fails. + +Which Icinga DB instance is responsible is determined by a specific database operation that +can only be performed by one instance first. +In the case of concurrent operations, simply put, only one wins via a locking mechanism. +Of course, this is only true if the environment is healthy. +Icinga DB is not trying to be responsible if its corresponding Redis server is unavailable or +Icinga 2 is not writing data to Redis. +If Icinga 2 or Redis become unavailable for more than 60 seconds, +Icinga DB releases responsibility so the other instance can take over. + +## Multiple Environments + +Icinga DB supports synchronization of monitoring data from multiple different Icinga environments into +a single database. This allows Icinga DB Web to provide a centralized view of the data. +Although everything is prepared in Icinga DB, there is no full support in Icinga DB Web yet. +As soon as it is ready, the documentation will be adapted and the feature will be explained in more detail. + +![Icinga DB Envs](images/icingadb-envs.png) + +## Environment ID + +!!! important + + Icinga 2 generates a unique environment ID from its CA certificate when it is first started with the + Icinga DB feature enabled. The ID is used in all Icinga DB components to separate data from + multiple different environments and is written to the file `/var/lib/icinga2/icingadb.env` by Icinga 2. + + It is strictly recommended not to change this ID afterwards, as all data would be resynchronized and + the old ones would remain in the database, resulting in duplicate data. As long as the file remains, + Icinga 2 will not regenerate the environment ID. This is also true if the CA is changed to avoid duplicate data. + Special care should be taken if you add or redeploy the master node(s) and + as a result or over time the CA has changed, which would result in a new environment ID. + If this is the case, make sure you copy the file between your Icinga 2 masters to get a consistent ID. + + For high-availability setups, it is a good idea to enable the Icinga DB feature on the secondary master after + you have successfully connected from/to the primary master so that the certificates are set up properly. + The secondary master will then generate the same environment ID since it is working with the same CA certificate. + + In any case make sure that the file `/var/lib/icinga2/icingadb.env` does not change over time and + is the same on all Icinga 2 master nodes per environment. diff --git a/doc/images/about/icinga-db-in-icinga-context.png b/doc/images/about/icinga-db-in-icinga-context.png deleted file mode 100644 index ed84463d3..000000000 Binary files a/doc/images/about/icinga-db-in-icinga-context.png and /dev/null differ diff --git a/doc/images/icingadb-architecture.png b/doc/images/icingadb-architecture.png new file mode 100644 index 000000000..3d55ff757 Binary files /dev/null and b/doc/images/icingadb-architecture.png differ diff --git a/doc/images/icingadb-daemon.png b/doc/images/icingadb-daemon.png new file mode 100644 index 000000000..de3f4c733 Binary files /dev/null and b/doc/images/icingadb-daemon.png differ diff --git a/doc/images/icingadb-dashboard.png b/doc/images/icingadb-dashboard.png new file mode 100644 index 000000000..521eb3957 Binary files /dev/null and b/doc/images/icingadb-dashboard.png differ diff --git a/doc/images/icingadb-database.png b/doc/images/icingadb-database.png new file mode 100644 index 000000000..c300095b5 Binary files /dev/null and b/doc/images/icingadb-database.png differ diff --git a/doc/images/icingadb-envs.png b/doc/images/icingadb-envs.png new file mode 100644 index 000000000..e9938d8b3 Binary files /dev/null and b/doc/images/icingadb-envs.png differ diff --git a/doc/images/icingadb-ha.png b/doc/images/icingadb-ha.png new file mode 100644 index 000000000..a86b6a038 Binary files /dev/null and b/doc/images/icingadb-ha.png differ diff --git a/doc/images/icingadb-web.png b/doc/images/icingadb-web.png new file mode 100644 index 000000000..05a3e3185 Binary files /dev/null and b/doc/images/icingadb-web.png differ diff --git a/pkg/config/redis.go b/pkg/config/redis.go index 0447c458e..38571e3c8 100644 --- a/pkg/config/redis.go +++ b/pkg/config/redis.go @@ -20,7 +20,7 @@ import ( // Redis defines Redis client configuration. type Redis struct { Host string `yaml:"host"` - Port int `yaml:"port" default:"6379"` + Port int `yaml:"port" default:"6380"` Password string `yaml:"password"` TlsOptions TLS `yaml:",inline"` Options icingaredis.Options `yaml:"options"`