WIP: Implement support for juju secrets and create MySQLCharmBase for…

… common charm code in vm and k8s

Update existing secrets instead of creating a new one every time

DPE-2154 profile configuration support (#240)

* initial profile configuration support

* going for juju latest

* bump lib

* key has default value

* doing renovate's work

* some test coverage

* temporary pin

* reflect value for testing profile

Refactor get_member_state (#242)

* retrieves all entries then filter

Some cases (pod rescheduling) the member_id is not populated,
failing the state query with the where clause

* explaining comments

Update charm libs + fix failing unit tests

Bump mysql charmlib patch to 37

Changes resulting from testing with juju 3.1.6

Fail installation if snap already installed (#235)

Avoid overriding snap if mysql charm & mysql-router charm installed on same machine

Context: https://chat.canonical.com/canonical/pl/b8e1daeskjrejxtryjuxuwi9ua

Disable codecov GitHub annotations (#245)

DPE-1979 Use /etc/hosts as a hostname resolution of nodes in the cluster (#237)

* WIP: Use /etc/hosts as a hostname resolution of nodes in the cluster

* Add IP address observer + address PR feedback

* Fix failing unit tests

* Address remaining PR feedback

* Fix failing unit test

* Send SIGTERM instead of SIGINT to terminate the ip address observer process

* Update cos_agent lib to v0.4

* Fix failing unit test

* Update S3 charmlib to v0.3

* Update data_interfaces charmlib to v0.14

DPE-1511 Autogenerate database and username in legacy mysql if not specified in config (#222)

* Auto generate username and database if not specified in config for the mysql legacy relation

* Clean up username and database from databag upon relation broken

* Rework the handling of usernames and databases

* Handle config changes other than username and database

* Change error log into an info log

* Update data_interfaces charm lib to v0.13

* Refactor legacy mysql relation with feedback in mind

* Account for empty strings as config values

* Update data_interfaces charm lib to v0.16

* Update s3 charm lib to v0.4

* Address PR feedback

* Address minor nit in feedback

* Avoid using .get() where unnecessary

* Update juju pin to 2.9.43.0 and pin macaroonbakery to 1.3.1 in tox

* Do not run event handler to update /etc/hosts if all passwords are not set

Add CODEOWNERS (#247)

Required for self-hosted runners

Fix incorrect tls integration test + some secret related changes in tls charmlib

Move relevant event handler observers into the mysql charm base class

Use secrets caching and address PR feedback

Update cos_agent charm lib to v0.5

Pin python dependencies with poetry (#192)

Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

Update keystone charm to yoga/stable in series jammy for shared_db integration test (#260)

Use block_until instead of wait_for_idle in shared_db integration test

Update cos_agent charm lib to v0.5

Add `poetry lock` commands to lint and format (#257)

Lint: `poetry lock --check` verifies that poetry.lock is valid for `pyproject.toml`
Format: `poetry lock --no-update` adds any changes in `pyproject.toml` to `poetry.lock` without updated locked versions

Address refresh from juju <= 3.1.4 to juju >= 3.1.5

trivial fix (#261)

port from k8s and fixes for constrained memory (#253)

* port from k8s and fixes for constrained memory

fix function call

typing fixes

updates cos-agent lib

Pin python dependencies with poetry (#192)

* Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

* bump lib

* fix for dpe-2274

* streamlined test to avoiding timeout

---------

Co-authored-by: Carl Csaposs <[email protected]>

Use snap with ppa sources + other misc snap related improvements (#249)

Fix lint warnings and failing unit tests

Update cos_agent charmlib to v0.5

Only start mysqld-exporter on cos relation created and stop it on relation broken

Expect mysqld-exporter to be disabled by default for replication integration tests

Add missing await statements in exporter integration test

Use retry to determine if mysqld-exporeter service has started

Set monitoring username and password for exporter in integration test

Fix how the username and password were being set for the exporter in the integration test

Run format

Sleep to wait until mysqld-exporter started

Run format

Address PR feedback

Address PR feedback

Address stylistic PR feedback

Update mysql charm libpatch to v0.38

.github/CODEOWNERS

@@ -0,0 +1 @@
+** @canonical/data-platform-mysql

.github/codecov.yml

@@ -0,0 +1,2 @@
+github_checks:
+  annotations: false

.github/renovate.json5

@@ -0,0 +1,90 @@
+{
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  "extends": [
+    "config:recommended",
+    ":disableRateLimiting",
+    ":noUnscheduledUpdates"
+  ],
+  "schedule": ["after 1am and before 2am every weekday"],
+  "lockFileMaintenance": {
+    "enabled": true,
+    "schedule": ["after 1am and before 2am every weekday"]
+  },
+  "timezone": "Etc/UTC",
+  "enabledManagers": ["poetry", "github-actions", "regex"],
+  "packageRules": [
+    // Later rules override earlier rules
+    {
+      "matchManagers": ["poetry"],
+      // Renovate uses "dependencies" instead of "main" for top-level dependency group
+      "matchDepTypes": ["dependencies"],
+      "rangeStrategy": "bump",
+      "commitMessagePrefix": "[charm]",
+      "groupName": "charm dependencies"
+    },
+    {
+      "matchManagers": ["poetry"],
+      "matchDepTypes": ["charm-libs"],
+      "rangeStrategy": "in-range-only",
+      "commitMessagePrefix": "[charm lib]",
+      "groupName": "charm lib dependencies"
+    },
+    {
+      "matchManagers": ["poetry"],
+      "matchDepTypes": ["format", "lint", "unit", "integration"],
+      "rangeStrategy": "bump",
+      "commitMessagePrefix": "[python ci]",
+      "groupName": "Python CI dependencies"
+    },
+    // MySQL 8.X does not follow semantic versioning (e.g. 8.0.1 -> 8.0.2 can include a breaking change)
+    // Therefore, use a separate Renovate group so that it has a separate PR
+    {
+      "matchManagers": ["poetry"],
+      "matchPackageNames": ["mysql-connector-python"],
+      "groupName": "MySQL Connector/Python"
+    },
+    // Group data-platform-workflows Python package & workflow updates into the same PR
+    {
+      "matchManagers": ["poetry"],
+      "matchPackageNames": ["canonical/data-platform-workflows"],
+      // Workaround for https://github.com/renovatebot/renovate/discussions/23628
+      "versioning": "semver",
+      "groupName": "data-platform-workflows",
+      // Workaround: data-platform-workflows Python packages use git tags instead of pyproject.toml
+      // for versioning. Therefore, Renovate will always think an update is a major version update.
+      "separateMajorMinor": false
+    },
+    {
+      "matchManagers": ["github-actions"],
+      "matchPackageNames": ["canonical/data-platform-workflows"],
+      "groupName": "data-platform-workflows",
+      // Workaround: data-platform-workflows Python packages use git tags instead of pyproject.toml
+      // for versioning. Therefore, Renovate will always think an update is a major version update.
+      // Since we want packages to be updated alongside workflows (actions), we must disable
+      // separate major PRs for workflows as well.
+      "separateMajorMinor": false
+    },
+    {
+      "matchManagers": ["regex"],
+      "matchPackageNames": ["juju/juju"],
+      "groupName": "Juju agents"
+    },
+    // Disable major version updates for Juju agent (they should be handled manually)
+    {
+      "matchManagers": ["regex"],
+      "matchPackageNames": ["juju/juju"],
+      "matchUpdateTypes": ["major"],
+      "enabled": false
+    },
+  ],
+  "regexManagers": [
+    {
+      "fileMatch": ["^\\.github/workflows/[^/]+\\.ya?ml$"],
+      "matchStrings": ["\"--agent-version[= ](?<currentValue>.*?)\" +# renovate: pin-major"],
+      "depNameTemplate": "juju/juju",
+      "extractVersionTemplate": "^juju-(?<version>.*)$",
+      "datasourceTemplate": "github-releases",
+      "versioningTemplate": "semver"
+    }
+  ]
+}

.github/workflows/ci.yaml

@@ -16,7 +16,7 @@ on:
 jobs:
   lint:
     name: Lint
-    uses: canonical/data-platform-workflows/.github/workflows/lint.yaml@v2
+    uses: canonical/data-platform-workflows/.github/workflows/lint.yaml@v4.1.0
 
   unit-test:
     name: Unit tests
@@ -25,9 +25,10 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v3
-      - name: Install tox
-        # TODO: Consider replacing with custom image on self-hosted runner OR pinning version
-        run: python3 -m pip install tox
+      - name: Install tox & poetry
+        run: |
+          pipx install tox
+          pipx install poetry
       - name: Run tests
         run: tox run -e unit
       - name: Upload Coverage to Codecov
@@ -52,7 +53,9 @@ jobs:
 
   build:
     name: Build charms
-    uses: canonical/data-platform-workflows/.github/workflows/build_charms_with_cache.yaml@v2
+    uses: canonical/data-platform-workflows/.github/workflows/[email protected]
+    with:
+      charmcraft-snap-revision: 1349  # version 2.3.0
 
   collect-integration-tests:
     name: Collect integration test groups
@@ -63,9 +66,10 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v3
-      - name: Install tox
-        # TODO: Consider replacing with custom image on self-hosted runner OR pinning version
-        run: python3 -m pip install tox
+      - name: Install tox & poetry
+        run: |
+          pipx install tox
+          pipx install poetry
       - name: Select test stability level
         id: select-test-stability
         run: |
@@ -103,12 +107,16 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v3
+      - name: Install tox & poetry
+        run: |
+          pipx install tox
+          pipx install poetry
       - name: Setup operator environment
         # TODO: Replace with custom image on self-hosted runner
         uses: charmed-kubernetes/actions-operator@main
         with:
           provider: lxd
-          bootstrap-options: "--agent-version 2.9.42"  # renovate: latest
+          bootstrap-options: "--agent-version 2.9.43"  # renovate: pin-major
       - name: Download packed charm(s)
         uses: actions/download-artifact@v3
         with:
@@ -125,7 +133,7 @@ jobs:
             echo "mark_expression=not unstable" >> "$GITHUB_OUTPUT"
           fi
       - name: Run integration tests
-        run: tox run -e integration -- "${{ matrix.groups.path_to_test_file }}" --group ${{ matrix.groups.group_number }} -m '${{ steps.select-test-stability.outputs.mark_expression }}' --mysql-charm-series="${{ matrix.ubuntu-versions.series }}" --mysql-charm-bases-index="${{ matrix.ubuntu-versions.bases-index }}"
+        run: tox run -e integration -- "${{ matrix.groups.path_to_test_file }}" --group="${{ matrix.groups.group_number }}" -m '${{ steps.select-test-stability.outputs.mark_expression }}' --mysql-charm-series="${{ matrix.ubuntu-versions.series }}" --mysql-charm-bases-index="${{ matrix.ubuntu-versions.bases-index }}"
         env:
           AWS_ACCESS_KEY: ${{ matrix.groups.path_to_test_file != 'tests/integration/test_backups.py' || secrets.AWS_ACCESS_KEY }}
           AWS_SECRET_KEY: ${{ matrix.groups.path_to_test_file != 'tests/integration/test_backups.py' || secrets.AWS_SECRET_KEY }}

.github/workflows/release.yaml

@@ -32,14 +32,16 @@ jobs:
 
   build:
     name: Build charm
-    uses: canonical/data-platform-workflows/.github/workflows/build_charm_without_cache.yaml@v2
+    uses: canonical/data-platform-workflows/.github/workflows/[email protected]
+    with:
+      charmcraft-snap-revision: 1349  # version 2.3.0
 
   release:
     name: Release charm
     needs:
       - ci-tests
       - build
-    uses: canonical/data-platform-workflows/.github/workflows/release_charm.yaml@v2
+    uses: canonical/data-platform-workflows/.github/workflows/release_charm.yaml@v4.1.0
     with:
       channel: 8.0/edge
       artifact-name: ${{ needs.build.outputs.artifact-name }}

.github/workflows/sync_issue_to_jira.yaml

@@ -9,7 +9,7 @@ on:
 jobs:
   sync:
     name: Sync GitHub issue to Jira
-    uses: canonical/data-platform-workflows/.github/workflows/sync_issue_to_jira.yaml@v2
+    uses: canonical/data-platform-workflows/.github/workflows/sync_issue_to_jira.yaml@v4.1.0
     with:
       jira-base-url: https://warthogs.atlassian.net
       jira-project-key: DPE

.gitignore

@@ -1,3 +1,5 @@
+/requirements.txt
+/requirements-last-build.txt
 *.charm
 *.py[cod]
 .coverage

CONTRIBUTING.md

@@ -20,12 +20,18 @@ this operator.
   the `main` branch. This also avoids merge commits and creates a linear Git commit history.
 
 ## Developing
+Install `tox` and `poetry`
+```shell
+python3 -m pip install --user pipx
+python3 -m pipx ensurepath
+pipx install tox
+pipx install poetry
+```
 
-You can create an environment for development with `tox`:
+You can create an environment for development:
 
 ```shell
-tox devenv -e integration
-source venv/bin/activate
+poetry install
 ```
 
 ### Testing
@@ -43,7 +49,7 @@ tox                      # runs 'lint' and 'unit' environments
 Build the charm in this git repository using:
 
 ```shell
-charmcraft pack
+tox run -e build
 ```
 
 ### Deploy

charmcraft.yaml

@@ -14,6 +14,13 @@ bases:
         channel: "22.04"
 parts:
   charm:
+    override-pull: |
+      craftctl default
+      if [[ ! -f requirements.txt ]]
+      then
+          echo 'ERROR: Use "tox run -e build" instead of calling "charmcraft pack" directly' >&2
+          exit 1
+      fi
     build-packages:
       - libffi-dev
       - libssl-dev

config.yaml

@@ -5,6 +5,13 @@ options:
   cluster-name:
     description: "Optional - Name of the MySQL InnoDB cluster"
     type: "string"
+  profile:
+   description: |
+      profile representing the scope of deployment, and used to be able to enable high-level
+      high-level customisation of sysconfigs, resource checks/allocation, warning levels, etc.
+      Allowed values are: “production” and “testing”.
+   type: string
+   default: production
 # Config options for the legacy 'mysql relation'
   mysql-interface-user:
     description: "The database username for the legacy 'mysql' relation"