Skip to content

Commit

Permalink
Miscellaneous improvements to the mysql legacy relation (#254)
Browse files Browse the repository at this point in the history
* parent 6e90ca7
author Shayan Patel <[email protected]> 1690560167 -0400
committer Shayan Patel <[email protected]> 1692738982 +0000
gpgsig -----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAg0GAga0rhzA34UTlbQlTVuF8/Hn
 wp8A23MiF5YOCeQ6sAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5
 AAAAQP08Hjeph0Squupw7oKnz9SMgghKunN3kIxZZ6vstzWSUEKXQnLg08as4enV5ZV9Wg
 GKtFP9BKOKS5Yy4luhXQ4=
 -----END SSH SIGNATURE-----

parent 6e90ca7
author Shayan Patel <[email protected]> 1690560167 -0400
committer Shayan Patel <[email protected]> 1692738754 +0000
gpgsig -----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAg0GAga0rhzA34UTlbQlTVuF8/Hn
 wp8A23MiF5YOCeQ6sAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5
 AAAAQOavuZ8DHFs11FQHwH/xIDTDJgebY25stheI3oJjhRLKSFvcD5fDf2wx+3p5r3zXIP
 8+8619ip29zfWkemkZKgs=
 -----END SSH SIGNATURE-----

Miscellaneous improvements to the mysql legacy relation

Update cos_agent charm lib to v0.5 and add missing import

Fix call of get_secret method on the charm instead of the unit

Address PR feedback

Pin python dependencies with poetry (#192)

Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

Update keystone charm to yoga/stable in series jammy for shared_db integration test (#260)

Use block_until instead of wait_for_idle in shared_db integration test

Update cos_agent charm lib to v0.5

Add `poetry lock` commands to lint and format (#257)

Lint: `poetry lock --check` verifies that poetry.lock is valid for `pyproject.toml`
Format: `poetry lock --no-update` adds any changes in `pyproject.toml` to `poetry.lock` without updated locked versions

trivial fix (#261)

port from k8s and fixes for constrained memory (#253)

* port from k8s and fixes for constrained memory

fix function call

typing fixes

updates cos-agent lib

Pin python dependencies with poetry (#192)

* Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

* bump lib

* fix for dpe-2274

* streamlined test to avoiding timeout

---------

Co-authored-by: Carl Csaposs <[email protected]>

Use snap with ppa sources + other misc snap related improvements (#249)

Fix lint warnings and failing unit tests

Update cos_agent charmlib to v0.5

Only start mysqld-exporter on cos relation created and stop it on relation broken

Expect mysqld-exporter to be disabled by default for replication integration tests

Add missing await statements in exporter integration test

Use retry to determine if mysqld-exporeter service has started

Set monitoring username and password for exporter in integration test

Fix how the username and password were being set for the exporter in the integration test

Run format

Sleep to wait until mysqld-exporter started

Run format

Address PR feedback

Address PR feedback

Update data-platform-workflows to v4.1.4 (#268)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Use self-hosted runners for integration tests (#255)

WIP: Implement support for juju secrets and create MySQLCharmBase for common charm code in vm and k8s (#243)

Update existing secrets instead of creating a new one every time

DPE-2154 profile configuration support (#240)

* initial profile configuration support

* going for juju latest

* bump lib

* key has default value

* doing renovate's work

* some test coverage

* temporary pin

* reflect value for testing profile

Refactor get_member_state (#242)

* retrieves all entries then filter

Some cases (pod rescheduling) the member_id is not populated,
failing the state query with the where clause

* explaining comments

Update charm libs + fix failing unit tests

Bump mysql charmlib patch to 37

Changes resulting from testing with juju 3.1.6

Fail installation if snap already installed (#235)

Avoid overriding snap if mysql charm & mysql-router charm installed on same machine

Context: https://chat.canonical.com/canonical/pl/b8e1daeskjrejxtryjuxuwi9ua

Disable codecov GitHub annotations (#245)

DPE-1979 Use /etc/hosts as a hostname resolution of nodes in the cluster (#237)

* WIP: Use /etc/hosts as a hostname resolution of nodes in the cluster

* Add IP address observer + address PR feedback

* Fix failing unit tests

* Address remaining PR feedback

* Fix failing unit test

* Send SIGTERM instead of SIGINT to terminate the ip address observer process

* Update cos_agent lib to v0.4

* Fix failing unit test

* Update S3 charmlib to v0.3

* Update data_interfaces charmlib to v0.14

DPE-1511 Autogenerate database and username in legacy mysql if not specified in config (#222)

* Auto generate username and database if not specified in config for the mysql legacy relation

* Clean up username and database from databag upon relation broken

* Rework the handling of usernames and databases

* Handle config changes other than username and database

* Change error log into an info log

* Update data_interfaces charm lib to v0.13

* Refactor legacy mysql relation with feedback in mind

* Account for empty strings as config values

* Update data_interfaces charm lib to v0.16

* Update s3 charm lib to v0.4

* Address PR feedback

* Address minor nit in feedback

* Avoid using .get() where unnecessary

* Update juju pin to 2.9.43.0 and pin macaroonbakery to 1.3.1 in tox

* Do not run event handler to update /etc/hosts if all passwords are not set

Add CODEOWNERS (#247)

Required for self-hosted runners

Fix incorrect tls integration test + some secret related changes in tls charmlib

Move relevant event handler observers into the mysql charm base class

Use secrets caching and address PR feedback

Update cos_agent charm lib to v0.5

Pin python dependencies with poetry (#192)

Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

Update keystone charm to yoga/stable in series jammy for shared_db integration test (#260)

Use block_until instead of wait_for_idle in shared_db integration test

Update cos_agent charm lib to v0.5

Add `poetry lock` commands to lint and format (#257)

Lint: `poetry lock --check` verifies that poetry.lock is valid for `pyproject.toml`
Format: `poetry lock --no-update` adds any changes in `pyproject.toml` to `poetry.lock` without updated locked versions

Address refresh from juju <= 3.1.4 to juju >= 3.1.5

trivial fix (#261)

port from k8s and fixes for constrained memory (#253)

* port from k8s and fixes for constrained memory

fix function call

typing fixes

updates cos-agent lib

Pin python dependencies with poetry (#192)

* Use charmcraft 2.3.0 to fix release build (#256)

charmcraft 2.3.0 was pinned in the integration test build but not in the release build

* bump lib

* fix for dpe-2274

* streamlined test to avoiding timeout

---------

Co-authored-by: Carl Csaposs <[email protected]>

Use snap with ppa sources + other misc snap related improvements (#249)

Fix lint warnings and failing unit tests

Update cos_agent charmlib to v0.5

Only start mysqld-exporter on cos relation created and stop it on relation broken

Expect mysqld-exporter to be disabled by default for replication integration tests

Add missing await statements in exporter integration test

Use retry to determine if mysqld-exporeter service has started

Set monitoring username and password for exporter in integration test

Fix how the username and password were being set for the exporter in the integration test

Run format

Sleep to wait until mysqld-exporter started

Run format

Address PR feedback

Address PR feedback

Address stylistic PR feedback

Update mysql charm libpatch to v0.38

Temporarily disable self-hosted runners (#280)

In order to request more self-hosted runners, we need to collect data about self-hosted runner performance compared to GitHub-hosted runners. (@taurus-forever)

Since there are a limited number of runners, temporarily disable self-hosted runners on other PRs so that we can collect data with controlled PRs

Use renovate preset (#274)

Delete CODEOWNERS (#277)

No longer a requirement for self-hosted runners

[charm] Update charm dependencies (#264)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

DPE-2352 Restart mysql exporter upon monitoring password change (#285)

* Restart mysql exporter upon monitoring password change

* Add restart_mysql_exporter as an abstract method on MySQLBase

[charm lib] Update charm lib dependencies to v41.0.3 (#263)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

methods used on in-place upgrades (#288)

* methods used on in-place upgrades

* bump libpatch

* pr comments

bump libpatch (#290)

Fix committed merge conflict

Add `poetry lock` commands to lint and format (#257)

Lint: `poetry lock --check` verifies that poetry.lock is valid for `pyproject.toml`
Format: `poetry lock --no-update` adds any changes in `pyproject.toml` to `poetry.lock` without updated locked versions

trivial fix (#261)

Temporarily disable self-hosted runners (#280)

In order to request more self-hosted runners, we need to collect data about self-hosted runner performance compared to GitHub-hosted runners. (@taurus-forever)

Since there are a limited number of runners, temporarily disable self-hosted runners on other PRs so that we can collect data with controlled PRs

Use renovate preset (#274)

Delete CODEOWNERS (#277)

No longer a requirement for self-hosted runners

[charm] Update charm dependencies (#264)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

[charm lib] Update charm lib dependencies to v41.0.3 (#263)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

methods used on in-place upgrades (#288)

* methods used on in-place upgrades

* bump libpatch

* pr comments

bump libpatch (#290)

Add `has_cos_relation` to the MySQLCharmBase class (#289)

* Restart mysql exporter upon monitoring password change

* Add restart_mysql_exporter as an abstract method on MySQLBase

* Update libpatch and port has_cos_relations property into MySQLCharmBase

* Update mysql patch lib to v0.40

* Add missing import

[MISC] Switch maintainers to the DPE mailing list (#271)

* Switch maintainers to DPE mailing list

* Maintainers

strip info line when present (#294)

* strip info line when present

* fix comment

Update data interfaces charm lib to v0.17

* Update upgrade charm lib to v0.10
  • Loading branch information
shayancanonical authored Aug 23, 2023
1 parent 2ff030b commit 7d94c8d
Show file tree
Hide file tree
Showing 3 changed files with 88 additions and 104 deletions.
148 changes: 54 additions & 94 deletions lib/charms/data_platform_libs/v0/data_interfaces.py
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ def _on_topic_requested(self, event: TopicRequestedEvent):

# Increment this PATCH version before using `charmcraft publish-lib` or reset
# to 0 if you are raising the major API version
LIBPATCH = 16
LIBPATCH = 17

PYDEPS = ["ops>=2.0.0"]

Expand Down Expand Up @@ -365,11 +365,11 @@ def diff(event: RelationChangedEvent, bucket: Union[Unit, Application]) -> Diff:
return Diff(added, changed, deleted)


# Base DataProvides and DataRequires
# Base DataRelation


class DataProvides(Object, ABC):
"""Base provides-side of the data products relation."""
class DataRelation(Object, ABC):
"""Base relation data mainpulation class."""

def __init__(self, charm: CharmBase, relation_name: str) -> None:
super().__init__(charm, relation_name)
Expand All @@ -379,23 +379,11 @@ def __init__(self, charm: CharmBase, relation_name: str) -> None:
self.relation_name = relation_name
self.framework.observe(
charm.on[relation_name].relation_changed,
self._on_relation_changed,
self._on_relation_changed_event,
)

def _diff(self, event: RelationChangedEvent) -> Diff:
"""Retrieves the diff of the data in the relation changed databag.
Args:
event: relation changed event.
Returns:
a Diff instance containing the added, deleted and changed
keys from the event relation databag.
"""
return diff(event, self.local_app)

@abstractmethod
def _on_relation_changed(self, event: RelationChangedEvent) -> None:
def _on_relation_changed_event(self, event: RelationChangedEvent) -> None:
"""Event emitted when the relation data has changed."""
raise NotImplementedError

Expand All @@ -404,10 +392,11 @@ def fetch_relation_data(self) -> dict:
This function can be used to retrieve data from a relation
in the charm code when outside an event callback.
Function cannot be used in `*-relation-broken` events and will raise an exception.
Returns:
a dict of the values stored in the relation data bag
for all relation instances (indexed by the relation id).
for all relation instances (indexed by the relation ID).
"""
data = {}
for relation in self.relations:
Expand All @@ -430,13 +419,49 @@ def _update_relation_data(self, relation_id: int, data: dict) -> None:
that should be updated in the relation.
"""
if self.local_unit.is_leader():
if relation := self.charm.model.get_relation(self.relation_name, relation_id):
relation = self.charm.model.get_relation(self.relation_name, relation_id)
if relation:
relation.data[self.local_app].update(data)

@staticmethod
def _is_relation_active(relation: Relation):
"""Whether the relation is active based on contained data."""
try:
_ = repr(relation.data)
return True
except (RuntimeError, ModelError):
return False

@property
def relations(self) -> List[Relation]:
"""The list of Relation instances associated with this relation_name."""
return list(self.charm.model.relations[self.relation_name])
return [
relation
for relation in self.charm.model.relations[self.relation_name]
if self._is_relation_active(relation)
]


# Base DataProvides and DataRequires


class DataProvides(DataRelation):
"""Base provides-side of the data products relation."""

def __init__(self, charm: CharmBase, relation_name: str) -> None:
super().__init__(charm, relation_name)

def _diff(self, event: RelationChangedEvent) -> Diff:
"""Retrieves the diff of the data in the relation changed databag.
Args:
event: relation changed event.
Returns:
a Diff instance containing the added, deleted and changed
keys from the event relation databag.
"""
return diff(event, self.local_app)

def set_credentials(self, relation_id: int, username: str, password: str) -> None:
"""Set credentials.
Expand Down Expand Up @@ -476,7 +501,7 @@ def set_tls_ca(self, relation_id: int, tls_ca: str) -> None:
self._update_relation_data(relation_id, {"tls-ca": tls_ca})


class DataRequires(Object, ABC):
class DataRequires(DataRelation):
"""Requires-side of the relation."""

def __init__(
Expand All @@ -487,62 +512,16 @@ def __init__(
):
"""Manager of base client relations."""
super().__init__(charm, relation_name)
self.charm = charm
self.extra_user_roles = extra_user_roles
self.local_app = self.charm.model.app
self.local_unit = self.charm.unit
self.relation_name = relation_name
self.framework.observe(
self.charm.on[relation_name].relation_created, self._on_relation_created_event
)
self.framework.observe(
self.charm.on[relation_name].relation_changed, self._on_relation_changed_event
)

@abstractmethod
def _on_relation_created_event(self, event: RelationCreatedEvent) -> None:
"""Event emitted when the relation is created."""
raise NotImplementedError

@abstractmethod
def _on_relation_changed_event(self, event: RelationChangedEvent) -> None:
raise NotImplementedError

def fetch_relation_data(self) -> dict:
"""Retrieves data from relation.
This function can be used to retrieve data from a relation
in the charm code when outside an event callback.
Function cannot be used in `*-relation-broken` events and will raise an exception.
Returns:
a dict of the values stored in the relation data bag
for all relation instances (indexed by the relation ID).
"""
data = {}
for relation in self.relations:
data[relation.id] = (
{key: value for key, value in relation.data[relation.app].items() if key != "data"}
if relation.app
else {}
)
return data

def _update_relation_data(self, relation_id: int, data: dict) -> None:
"""Updates a set of key-value pairs in the relation.
This function writes in the application data bag, therefore,
only the leader unit can call it.
Args:
relation_id: the identifier for a particular relation.
data: dict containing the key-value pairs
that should be updated in the relation.
"""
if self.local_unit.is_leader():
relation = self.charm.model.get_relation(self.relation_name, relation_id)
relation.data[self.local_app].update(data)

def _diff(self, event: RelationChangedEvent) -> Diff:
"""Retrieves the diff of the data in the relation changed databag.
Expand All @@ -555,23 +534,6 @@ def _diff(self, event: RelationChangedEvent) -> Diff:
"""
return diff(event, self.local_unit)

@property
def relations(self) -> List[Relation]:
"""The list of Relation instances associated with this relation_name."""
return [
relation
for relation in self.charm.model.relations[self.relation_name]
if self._is_relation_active(relation)
]

@staticmethod
def _is_relation_active(relation: Relation):
try:
_ = repr(relation.data)
return True
except (RuntimeError, ModelError):
return False

@staticmethod
def _is_resource_created_for_relation(relation: Relation) -> bool:
if not relation.app:
Expand Down Expand Up @@ -797,7 +759,7 @@ class DatabaseProvides(DataProvides):
def __init__(self, charm: CharmBase, relation_name: str) -> None:
super().__init__(charm, relation_name)

def _on_relation_changed(self, event: RelationChangedEvent) -> None:
def _on_relation_changed_event(self, event: RelationChangedEvent) -> None:
"""Event emitted when the relation has changed."""
# Only the leader should handle this event.
if not self.local_unit.is_leader():
Expand Down Expand Up @@ -938,11 +900,8 @@ def _assign_relation_alias(self, relation_id: int) -> None:

# Return if an alias was already assigned to this relation
# (like when there are more than one unit joining the relation).
if (
self.charm.model.get_relation(self.relation_name, relation_id)
.data[self.local_unit]
.get("alias")
):
relation = self.charm.model.get_relation(self.relation_name, relation_id)
if relation and relation.data[self.local_unit].get("alias"):
return

# Retrieve the available aliases (the ones that weren't assigned to any relation).
Expand All @@ -955,7 +914,8 @@ def _assign_relation_alias(self, relation_id: int) -> None:

# Set the alias in the unit relation databag of the specific relation.
relation = self.charm.model.get_relation(self.relation_name, relation_id)
relation.data[self.local_unit].update({"alias": available_aliases[0]})
if relation:
relation.data[self.local_unit].update({"alias": available_aliases[0]})

def _emit_aliased_event(self, event: RelationChangedEvent, event_name: str) -> None:
"""Emit an aliased event to a particular relation if it has an alias.
Expand Down Expand Up @@ -1197,7 +1157,7 @@ class KafkaProvides(DataProvides):
def __init__(self, charm: CharmBase, relation_name: str) -> None:
super().__init__(charm, relation_name)

def _on_relation_changed(self, event: RelationChangedEvent) -> None:
def _on_relation_changed_event(self, event: RelationChangedEvent) -> None:
"""Event emitted when the relation has changed."""
# Only the leader should handle this event.
if not self.local_unit.is_leader():
Expand Down Expand Up @@ -1377,7 +1337,7 @@ class OpenSearchProvides(DataProvides):
def __init__(self, charm: CharmBase, relation_name: str) -> None:
super().__init__(charm, relation_name)

def _on_relation_changed(self, event: RelationChangedEvent) -> None:
def _on_relation_changed_event(self, event: RelationChangedEvent) -> None:
"""Event emitted when the relation has changed."""
# Only the leader should handle this event.
if not self.local_unit.is_leader():
Expand Down
4 changes: 2 additions & 2 deletions lib/charms/data_platform_libs/v0/upgrade.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ def restart(self, event) -> None:

# Increment this PATCH version before using `charmcraft publish-lib` or reset
# to 0 if you are raising the major API version
LIBPATCH = 9
LIBPATCH = 10

PYDEPS = ["pydantic>=1.10,<2"]

Expand Down Expand Up @@ -817,7 +817,7 @@ def idle(self) -> Optional[bool]:
Returns:
True if all application units in idle state. Otherwise False
"""
return self.cluster_state == "idle"
return set(self.unit_states) == {"idle"}

@abstractmethod
def pre_upgrade_check(self) -> None:
Expand Down
40 changes: 32 additions & 8 deletions src/relations/mysql.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import logging

from charms.mysql.v0.mysql import (
MySQLCheckUserExistenceError,
MySQLCreateApplicationDatabaseAndScopedUserError,
MySQLDeleteUsersForUnitError,
MySQLGetClusterPrimaryAddressError,
Expand Down Expand Up @@ -103,8 +104,14 @@ def _on_leader_elected(self, _) -> None:
relation_databag[self.charm.unit][key] = value

# Assign the cluster primary's address as the database host
primary_address = self.charm._mysql.get_cluster_primary_address().split(":")[0]
relation_databag[self.charm.unit]["host"] = primary_address
primary_address = self.charm._mysql.get_cluster_primary_address()
if not primary_address:
self.charm.unit.status = BlockedStatus(
"Failed to retrieve cluster primary address"
)
return

relation_databag[self.charm.unit]["host"] = primary_address.split(":")[0]

def _on_config_changed(self, _) -> None:
"""Handle the change of the username/database in config."""
Expand Down Expand Up @@ -145,19 +152,30 @@ def _on_mysql_relation_created(self, event: RelationCreatedEvent) -> None:
event.defer()
return

logger.warning("DEPRECATION WARNING - `mysql` is a legacy interface")

# wait until the unit is initialized
if not self.charm.unit_peer_data.get("unit-initialized"):
event.defer()
return

logger.warning("DEPRECATION WARNING - `mysql` is a legacy interface")

username = self._get_or_generate_username(event.relation.id)
database = self._get_or_generate_database(event.relation.id)

try:
user_exists = self.charm._mysql.does_mysql_user_exist(username, "%")
except MySQLCheckUserExistenceError:
self.charm.unit.status = BlockedStatus("Failed to check user existence")
return

# Only execute if the application user does not exist
# since it could have been created by another related app
if self.charm._mysql.does_mysql_user_exist(username, "%"):
if user_exists:
mysql_relation_data = self.charm.app_peer_data[MYSQL_RELATION_DATA_KEY]

updates = json.loads(mysql_relation_data)
event.relation.data[self.charm.unit].update(updates)

return

password = self._get_or_set_password_in_peer_secrets(username)
Expand All @@ -171,7 +189,7 @@ def _on_mysql_relation_created(self, event: RelationCreatedEvent) -> None:
unit_name="mysql-legacy-relation",
)

primary_address = self.charm._mysql.get_cluster_primary_address().split(":")[0]
primary_address = self.charm._mysql.get_cluster_primary_address()

except (
MySQLCreateApplicationDatabaseAndScopedUserError,
Expand All @@ -180,9 +198,13 @@ def _on_mysql_relation_created(self, event: RelationCreatedEvent) -> None:
self.charm.unit.status = BlockedStatus("Failed to initialize `mysql` relation")
return

if not primary_address:
self.charm.unit.status = BlockedStatus("Failed to retrieve cluster primary address")
return

updates = {
"database": database,
"host": primary_address,
"host": primary_address.split(":")[0],
"password": password,
"port": "3306",
"root_password": self.charm.get_secret("app", ROOT_PASSWORD_KEY),
Expand All @@ -195,7 +217,7 @@ def _on_mysql_relation_created(self, event: RelationCreatedEvent) -> None:
self.charm.app_peer_data[MYSQL_RELATION_DATABASE_KEY] = database

# Store the relation data into the peer relation databag
self.charm.app_peer_data["mysql_relation_data"] = json.dumps(updates)
self.charm.app_peer_data[MYSQL_RELATION_DATA_KEY] = json.dumps(updates)

def _on_mysql_relation_broken(self, event: RelationBrokenEvent) -> None:
"""Handle the `mysql` legacy relation broken event.
Expand All @@ -222,6 +244,8 @@ def _on_mysql_relation_broken(self, event: RelationBrokenEvent) -> None:
del self.charm.app_peer_data[MYSQL_RELATION_USER_KEY]
del self.charm.app_peer_data[MYSQL_RELATION_DATABASE_KEY]

del self.charm.app_peer_data[MYSQL_RELATION_DATA_KEY]

if isinstance(
self.charm.app.status, BlockedStatus
) and self.charm.app.status.message.startswith(
Expand Down

0 comments on commit 7d94c8d

Please sign in to comment.