Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBMON-2051: Adding a deadlock metric to MySQL databases #16904

Merged
merged 32 commits into from
Feb 28, 2024

Conversation

kozlovb
Copy link
Contributor

@kozlovb kozlovb commented Feb 20, 2024

What does this PR do?

This PR adds a new deadlock count metric to the MySQL integrations.
This metrics tracks a number of deadlocks that have been detected in the DB.

Motivation

The deadlock metric is required by customers.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Changelog entries must be created for modifications to shipped code
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

codecov bot commented Feb 20, 2024

Codecov Report

Attention: Patch coverage is 91.35802% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 87.61%. Comparing base (fec9a72) to head (e513c14).
Report is 32 commits behind head on master.

Additional details and impacted files
Flag Coverage Δ
active_directory ?
activemq ?
activemq_xml ?
aerospike ?
airflow ?
amazon_msk ?
ambari ?
apache ?
arangodb ?
argocd ?
aspdotnet ?
avi_vantage ?
azure_iot_edge ?
boundary ?
btrfs ?
cacti ?
calico ?
cassandra ?
cassandra_nodetool ?
ceph ?
cert_manager ?
cilium ?
cisco_aci ?
citrix_hypervisor ?
clickhouse ?
cloud_foundry_api ?
cloudera ?
cockroachdb ?
confluent_platform ?
consul ?
coredns ?
couch ?
couchbase ?
crio ?
datadog_checks_base ?
datadog_checks_dev ?
datadog_checks_downloader ?
datadog_cluster_agent ?
dcgm ?
ddev ?
directory ?
disk ?
dns_check ?
dotnetclr ?
druid ?
ecs_fargate ?
eks_fargate ?
elastic ?
envoy ?
etcd ?
exchange_server ?
external_dns ?
fluentd ?
fluxcd ?
foundationdb ?
gearmand ?
gitlab ?
gitlab_runner ?
glusterfs ?
go_expvar ?
gunicorn ?
haproxy ?
harbor ?
hazelcast ?
hdfs_datanode ?
hdfs_namenode ?
hive ?
hivemq ?
http_check ?
hudi ?
ibm_ace ?
ibm_db2 ?
ibm_i ?
ibm_mq ?
ibm_was ?
ignite ?
iis ?
impala ?
istio ?
jboss_wildfly ?
kafka ?
kafka_consumer ?
karpenter ?
kong ?
kube_apiserver_metrics ?
kube_controller_manager ?
kube_dns ?
kube_metrics_server ?
kube_proxy ?
kube_scheduler ?
kubelet ?
kubernetes_state ?
kyototycoon ?
lighttpd ?
linkerd ?
linux_proc_extras ?
mapr ?
mapreduce ?
marathon ?
marklogic ?
mcache ?
mesos_master ?
mesos_slave ?
mongo ?
mysql 87.61% <91.35%> (+0.12%) ⬆️
nagios ?
network ?
nfsstat ?
nginx ?
nginx_ingress_controller ?
nvidia_triton ?
openldap ?
openmetrics ?
openstack ?
openstack_controller ?
oracle ?
pdh_check ?
pgbouncer ?
php_fpm ?
postfix ?
postgres ?
powerdns_recursor ?
presto ?
process ?
prometheus ?
proxysql ?
pulsar ?
rabbitmq ?
ray ?
redisdb ?
rethinkdb ?
riak ?
riakcs ?
sap_hana ?
scylla ?
silk ?
singlestore ?
snmp ?
snowflake ?
solr ?
sonarqube ?
spark ?
sqlserver ?
squid ?
ssh_check ?
statsd ?
strimzi ?
supervisord ?
system_core ?
system_swap ?
tcp_check ?
teamcity ?
temporal ?
teradata ?
tls ?
tokumx ?
torchserve ?
traffic_server ?
twemproxy ?
twistlock ?
varnish ?
vault ?
vertica ?
voltdb ?
vsphere ?
weaviate ?
weblogic ?
win32_event_log ?
windows_performance_counters ?
windows_service ?
wmi_check ?
yarn ?
zk ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

7 similar comments
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@kozlovb kozlovb force-pushed the boris-kozlov/add-deadlock-metric branch from 8849ee1 to 14776c4 Compare February 21, 2024 15:43
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

8 similar comments
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

mysql/datadog_checks/mysql/mysql.py Outdated Show resolved Hide resolved
mysql/tests/test_query_activity.py Outdated Show resolved Hide resolved
@kozlovb kozlovb force-pushed the boris-kozlov/add-deadlock-metric branch from f4785d4 to 8e89bf4 Compare February 23, 2024 13:39
Comment on lines 144 to 145
with self._connect() as db:
self._is_innodb_engine_enabled = self._is_innodb_engine_enabled(db)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to put this under def check after db connection is established

with self._connect() as db:
.
We can also rename the function _is_innodb_engine_enabled to something like _check_innodb_engine_enabled.
In the __init__ constructor, we can just initialize a variable self.innodb_engine_enabled = None as None to indicate the check has not been done.

Copy link
Contributor Author

@kozlovb kozlovb Feb 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx, for renaming will do.

as for moving it to check - that was my initial idea too. But I didn't like that I had to add db as an argument to runtime_queries. Although, now, I greped the code and no one is calling this from outside of the class. I thought if it's without _ then it's public and I cant change the signature.

The other option is to modify _check_innodb_engine_enabled, I could've check there if self.conn is None or not, and if not open a connection there. Pls let me know which one is better or something else.

What I don't like in my solution and kind of in existing one is the absence of clarity - how one can know that _check_innodb_engine_enabled has to be called and not _is_innodb_engine_enabled. Same for the runtime_queries property vs _runtime_queries. May be we call _runtime_queries -> _runtime_queries_cached and same for _is_innodb_engine_enabled -> _is_innodb_engine_enabled_cached. Like this it would be clear that this field might be None.

mysql/datadog_checks/mysql/mysql.py Show resolved Hide resolved
mysql/datadog_checks/mysql/mysql.py Show resolved Hide resolved
mysql/datadog_checks/mysql/mysql.py Show resolved Hide resolved
mysql/changelog.d/16904.added Outdated Show resolved Hide resolved
@kozlovb kozlovb changed the title DBMON-2051: Adding a deadlock metric to Maria DB DBMON-2051: Adding a deadlock metric to MySQL databases Feb 26, 2024
lu-zhengda
lu-zhengda previously approved these changes Feb 27, 2024
FlorentClarret
FlorentClarret previously approved these changes Feb 28, 2024
@@ -0,0 +1 @@
DBMON-2051: Adding a deadlock metric to Maria DB
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not include the ticket in the changelog since this is mostly for customers and they can't have access to our Jira board

Copy link
Contributor Author

@kozlovb kozlovb Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, now, I remembered in my previous company we had two separate entries one for customers and one internal with the ticket. Ok, let me remove it, thx.

@kozlovb kozlovb dismissed stale reviews from FlorentClarret and lu-zhengda via e513c14 February 28, 2024 09:15
@kozlovb kozlovb merged commit 3f8232a into master Feb 28, 2024
38 checks passed
@kozlovb kozlovb deleted the boris-kozlov/add-deadlock-metric branch February 28, 2024 15:16
FlorentClarret pushed a commit that referenced this pull request Mar 1, 2024
* DBMON-2051: Adding a deadlock metric to Maria DB

* removing print outs

* removed prints

* trying to create a proper deadlock

* Fix thread pool limit

* Add a check

* Add wait to let update db threads to finish

* Add select query

* fix query

* revert changes to conftest

* fix test

* Add events

* remove obsolete diff

* cleaned up pr

* moved tests to the end

* Put back empty lines

* put back empty line

* added a changelog

* changed pr number in changelog

* removed patched config

* reverting back conftest

* reverted common py

* applied linter

* Added the correct changelog

* add a check for innnodb

* removed superfluous method

* removed whitespace

* changed to monotonic_count

* removed check from constructor

* modified variable names

* fixed changelog and test error

* improved the changelog

---------

Co-authored-by: root <[email protected]>
NouemanKHAL pushed a commit that referenced this pull request Mar 4, 2024
* DBMON-2051: Adding a deadlock metric to Maria DB

* removing print outs

* removed prints

* trying to create a proper deadlock

* Fix thread pool limit

* Add a check

* Add wait to let update db threads to finish

* Add select query

* fix query

* revert changes to conftest

* fix test

* Add events

* remove obsolete diff

* cleaned up pr

* moved tests to the end

* Put back empty lines

* put back empty line

* added a changelog

* changed pr number in changelog

* removed patched config

* reverting back conftest

* reverted common py

* applied linter

* Added the correct changelog

* add a check for innnodb

* removed superfluous method

* removed whitespace

* changed to monotonic_count

* removed check from constructor

* modified variable names

* fixed changelog and test error

* improved the changelog

---------

Co-authored-by: root <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants