From 363fdbea0c61b0d67a981cd039b69aff7e724363 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 18 Mar 2022 13:38:58 -0700 Subject: [PATCH 01/15] `opentelemetry-instrumentation-system-metrics` restore package Restore the system-metrics instrumentation package into main from the `metrics` branch and update its usage of the api/sdk. Fix #1011 --- .../LICENSE | 201 +++++ .../MANIFEST.in | 9 + .../README.rst | 24 + .../setup.cfg | 52 ++ .../setup.py | 31 + .../system_metrics/__init__.py | 648 ++++++++++++++ .../instrumentation/system_metrics/version.py | 15 + .../tests/__init__.py | 0 .../tests/test_system_metrics.py | 835 ++++++++++++++++++ tox.ini | 7 + 10 files changed, 1822 insertions(+) create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/README.rst create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/setup.py create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/tests/__init__.py create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE b/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE new file mode 100644 index 0000000000..261eeb9e9f --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in b/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in new file mode 100644 index 0000000000..aed3e33273 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in @@ -0,0 +1,9 @@ +graft src +graft tests +global-exclude *.pyc +global-exclude *.pyo +global-exclude __pycache__/* +include CHANGELOG.md +include MANIFEST.in +include README.rst +include LICENSE diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst b/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst new file mode 100644 index 0000000000..7a6c65b59b --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst @@ -0,0 +1,24 @@ +OpenTelemetry System Metrics Instrumentation +============================================ + +|pypi| + +.. |pypi| image:: https://badge.fury.io/py/opentelemetry-instrumentation-system-metrics.svg + :target: https://pypi.org/project/opentelemetry-instrumentation-system-metrics/ + +Instrumentation to collect system performance metrics. + + +Installation +------------ + +:: + + pip install opentelemetry-instrumentation-system-metrics + + +References +---------- +* `OpenTelemetry System Metrics Instrumentation `_ +* `OpenTelemetry Project `_ + diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg new file mode 100644 index 0000000000..918a62c293 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg @@ -0,0 +1,52 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +[metadata] +name = opentelemetry-instrumentation-system-metrics +description = OpenTelemetry System Metrics Instrumentation +long_description = file: README.rst +long_description_content_type = text/x-rst +author = OpenTelemetry Authors +author_email = cncf-opentelemetry-contributors@lists.cncf.io +url = https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-system-metrics +platforms = any +license = Apache-2.0 +classifiers = + Development Status :: 4 - Beta + Intended Audience :: Developers + License :: OSI Approved :: Apache Software License + Programming Language :: Python + Programming Language :: Python :: 3 + Programming Language :: Python :: 3.6 + Programming Language :: Python :: 3.7 + Programming Language :: Python :: 3.8 + Programming Language :: Python :: 3.9 + Programming Language :: Python :: 3.10 + +[options] +python_requires = >=3.5 +package_dir= + =src +packages=find_namespace: +install_requires = + opentelemetry-api ~= 1.10.0 + opentelemetry-sdk ~= 1.10.0 + psutil ~= 5.7.0 + +[options.extras_require] +test = + opentelemetry-test-utils == 0.29b0 + +[options.packages.find] +where = src diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py new file mode 100644 index 0000000000..f0bbf9eff0 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py @@ -0,0 +1,31 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os + +import setuptools + +BASE_DIR = os.path.dirname(__file__) +VERSION_FILENAME = os.path.join( + BASE_DIR, + "src", + "opentelemetry", + "instrumentation", + "system_metrics", + "version.py", +) +PACKAGE_INFO = {} +with open(VERSION_FILENAME) as f: + exec(f.read(), PACKAGE_INFO) + +setuptools.setup(version=PACKAGE_INFO["__version__"]) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py new file mode 100644 index 0000000000..a5ba6c58e4 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -0,0 +1,648 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Instrument to report system (CPU, memory, network) and +process (CPU, memory, garbage collection) metrics. By default, the +following metrics are configured: + +.. code:: python + + { + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.cpu.utilization": ["idle", "user", "system", "irq"], + "system.memory.usage": ["used", "free", "cached"], + "system.memory.utilization": ["used", "free", "cached"], + "system.swap.usage": ["used", "free"], + "system.swap.utilization": ["used", "free"], + "system.disk.io": ["read", "write"], + "system.disk.operations": ["read", "write"], + "system.disk.time": ["read", "write"], + "system.disk.merged": ["read", "write"], + "system.network.dropped.packets": ["transmit", "receive"], + "system.network.packets": ["transmit", "receive"], + "system.network.errors": ["transmit", "receive"], + "system.network.io": ["trasmit", "receive"], + "system.network.connections": ["family", "type"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], + } + +Usage +----- + +.. code:: python + + from opentelemetry._metrics import set_meter_provider + from opentelemetry.instrumentation.system_metrics import SystemMetrics + from opentelemetry.sdk._metrics import MeterProvider + from opentelemetry.sdk._metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader + + exporter = ConsoleMetricExporter() + + set_meter_provider(MeterProvider(PeriodicExportingMetricReader(exporter))) + SystemMetrics() + + # metrics are collected asynchronously + input("...") + + # to configure custom metrics + configuration = { + "system.memory.usage": ["used", "free", "cached"], + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.network.io": ["trasmit", "receive"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], + } + SystemMetrics(exporter, config=configuration) + +API +--- +""" + +import gc +import os +from typing import Dict, List, Iterable, Optional +from platform import python_implementation + +import psutil + +from opentelemetry._metrics import get_meter_provider +from opentelemetry._metrics.measurement import Measurement +from opentelemetry.sdk.util import get_dict_as_key + + +class SystemMetrics: + # pylint: disable=too-many-statements + def __init__( + self, + interval: int = 30, + labels: Optional[Dict[str, str]] = None, + config: Optional[Dict[str, List[str]]] = None, + ): + self._labels = {} if labels is None else labels + self.meter = get_meter_provider().get_meter( + "io.otel.python.instrumentation.system-metrics" + ) # TODO: find a better name here + self._python_implementation = python_implementation().lower() + if config is None: + self._config = { + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.cpu.utilization": ["idle", "user", "system", "irq"], + "system.memory.usage": ["used", "free", "cached"], + "system.memory.utilization": ["used", "free", "cached"], + "system.swap.usage": ["used", "free"], + "system.swap.utilization": ["used", "free"], + # system.swap.page.faults: [], + # system.swap.page.operations: [], + "system.disk.io": ["read", "write"], + "system.disk.operations": ["read", "write"], + "system.disk.time": ["read", "write"], + "system.disk.merged": ["read", "write"], + # "system.filesystem.usage": [], + # "system.filesystem.utilization": [], + "system.network.dropped.packets": ["transmit", "receive"], + "system.network.packets": ["transmit", "receive"], + "system.network.errors": ["transmit", "receive"], + "system.network.io": ["trasmit", "receive"], + "system.network.connections": ["family", "type"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], + } + else: + self._config = config + + self._proc = psutil.Process(os.getpid()) + + self._system_cpu_time_labels = self._labels.copy() + self._system_cpu_utilization_labels = self._labels.copy() + + self._system_memory_usage_labels = self._labels.copy() + self._system_memory_utilization_labels = self._labels.copy() + + self._system_swap_usage_labels = self._labels.copy() + self._system_swap_utilization_labels = self._labels.copy() + # self._system_swap_page_faults = self._labels.copy() + # self._system_swap_page_operations = self._labels.copy() + + self._system_disk_io_labels = self._labels.copy() + self._system_disk_operations_labels = self._labels.copy() + self._system_disk_time_labels = self._labels.copy() + self._system_disk_merged_labels = self._labels.copy() + + # self._system_filesystem_usage_labels = self._labels.copy() + # self._system_filesystem_utilization_labels = self._labels.copy() + + self._system_network_dropped_packets_labels = self._labels.copy() + self._system_network_packets_labels = self._labels.copy() + self._system_network_errors_labels = self._labels.copy() + self._system_network_io_labels = self._labels.copy() + self._system_network_connections_labels = self._labels.copy() + + self._runtime_memory_labels = self._labels.copy() + self._runtime_cpu_time_labels = self._labels.copy() + self._runtime_gc_count_labels = self._labels.copy() + + self.meter.create_observable_counter( + callback=self._get_system_cpu_time, + name="system.cpu.time", + description="System CPU time", + unit="seconds", + ) + + self.meter.create_observable_gauge( + callback=self._get_system_cpu_utilization, + name="system.cpu.utilization", + description="System CPU utilization", + unit="1", + ) + + self.meter.create_observable_gauge( + callback=self._get_system_memory_usage, + name="system.memory.usage", + description="System memory usage", + unit="bytes", + ) + + self.meter.create_observable_gauge( + callback=self._get_system_memory_utilization, + name="system.memory.utilization", + description="System memory utilization", + unit="1", + ) + + self.meter.create_observable_gauge( + callback=self._get_system_swap_usage, + name="system.swap.usage", + description="System swap usage", + unit="pages", + ) + + self.meter.create_observable_gauge( + callback=self._get_system_swap_utilization, + name="system.swap.utilization", + description="System swap utilization", + unit="1", + ) + + # # self.meter.create_observable_counter( + # # callback=self._get_system_swap_page_faults, + # # name="system.swap.page_faults", + # # description="System swap page faults", + # # unit="faults", + # # value_type=int, + # # ) + + # # self.meter.create_observable_counter( + # # callback=self._get_system_swap_page_operations, + # # name="system.swap.page_operations", + # # description="System swap page operations", + # # unit="operations", + # # value_type=int, + # # ) + + self.meter.create_observable_counter( + callback=self._get_system_disk_io, + name="system.disk.io", + description="System disk IO", + unit="bytes", + ) + + self.meter.create_observable_counter( + callback=self._get_system_disk_operations, + name="system.disk.operations", + description="System disk operations", + unit="operations", + ) + + self.meter.create_observable_counter( + callback=self._get_system_disk_time, + name="system.disk.time", + description="System disk time", + unit="seconds", + ) + + self.meter.create_observable_counter( + callback=self._get_system_disk_merged, + name="system.disk.merged", + description="System disk merged", + unit="1", + ) + + # self.accumulator.register_valueobserver( + # callback=self._get_system_filesystem_usage, + # name="system.filesystem.usage", + # description="System filesystem usage", + # unit="bytes", + # value_type=int, + # ) + + # self.meter.create_observable_gauge( + # callback=self._get_system_filesystem_utilization, + # name="system.filesystem.utilization", + # description="System filesystem utilization", + # unit="1", + # value_type=float, + # ) + + self.meter.create_observable_counter( + callback=self._get_system_network_dropped_packets, + name="system.network.dropped_packets", + description="System network dropped_packets", + unit="packets", + ) + + self.meter.create_observable_counter( + callback=self._get_system_network_packets, + name="system.network.packets", + description="System network packets", + unit="packets", + ) + + self.meter.create_observable_counter( + callback=self._get_system_network_errors, + name="system.network.errors", + description="System network errors", + unit="errors", + ) + + self.meter.create_observable_counter( + callback=self._get_system_network_io, + name="system.network.io", + description="System network io", + unit="bytes", + ) + + self.meter.create_observable_up_down_counter( + callback=self._get_system_network_connections, + name="system.network.connections", + description="System network connections", + unit="connections", + ) + + self.meter.create_observable_counter( + callback=self._get_runtime_memory, + name="runtime.{}.memory".format(self._python_implementation), + description="Runtime {} memory".format( + self._python_implementation + ), + unit="bytes", + ) + + self.meter.create_observable_counter( + callback=self._get_runtime_cpu_time, + name="runtime.{}.cpu_time".format(self._python_implementation), + description="Runtime {} CPU time".format( + self._python_implementation + ), + unit="seconds", + ) + + self.meter.create_observable_counter( + callback=self._get_runtime_gc_count, + name="runtime.{}.gc_count".format(self._python_implementation), + description="Runtime {} GC count".format( + self._python_implementation + ), + unit="bytes", + ) + + def _get_system_cpu_time(self) -> Iterable[Measurement]: + """Observer callback for system CPU time + + Args: + observer: the observer to update + """ + for cpu, times in enumerate(psutil.cpu_times(percpu=True)): + for metric in self._config["system.cpu.time"]: + if hasattr(times, metric): + self._system_cpu_time_labels["state"] = metric + self._system_cpu_time_labels["cpu"] = cpu + 1 + yield Measurement( + getattr(times, metric), self._system_cpu_time_labels + ) + + def _get_system_cpu_utilization(self) -> Iterable[Measurement]: + """Observer callback for system CPU utilization + + Args: + observer: the observer to update + """ + + for cpu, times_percent in enumerate( + psutil.cpu_times_percent(percpu=True) + ): + for metric in self._config["system.cpu.utilization"]: + if hasattr(times_percent, metric): + self._system_cpu_utilization_labels["state"] = metric + self._system_cpu_utilization_labels["cpu"] = cpu + 1 + yield Measurement( + getattr(times_percent, metric) / 100, + self._system_cpu_utilization_labels, + ) + + def _get_system_memory_usage(self) -> Iterable[Measurement]: + """Observer callback for memory usage + + Args: + observer: the observer to update + """ + virtual_memory = psutil.virtual_memory() + for metric in self._config["system.memory.usage"]: + self._system_memory_usage_labels["state"] = metric + if hasattr(virtual_memory, metric): + yield Measurement( + getattr(virtual_memory, metric), + self._system_memory_usage_labels, + ) + + def _get_system_memory_utilization(self) -> Iterable[Measurement]: + """Observer callback for memory utilization + + Args: + observer: the observer to update + """ + system_memory = psutil.virtual_memory() + + for metric in self._config["system.memory.utilization"]: + self._system_memory_utilization_labels["state"] = metric + if hasattr(system_memory, metric): + yield Measurement( + getattr(system_memory, metric) / system_memory.total, + self._system_memory_utilization_labels, + ) + + def _get_system_swap_usage(self) -> Iterable[Measurement]: + """Observer callback for swap usage + + Args: + observer: the observer to update + """ + system_swap = psutil.swap_memory() + + for metric in self._config["system.swap.usage"]: + self._system_swap_usage_labels["state"] = metric + if hasattr(system_swap, metric): + yield Measurement( + getattr(system_swap, metric), + self._system_swap_usage_labels, + ) + + def _get_system_swap_utilization(self) -> Iterable[Measurement]: + """Observer callback for swap utilization + + Args: + observer: the observer to update + """ + system_swap = psutil.swap_memory() + + for metric in self._config["system.swap.utilization"]: + if hasattr(system_swap, metric): + self._system_swap_utilization_labels["state"] = metric + yield Measurement( + getattr(system_swap, metric) / system_swap.total, + self._system_swap_utilization_labels, + ) + + # TODO Add _get_system_swap_page_faults + # TODO Add _get_system_swap_page_operations + + def _get_system_disk_io(self) -> Iterable[Measurement]: + """Observer callback for disk IO + + Args: + observer: the observer to update + """ + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.io"]: + if hasattr(counters, "{}_bytes".format(metric)): + self._system_disk_io_labels["device"] = device + self._system_disk_io_labels["direction"] = metric + yield Measurement( + getattr(counters, "{}_bytes".format(metric)), + self._system_disk_io_labels, + ) + + def _get_system_disk_operations(self) -> Iterable[Measurement]: + """Observer callback for disk operations + + Args: + observer: the observer to update + """ + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.operations"]: + if hasattr(counters, "{}_count".format(metric)): + self._system_disk_operations_labels["device"] = device + self._system_disk_operations_labels["direction"] = metric + yield Measurement( + getattr(counters, "{}_count".format(metric)), + self._system_disk_operations_labels, + ) + + def _get_system_disk_time(self) -> Iterable[Measurement]: + """Observer callback for disk time + + Args: + observer: the observer to update + """ + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.time"]: + if hasattr(counters, "{}_time".format(metric)): + self._system_disk_time_labels["device"] = device + self._system_disk_time_labels["direction"] = metric + yield Measurement( + getattr(counters, "{}_time".format(metric)) / 1000, + self._system_disk_time_labels, + ) + + def _get_system_disk_merged(self) -> Iterable[Measurement]: + """Observer callback for disk merged operations + + Args: + observer: the observer to update + """ + + # FIXME The units in the spec is 1, it seems like it should be + # operations or the value type should be Double + + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.time"]: + if hasattr(counters, "{}_merged_count".format(metric)): + self._system_disk_merged_labels["device"] = device + self._system_disk_merged_labels["direction"] = metric + yield Measurement( + getattr(counters, "{}_merged_count".format(metric)), + self._system_disk_merged_labels, + ) + + # TODO Add _get_system_filesystem_usage + # TODO Add _get_system_filesystem_utilization + # TODO Filesystem information can be obtained with os.statvfs in Unix-like + # OSs, how to do the same in Windows? + + def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: + """Observer callback for network dropped packets + + Args: + observer: the observer to update + """ + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + in_out = {"receive": "in", "transmit": "out"}[metric] + if hasattr(counters, "drop{}".format(in_out)): + self._system_network_dropped_packets_labels[ + "device" + ] = device + self._system_network_dropped_packets_labels[ + "direction" + ] = metric + yield Measurement( + getattr(counters, "drop{}".format(in_out)), + self._system_network_dropped_packets_labels, + ) + + def _get_system_network_packets(self) -> Iterable[Measurement]: + """Observer callback for network packets + + Args: + observer: the observer to update + """ + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + recv_sent = {"receive": "recv", "transmit": "sent"}[metric] + if hasattr(counters, "packets_{}".format(recv_sent)): + self._system_network_packets_labels["device"] = device + self._system_network_packets_labels["direction"] = metric + yield Measurement( + getattr(counters, "packets_{}".format(recv_sent)), + self._system_network_packets_labels, + ) + + def _get_system_network_errors(self) -> Iterable[Measurement]: + """Observer callback for network errors + + Args: + observer: the observer to update + """ + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.errors"]: + in_out = {"receive": "in", "transmit": "out"}[metric] + if hasattr(counters, "err{}".format(in_out)): + self._system_network_errors_labels["device"] = device + self._system_network_errors_labels["direction"] = metric + yield Measurement( + getattr(counters, "err{}".format(in_out)), + self._system_network_errors_labels, + ) + + def _get_system_network_io(self) -> Iterable[Measurement]: + """Observer callback for network IO + + Args: + observer: the observer to update + """ + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + recv_sent = {"receive": "recv", "transmit": "sent"}[metric] + if hasattr(counters, "bytes_{}".format(recv_sent)): + self._system_network_io_labels["device"] = device + self._system_network_io_labels["direction"] = metric + yield Measurement( + getattr(counters, "bytes_{}".format(recv_sent)), + self._system_network_io_labels, + ) + + def _get_system_network_connections(self) -> Iterable[Measurement]: + """Observer callback for network connections + + Args: + observer: the observer to update + """ + # TODO How to find the device identifier for a particular + # connection? + + connection_counters = {} + + for net_connection in psutil.net_connections(): + for metric in self._config["system.network.connections"]: + self._system_network_connections_labels["protocol"] = { + 1: "tcp", + 2: "udp", + }[net_connection.type.value] + self._system_network_connections_labels[ + "state" + ] = net_connection.status + self._system_network_connections_labels[metric] = getattr( + net_connection, metric + ) + + connection_counters_key = get_dict_as_key( + self._system_network_connections_labels + ) + + if connection_counters_key in connection_counters.keys(): + connection_counters[connection_counters_key]["counter"] += 1 + else: + connection_counters[connection_counters_key] = { + "counter": 1, + "labels": self._system_network_connections_labels.copy(), + } + + for connection_counter in connection_counters.values(): + yield Measurement( + connection_counter["counter"], + connection_counter["labels"], + ) + + def _get_runtime_memory(self) -> Iterable[Measurement]: + """Observer callback for runtime memory + + Args: + observer: the observer to update + """ + proc_memory = self._proc.memory_info() + for metric in self._config["runtime.memory"]: + if hasattr(proc_memory, metric): + self._runtime_memory_labels["type"] = metric + yield Measurement( + getattr(proc_memory, metric), + self._runtime_memory_labels, + ) + + def _get_runtime_cpu_time(self) -> Iterable[Measurement]: + """Observer callback for runtime CPU time + + Args: + observer: the observer to update + """ + proc_cpu = self._proc.cpu_times() + for metric in self._config["runtime.cpu.time"]: + if hasattr(proc_cpu, metric): + self._runtime_cpu_time_labels["type"] = metric + yield Measurement( + getattr(proc_cpu, metric), + self._runtime_cpu_time_labels, + ) + + def _get_runtime_gc_count(self) -> Iterable[Measurement]: + """Observer callback for garbage collection + + Args: + observer: the observer to update + """ + for index, count in enumerate(gc.get_count()): + self._runtime_gc_count_labels["count"] = str(index) + yield Measurement(count, self._runtime_gc_count_labels) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py new file mode 100644 index 0000000000..5743fb8ce1 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py @@ -0,0 +1,15 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +__version__ = "0.29b0" diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py new file mode 100644 index 0000000000..e8ef6257dd --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -0,0 +1,835 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# pylint: disable=protected-access + +from collections import namedtuple +from platform import python_implementation +from unittest import mock + +from opentelemetry._metrics import get_meter_provider +from opentelemetry._metrics.measurement import Measurement +from opentelemetry.instrumentation.system_metrics import SystemMetrics +from opentelemetry.test.test_base import TestBase + + +class TestSystemMetrics(TestBase): + def setUp(self): + super().setUp() + self.implementation = python_implementation().lower() + + def test_system_metrics_constructor(self): + # ensure the observers have been registered + meter = get_meter_provider().get_meter(__name__) + with mock.patch("opentelemetry.metrics.get_meter") as mock_get_meter: + mock_get_meter.return_value = meter + SystemMetrics() + + self.assertEqual(len(meter.instruments), 18) + + observer_names = [ + "system.cpu.time", + "system.cpu.utilization", + "system.memory.usage", + "system.memory.utilization", + "system.swap.usage", + "system.swap.utilization", + "system.disk.io", + "system.disk.operations", + "system.disk.time", + "system.disk.merged", + "system.network.dropped_packets", + "system.network.packets", + "system.network.errors", + "system.network.io", + "system.network.connections", + "runtime.{}.memory".format(self.implementation), + "runtime.{}.cpu_time".format(self.implementation), + "runtime.{}.gc_count".format(self.implementation), + ] + + for observer in meter.instruments.values(): + self.assertIn(observer.name, observer_names) + observer_names.remove(observer.name) + + def _assert_metrics(self, observer_name, system_metrics, expected): + system_metrics.controller.tick() + assertions = 0 + for ( + metric + ) in ( + self.memory_metrics_exporter._exported_metrics # pylint: disable=protected-access + ): + if ( + metric.labels in expected + and metric.instrument.name == observer_name + ): + self.assertEqual( + metric.aggregator.checkpoint, + expected[metric.labels], + ) + assertions += 1 + self.assertEqual(len(expected), assertions) + + def _test_metrics(self, observer_name, expected): + meter = self.meter_provider.get_meter(__name__) + + with mock.patch("opentelemetry.metrics.get_meter") as mock_get_meter: + mock_get_meter.return_value = meter + system_metrics = SystemMetrics() + self._assert_metrics(observer_name, system_metrics, expected) + + # When this test case is executed, _get_system_cpu_utilization gets run + # too because of the controller thread which runs all observers. This patch + # is added here to stop a warning that would otherwise be raised. + # pylint: disable=unused-argument + @mock.patch("psutil.cpu_times_percent") + @mock.patch("psutil.cpu_times") + def test_system_cpu_time(self, mock_cpu_times, mock_cpu_times_percent): + CPUTimes = namedtuple("CPUTimes", ["idle", "user", "system", "irq"]) + mock_cpu_times.return_value = [ + CPUTimes(idle=1.2, user=3.4, system=5.6, irq=7.8), + CPUTimes(idle=1.2, user=3.4, system=5.6, irq=7.8), + ] + + expected = { + ( + ("cpu", 1), + ("state", "idle"), + ): 1.2, + ( + ("cpu", 1), + ("state", "user"), + ): 3.4, + ( + ("cpu", 1), + ("state", "system"), + ): 5.6, + ( + ("cpu", 1), + ("state", "irq"), + ): 7.8, + ( + ("cpu", 2), + ("state", "idle"), + ): 1.2, + ( + ("cpu", 2), + ("state", "user"), + ): 3.4, + ( + ("cpu", 2), + ("state", "system"), + ): 5.6, + ( + ("cpu", 2), + ("state", "irq"), + ): 7.8, + } + self._test_metrics("system.cpu.time", expected) + + @mock.patch("psutil.cpu_times_percent") + def test_system_cpu_utilization(self, mock_cpu_times_percent): + CPUTimesPercent = namedtuple( + "CPUTimesPercent", ["idle", "user", "system", "irq"] + ) + mock_cpu_times_percent.return_value = [ + CPUTimesPercent(idle=1.2, user=3.4, system=5.6, irq=7.8), + CPUTimesPercent(idle=1.2, user=3.4, system=5.6, irq=7.8), + ] + + expected = { + (("cpu", 1), ("state", "idle"),): Measurement._TYPE( + min=1.2 / 100, + max=1.2 / 100, + sum=1.2 / 100, + count=1, + last=1.2 / 100, + ), + (("cpu", 1), ("state", "user"),): Measurement._TYPE( + min=3.4 / 100, + max=3.4 / 100, + sum=3.4 / 100, + count=1, + last=3.4 / 100, + ), + (("cpu", 1), ("state", "system"),): Measurement._TYPE( + min=5.6 / 100, + max=5.6 / 100, + sum=5.6 / 100, + count=1, + last=5.6 / 100, + ), + (("cpu", 1), ("state", "irq"),): Measurement._TYPE( + min=7.8 / 100, + max=7.8 / 100, + sum=7.8 / 100, + count=1, + last=7.8 / 100, + ), + (("cpu", 2), ("state", "idle"),): Measurement._TYPE( + min=1.2 / 100, + max=1.2 / 100, + sum=1.2 / 100, + count=1, + last=1.2 / 100, + ), + (("cpu", 2), ("state", "user"),): Measurement._TYPE( + min=3.4 / 100, + max=3.4 / 100, + sum=3.4 / 100, + count=1, + last=3.4 / 100, + ), + (("cpu", 2), ("state", "system"),): Measurement._TYPE( + min=5.6 / 100, + max=5.6 / 100, + sum=5.6 / 100, + count=1, + last=5.6 / 100, + ), + (("cpu", 2), ("state", "irq"),): Measurement._TYPE( + min=7.8 / 100, + max=7.8 / 100, + sum=7.8 / 100, + count=1, + last=7.8 / 100, + ), + } + self._test_metrics("system.cpu.utilization", expected) + + @mock.patch("psutil.virtual_memory") + def test_system_memory_usage(self, mock_virtual_memory): + VirtualMemory = namedtuple( + "VirtualMemory", ["used", "free", "cached", "total"] + ) + mock_virtual_memory.return_value = VirtualMemory( + used=1, free=2, cached=3, total=4 + ) + + expected = { + (("state", "used"),): Measurement._TYPE( + min=1, max=1, sum=1, count=1, last=1 + ), + (("state", "free"),): Measurement._TYPE( + min=2, max=2, sum=2, count=1, last=2 + ), + (("state", "cached"),): Measurement._TYPE( + min=3, max=3, sum=3, count=1, last=3 + ), + } + self._test_metrics("system.memory.usage", expected) + + @mock.patch("psutil.virtual_memory") + def test_system_memory_utilization(self, mock_virtual_memory): + VirtualMemory = namedtuple( + "VirtualMemory", ["used", "free", "cached", "total"] + ) + mock_virtual_memory.return_value = VirtualMemory( + used=1, free=2, cached=3, total=4 + ) + + expected = { + (("state", "used"),): Measurement._TYPE( + min=1 / 4, max=1 / 4, sum=1 / 4, count=1, last=1 / 4 + ), + (("state", "free"),): Measurement._TYPE( + min=2 / 4, max=2 / 4, sum=2 / 4, count=1, last=2 / 4 + ), + (("state", "cached"),): Measurement._TYPE( + min=3 / 4, max=3 / 4, sum=3 / 4, count=1, last=3 / 4 + ), + } + self._test_metrics("system.memory.utilization", expected) + + @mock.patch("psutil.swap_memory") + def test_system_swap_usage(self, mock_swap_memory): + SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) + mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) + + expected = { + (("state", "used"),): Measurement._TYPE( + min=1, max=1, sum=1, count=1, last=1 + ), + (("state", "free"),): Measurement._TYPE( + min=2, max=2, sum=2, count=1, last=2 + ), + } + self._test_metrics("system.swap.usage", expected) + + @mock.patch("psutil.swap_memory") + def test_system_swap_utilization(self, mock_swap_memory): + SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) + mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) + + expected = { + (("state", "used"),): Measurement._TYPE( + min=1 / 3, max=1 / 3, sum=1 / 3, count=1, last=1 / 3 + ), + (("state", "free"),): Measurement._TYPE( + min=2 / 3, max=2 / 3, sum=2 / 3, count=1, last=2 / 3 + ), + } + self._test_metrics("system.swap.utilization", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_io(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = { + ( + ("device", "sda"), + ("direction", "read"), + ): 3, + ( + ("device", "sda"), + ("direction", "write"), + ): 4, + ( + ("device", "sdb"), + ("direction", "read"), + ): 11, + ( + ("device", "sdb"), + ("direction", "write"), + ): 12, + } + self._test_metrics("system.disk.io", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_operations(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = { + ( + ("device", "sda"), + ("direction", "read"), + ): 1, + ( + ("device", "sda"), + ("direction", "write"), + ): 2, + ( + ("device", "sdb"), + ("direction", "read"), + ): 9, + ( + ("device", "sdb"), + ("direction", "write"), + ): 10, + } + self._test_metrics("system.disk.operations", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_time(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = { + ( + ("device", "sda"), + ("direction", "read"), + ): 5 + / 1000, + ( + ("device", "sda"), + ("direction", "write"), + ): 6 + / 1000, + ( + ("device", "sdb"), + ("direction", "read"), + ): 13 + / 1000, + ( + ("device", "sdb"), + ("direction", "write"), + ): 14 + / 1000, + } + self._test_metrics("system.disk.time", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_merged(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = { + ( + ("device", "sda"), + ("direction", "read"), + ): 7, + ( + ("device", "sda"), + ("direction", "write"), + ): 8, + ( + ("device", "sdb"), + ("direction", "read"), + ): 15, + ( + ("device", "sdb"), + ("direction", "write"), + ): 16, + } + self._test_metrics("system.disk.merged", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_dropped_packets(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = { + ( + ("device", "eth0"), + ("direction", "receive"), + ): 1, + ( + ("device", "eth0"), + ("direction", "transmit"), + ): 2, + ( + ("device", "eth1"), + ("direction", "receive"), + ): 9, + ( + ("device", "eth1"), + ("direction", "transmit"), + ): 10, + } + self._test_metrics("system.network.dropped_packets", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_packets(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = { + ( + ("device", "eth0"), + ("direction", "receive"), + ): 4, + ( + ("device", "eth0"), + ("direction", "transmit"), + ): 3, + ( + ("device", "eth1"), + ("direction", "receive"), + ): 12, + ( + ("device", "eth1"), + ("direction", "transmit"), + ): 11, + } + self._test_metrics("system.network.packets", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_errors(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = { + ( + ("device", "eth0"), + ("direction", "receive"), + ): 5, + ( + ("device", "eth0"), + ("direction", "transmit"), + ): 6, + ( + ("device", "eth1"), + ("direction", "receive"), + ): 13, + ( + ("device", "eth1"), + ("direction", "transmit"), + ): 14, + } + self._test_metrics("system.network.errors", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_io(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = { + ( + ("device", "eth0"), + ("direction", "receive"), + ): 8, + ( + ("device", "eth0"), + ("direction", "transmit"), + ): 7, + ( + ("device", "eth1"), + ("direction", "receive"), + ): 16, + ( + ("device", "eth1"), + ("direction", "transmit"), + ): 15, + } + self._test_metrics("system.network.io", expected) + + @mock.patch("psutil.net_connections") + def test_system_network_connections(self, mock_net_connections): + NetConnection = namedtuple( + "NetworkConnection", ["family", "type", "status"] + ) + Type = namedtuple("Type", ["value"]) + mock_net_connections.return_value = [ + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=2), + ), + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=1), + ), + ] + + expected = { + ( + ("family", 1), + ("protocol", "udp"), + ("state", "ESTABLISHED"), + ("type", Type(value=2)), + ): 1, + ( + ("family", 1), + ("protocol", "tcp"), + ("state", "ESTABLISHED"), + ("type", Type(value=1)), + ): 1, + } + self._test_metrics("system.network.connections", expected) + + @mock.patch("psutil.Process.memory_info") + def test_runtime_memory(self, mock_process_memory_info): + + PMem = namedtuple("PMem", ["rss", "vms"]) + + mock_process_memory_info.configure_mock( + **{"return_value": PMem(rss=1, vms=2)} + ) + + expected = { + (("type", "rss"),): 1, + (("type", "vms"),): 2, + } + self._test_metrics( + "runtime.{}.memory".format(self.implementation), expected + ) + + @mock.patch("psutil.Process.cpu_times") + def test_runtime_cpu_time(self, mock_process_cpu_times): + + PCPUTimes = namedtuple("PCPUTimes", ["user", "system"]) + + mock_process_cpu_times.configure_mock( + **{"return_value": PCPUTimes(user=1.1, system=2.2)} + ) + + expected = { + (("type", "user"),): 1.1, + (("type", "system"),): 2.2, + } + self._test_metrics( + "runtime.{}.cpu_time".format(self.implementation), expected + ) + + @mock.patch("gc.get_count") + def test_runtime_get_count(self, mock_gc_get_count): + + mock_gc_get_count.configure_mock(**{"return_value": (1, 2, 3)}) + + expected = { + (("count", "0"),): 1, + (("count", "1"),): 2, + (("count", "2"),): 3, + } + self._test_metrics( + "runtime.{}.gc_count".format(self.implementation), expected + ) diff --git a/tox.ini b/tox.ini index d2e1dd0f04..5f22749ba8 100644 --- a/tox.ini +++ b/tox.ini @@ -161,6 +161,10 @@ envlist = ; opentelemetry-instrumentation-sklearn py3{6,7,8}-test-instrumentation-sklearn + ; opentelemetry-instrumentation-system-metrics + py3{6,7,8,9,10}-test-instrumentation-system-metrics + pypy3-test-instrumentation-system-metrics + ; opentelemetry-instrumentation-tornado py3{6,7,8,9,10}-test-instrumentation-tornado pypy3-test-instrumentation-tornado @@ -272,6 +276,7 @@ changedir = test-instrumentation-sqlalchemy{11,14}: instrumentation/opentelemetry-instrumentation-sqlalchemy/tests test-instrumentation-sqlite3: instrumentation/opentelemetry-instrumentation-sqlite3/tests test-instrumentation-starlette: instrumentation/opentelemetry-instrumentation-starlette/tests + test-instrumentation-system-metrics: instrumentation/opentelemetry-instrumentation-system-metrics/tests test-instrumentation-tornado: instrumentation/opentelemetry-instrumentation-tornado/tests test-instrumentation-wsgi: instrumentation/opentelemetry-instrumentation-wsgi/tests test-instrumentation-httpx{18,21}: instrumentation/opentelemetry-instrumentation-httpx/tests @@ -350,6 +355,8 @@ commands_pre = starlette: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-starlette[test] + system-metrics: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics[test] + tornado: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-tornado[test] jinja2: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-jinja2[test] From 0529646adf1097f8857bbac7e5c8edaf11a9ea5c Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 1 Apr 2022 09:09:58 -0700 Subject: [PATCH 02/15] update changelog --- CHANGELOG.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ac49766f6..eae8a960e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,20 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased](https://github.com/open-telemetry/opentelemetry-python/compare/v1.10.0-0.29b0...HEAD) -### Added +### Fixed -- `opentelemetry-instrumentation-psycopg2` extended the sql commenter support of dbapi into psycopg2 - ([#940](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/940)) - `opentelemetry-instrumentation-flask` Fix non-recording span bug ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) - `opentelemetry-instrumentation-tornado` Fix non-recording span bug ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) - `opentelemetry-instrumentation-pyramid` Handle non-HTTPException exceptions ([#1001](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1001)) + +### Added + +- `opentelemetry-instrumentation-psycopg2` extended the sql commenter support of dbapi into psycopg2 + ([#940](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/940)) - `opentelemetry-instrumentation-falcon` Falcon: Capture custom request/response headers in span attributes ([#1003])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1003) - `opentelemetry-instrumentation-elasticsearch` no longer creates unique span names by including search target, replaces them with `` and puts the value in attribute `elasticsearch.target` ([#1018](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1018)) +- `opentelemetry-instrumentation-system-metrics` restore `SystemMetrics` instrumentation as `SystemMetricsInstrumentor` + ([#1012](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1012)) ## [1.10.0-0.29b0](https://github.com/open-telemetry/opentelemetry-python/releases/tag/v1.10.0-0.29b0) - 2022-03-10 From 0b59ab8b051489b6e07fc44525f868e405eed421 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 4 Apr 2022 11:03:17 -0700 Subject: [PATCH 03/15] fix tests --- .../system_metrics/__init__.py | 85 +-- .../instrumentation/system_metrics/package.py | 16 + .../tests/test_system_metrics.py | 558 ++++++------------ 3 files changed, 245 insertions(+), 414 deletions(-) create mode 100644 instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index a5ba6c58e4..d41ed6dcce 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -28,7 +28,6 @@ "system.disk.io": ["read", "write"], "system.disk.operations": ["read", "write"], "system.disk.time": ["read", "write"], - "system.disk.merged": ["read", "write"], "system.network.dropped.packets": ["transmit", "receive"], "system.network.packets": ["transmit", "receive"], "system.network.errors": ["transmit", "receive"], @@ -64,12 +63,14 @@ "runtime.memory": ["rss", "vms"], "runtime.cpu.time": ["user", "system"], } - SystemMetrics(exporter, config=configuration) + SystemMetrics(config=configuration) API --- """ +from typing import Collection + import gc import os from typing import Dict, List, Iterable, Optional @@ -77,24 +78,22 @@ import psutil -from opentelemetry._metrics import get_meter_provider +from opentelemetry._metrics import get_meter from opentelemetry._metrics.measurement import Measurement +from opentelemetry.instrumentation.instrumentor import BaseInstrumentor +from opentelemetry.instrumentation.system_metrics.version import __version__ +from opentelemetry.instrumentation.system_metrics.package import _instruments from opentelemetry.sdk.util import get_dict_as_key -class SystemMetrics: +class SystemMetrics(BaseInstrumentor): # pylint: disable=too-many-statements def __init__( self, - interval: int = 30, labels: Optional[Dict[str, str]] = None, config: Optional[Dict[str, List[str]]] = None, ): - self._labels = {} if labels is None else labels - self.meter = get_meter_provider().get_meter( - "io.otel.python.instrumentation.system-metrics" - ) # TODO: find a better name here - self._python_implementation = python_implementation().lower() + super().__init__() if config is None: self._config = { "system.cpu.time": ["idle", "user", "system", "irq"], @@ -108,7 +107,6 @@ def __init__( "system.disk.io": ["read", "write"], "system.disk.operations": ["read", "write"], "system.disk.time": ["read", "write"], - "system.disk.merged": ["read", "write"], # "system.filesystem.usage": [], # "system.filesystem.utilization": [], "system.network.dropped.packets": ["transmit", "receive"], @@ -121,6 +119,20 @@ def __init__( } else: self._config = config + self._labels = {} if labels is None else labels + self._meter = None + + def instrumentation_dependencies(self) -> Collection[str]: + return _instruments + + def _instrument(self, **kwargs): + meter_provider = kwargs.get("meter_provider") + self._meter = get_meter( + "io.otel.python.instrumentation.system_metrics", + __version__, + meter_provider, + ) + self._python_implementation = python_implementation().lower() self._proc = psutil.Process(os.getpid()) @@ -153,49 +165,49 @@ def __init__( self._runtime_cpu_time_labels = self._labels.copy() self._runtime_gc_count_labels = self._labels.copy() - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_cpu_time, name="system.cpu.time", description="System CPU time", unit="seconds", ) - self.meter.create_observable_gauge( + self._meter.create_observable_gauge( callback=self._get_system_cpu_utilization, name="system.cpu.utilization", description="System CPU utilization", unit="1", ) - self.meter.create_observable_gauge( + self._meter.create_observable_gauge( callback=self._get_system_memory_usage, name="system.memory.usage", description="System memory usage", unit="bytes", ) - self.meter.create_observable_gauge( + self._meter.create_observable_gauge( callback=self._get_system_memory_utilization, name="system.memory.utilization", description="System memory utilization", unit="1", ) - self.meter.create_observable_gauge( + self._meter.create_observable_gauge( callback=self._get_system_swap_usage, name="system.swap.usage", description="System swap usage", unit="pages", ) - self.meter.create_observable_gauge( + self._meter.create_observable_gauge( callback=self._get_system_swap_utilization, name="system.swap.utilization", description="System swap utilization", unit="1", ) - # # self.meter.create_observable_counter( + # # self._meter.create_observable_counter( # # callback=self._get_system_swap_page_faults, # # name="system.swap.page_faults", # # description="System swap page faults", @@ -203,42 +215,34 @@ def __init__( # # value_type=int, # # ) - # # self.meter.create_observable_counter( + # # self._meter.create_observable_counter( # # callback=self._get_system_swap_page_operations, # # name="system.swap.page_operations", # # description="System swap page operations", # # unit="operations", # # value_type=int, # # ) - - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_disk_io, name="system.disk.io", description="System disk IO", unit="bytes", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_disk_operations, name="system.disk.operations", description="System disk operations", unit="operations", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_disk_time, name="system.disk.time", description="System disk time", unit="seconds", ) - self.meter.create_observable_counter( - callback=self._get_system_disk_merged, - name="system.disk.merged", - description="System disk merged", - unit="1", - ) - # self.accumulator.register_valueobserver( # callback=self._get_system_filesystem_usage, # name="system.filesystem.usage", @@ -247,7 +251,7 @@ def __init__( # value_type=int, # ) - # self.meter.create_observable_gauge( + # self._meter.create_observable_gauge( # callback=self._get_system_filesystem_utilization, # name="system.filesystem.utilization", # description="System filesystem utilization", @@ -255,42 +259,42 @@ def __init__( # value_type=float, # ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_network_dropped_packets, name="system.network.dropped_packets", description="System network dropped_packets", unit="packets", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_network_packets, name="system.network.packets", description="System network packets", unit="packets", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_network_errors, name="system.network.errors", description="System network errors", unit="errors", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_system_network_io, name="system.network.io", description="System network io", unit="bytes", ) - self.meter.create_observable_up_down_counter( + self._meter.create_observable_up_down_counter( callback=self._get_system_network_connections, name="system.network.connections", description="System network connections", unit="connections", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_runtime_memory, name="runtime.{}.memory".format(self._python_implementation), description="Runtime {} memory".format( @@ -299,7 +303,7 @@ def __init__( unit="bytes", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_runtime_cpu_time, name="runtime.{}.cpu_time".format(self._python_implementation), description="Runtime {} CPU time".format( @@ -308,7 +312,7 @@ def __init__( unit="seconds", ) - self.meter.create_observable_counter( + self._meter.create_observable_counter( callback=self._get_runtime_gc_count, name="runtime.{}.gc_count".format(self._python_implementation), description="Runtime {} GC count".format( @@ -316,6 +320,9 @@ def __init__( ), unit="bytes", ) + + def _uninstrument(self, **__): + pass def _get_system_cpu_time(self) -> Iterable[Measurement]: """Observer callback for system CPU time diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py new file mode 100644 index 0000000000..7eb5b17a99 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py @@ -0,0 +1,16 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +_instruments = ("psutil >= 5.7.3",) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index e8ef6257dd..9b37bfeddb 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -21,22 +21,54 @@ from opentelemetry._metrics import get_meter_provider from opentelemetry._metrics.measurement import Measurement from opentelemetry.instrumentation.system_metrics import SystemMetrics +from opentelemetry.sdk._metrics import MeterProvider +from opentelemetry.sdk._metrics.export import InMemoryMetricReader from opentelemetry.test.test_base import TestBase +def _mock_netconnection(): + NetConnection = namedtuple( + "NetworkConnection", ["family", "type", "status"] + ) + Type = namedtuple("Type", ["value"]) + return [ + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=2), + ), + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=1), + ), + ] + +class _SystemMetricsResult(): + def __init__(self, attributes, value) -> None: + self.attributes = attributes + self.value = value + class TestSystemMetrics(TestBase): def setUp(self): super().setUp() self.implementation = python_implementation().lower() - - def test_system_metrics_constructor(self): - # ensure the observers have been registered - meter = get_meter_provider().get_meter(__name__) - with mock.patch("opentelemetry.metrics.get_meter") as mock_get_meter: - mock_get_meter.return_value = meter - SystemMetrics() - - self.assertEqual(len(meter.instruments), 18) + self._patch_net_connections = mock.patch("psutil.net_connections", _mock_netconnection) + self._patch_net_connections.start() + + def tearDown(self): + super().tearDown() + self._patch_net_connections.stop() + SystemMetrics().uninstrument() + + def test_system_metrics_instrument(self): + reader = InMemoryMetricReader() + meter_provider = MeterProvider(metric_readers=[reader]) + system_metrics = SystemMetrics() + system_metrics.instrument(meter_provider=meter_provider) + metrics = reader.get_metrics() + metric_names = list({x.name for x in metrics }) + self.assertEqual(len(metric_names), 17) observer_names = [ "system.cpu.time", @@ -48,7 +80,6 @@ def test_system_metrics_constructor(self): "system.disk.io", "system.disk.operations", "system.disk.time", - "system.disk.merged", "system.network.dropped_packets", "system.network.packets", "system.network.errors", @@ -59,36 +90,37 @@ def test_system_metrics_constructor(self): "runtime.{}.gc_count".format(self.implementation), ] - for observer in meter.instruments.values(): - self.assertIn(observer.name, observer_names) - observer_names.remove(observer.name) + for observer in metric_names: + self.assertIn(observer, observer_names) + observer_names.remove(observer) - def _assert_metrics(self, observer_name, system_metrics, expected): - system_metrics.controller.tick() + def _assert_metrics(self, observer_name, reader, expected): assertions = 0 for ( metric ) in ( - self.memory_metrics_exporter._exported_metrics # pylint: disable=protected-access + reader.get_metrics() # pylint: disable=protected-access ): - if ( - metric.labels in expected - and metric.instrument.name == observer_name - ): - self.assertEqual( - metric.aggregator.checkpoint, - expected[metric.labels], - ) - assertions += 1 + for e in expected: + if ( + metric.attributes == e.attributes + and metric.name == observer_name + ): + self.assertEqual( + metric.point.value, + e.value, + ) + assertions += 1 self.assertEqual(len(expected), assertions) def _test_metrics(self, observer_name, expected): - meter = self.meter_provider.get_meter(__name__) + reader = InMemoryMetricReader() + meter_provider = MeterProvider(metric_readers=[reader]) - with mock.patch("opentelemetry.metrics.get_meter") as mock_get_meter: - mock_get_meter.return_value = meter - system_metrics = SystemMetrics() - self._assert_metrics(observer_name, system_metrics, expected) + system_metrics = SystemMetrics() + system_metrics.instrument(meter_provider=meter_provider) + self._assert_metrics(observer_name, reader, expected) + system_metrics.uninstrument() # When this test case is executed, _get_system_cpu_utilization gets run # too because of the controller thread which runs all observers. This patch @@ -103,40 +135,40 @@ def test_system_cpu_time(self, mock_cpu_times, mock_cpu_times_percent): CPUTimes(idle=1.2, user=3.4, system=5.6, irq=7.8), ] - expected = { - ( - ("cpu", 1), - ("state", "idle"), - ): 1.2, - ( - ("cpu", 1), - ("state", "user"), - ): 3.4, - ( - ("cpu", 1), - ("state", "system"), - ): 5.6, - ( - ("cpu", 1), - ("state", "irq"), - ): 7.8, - ( - ("cpu", 2), - ("state", "idle"), - ): 1.2, - ( - ("cpu", 2), - ("state", "user"), - ): 3.4, - ( - ("cpu", 2), - ("state", "system"), - ): 5.6, - ( - ("cpu", 2), - ("state", "irq"), - ): 7.8, - } + expected = [ + _SystemMetricsResult({ + "cpu": 1, + "state": "idle", + }, 1.2), + _SystemMetricsResult({ + "cpu": 1, + "state": "user", + }, 3.4), + _SystemMetricsResult({ + "cpu": 1, + "state": "system", + }, 5.6), + _SystemMetricsResult({ + "cpu": 1, + "state": "irq", + }, 7.8), + _SystemMetricsResult({ + "cpu": 2, + "state": "idle", + }, 1.2), + _SystemMetricsResult({ + "cpu": 2, + "state": "user", + }, 3.4), + _SystemMetricsResult({ + "cpu": 2, + "state": "system", + }, 5.6), + _SystemMetricsResult({ + "cpu": 2, + "state": "irq", + }, 7.8), + ] self._test_metrics("system.cpu.time", expected) @mock.patch("psutil.cpu_times_percent") @@ -149,64 +181,16 @@ def test_system_cpu_utilization(self, mock_cpu_times_percent): CPUTimesPercent(idle=1.2, user=3.4, system=5.6, irq=7.8), ] - expected = { - (("cpu", 1), ("state", "idle"),): Measurement._TYPE( - min=1.2 / 100, - max=1.2 / 100, - sum=1.2 / 100, - count=1, - last=1.2 / 100, - ), - (("cpu", 1), ("state", "user"),): Measurement._TYPE( - min=3.4 / 100, - max=3.4 / 100, - sum=3.4 / 100, - count=1, - last=3.4 / 100, - ), - (("cpu", 1), ("state", "system"),): Measurement._TYPE( - min=5.6 / 100, - max=5.6 / 100, - sum=5.6 / 100, - count=1, - last=5.6 / 100, - ), - (("cpu", 1), ("state", "irq"),): Measurement._TYPE( - min=7.8 / 100, - max=7.8 / 100, - sum=7.8 / 100, - count=1, - last=7.8 / 100, - ), - (("cpu", 2), ("state", "idle"),): Measurement._TYPE( - min=1.2 / 100, - max=1.2 / 100, - sum=1.2 / 100, - count=1, - last=1.2 / 100, - ), - (("cpu", 2), ("state", "user"),): Measurement._TYPE( - min=3.4 / 100, - max=3.4 / 100, - sum=3.4 / 100, - count=1, - last=3.4 / 100, - ), - (("cpu", 2), ("state", "system"),): Measurement._TYPE( - min=5.6 / 100, - max=5.6 / 100, - sum=5.6 / 100, - count=1, - last=5.6 / 100, - ), - (("cpu", 2), ("state", "irq"),): Measurement._TYPE( - min=7.8 / 100, - max=7.8 / 100, - sum=7.8 / 100, - count=1, - last=7.8 / 100, - ), - } + expected = [ + _SystemMetricsResult({"cpu": 1, "state": "idle"}, 1.2 / 100), + _SystemMetricsResult({"cpu": 1, "state": "user"},3.4 / 100), + _SystemMetricsResult({"cpu": 1, "state": "system"}, 5.6 / 100), + _SystemMetricsResult({"cpu": 1, "state": "irq"}, 7.8 / 100), + _SystemMetricsResult({"cpu": 2, "state": "idle"}, 1.2 / 100), + _SystemMetricsResult({"cpu": 2, "state": "user"}, 3.4 / 100), + _SystemMetricsResult({"cpu": 2, "state": "system"}, 5.6 / 100), + _SystemMetricsResult({"cpu": 2, "state": "irq"}, 7.8 / 100), + ] self._test_metrics("system.cpu.utilization", expected) @mock.patch("psutil.virtual_memory") @@ -218,17 +202,11 @@ def test_system_memory_usage(self, mock_virtual_memory): used=1, free=2, cached=3, total=4 ) - expected = { - (("state", "used"),): Measurement._TYPE( - min=1, max=1, sum=1, count=1, last=1 - ), - (("state", "free"),): Measurement._TYPE( - min=2, max=2, sum=2, count=1, last=2 - ), - (("state", "cached"),): Measurement._TYPE( - min=3, max=3, sum=3, count=1, last=3 - ), - } + expected = [ + _SystemMetricsResult({"state": "used"}, 1), + _SystemMetricsResult({"state": "free"}, 2), + _SystemMetricsResult({"state": "cached"}, 3), + ] self._test_metrics("system.memory.usage", expected) @mock.patch("psutil.virtual_memory") @@ -240,17 +218,11 @@ def test_system_memory_utilization(self, mock_virtual_memory): used=1, free=2, cached=3, total=4 ) - expected = { - (("state", "used"),): Measurement._TYPE( - min=1 / 4, max=1 / 4, sum=1 / 4, count=1, last=1 / 4 - ), - (("state", "free"),): Measurement._TYPE( - min=2 / 4, max=2 / 4, sum=2 / 4, count=1, last=2 / 4 - ), - (("state", "cached"),): Measurement._TYPE( - min=3 / 4, max=3 / 4, sum=3 / 4, count=1, last=3 / 4 - ), - } + expected = [ + _SystemMetricsResult({"state": "used"}, 1 / 4), + _SystemMetricsResult({"state": "free"}, 2 / 4), + _SystemMetricsResult({"state": "cached"}, 3 / 4), + ] self._test_metrics("system.memory.utilization", expected) @mock.patch("psutil.swap_memory") @@ -258,14 +230,10 @@ def test_system_swap_usage(self, mock_swap_memory): SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) - expected = { - (("state", "used"),): Measurement._TYPE( - min=1, max=1, sum=1, count=1, last=1 - ), - (("state", "free"),): Measurement._TYPE( - min=2, max=2, sum=2, count=1, last=2 - ), - } + expected = [ + _SystemMetricsResult({"state": "used"}, 1), + _SystemMetricsResult({"state": "free"}, 2), + ] self._test_metrics("system.swap.usage", expected) @mock.patch("psutil.swap_memory") @@ -273,14 +241,10 @@ def test_system_swap_utilization(self, mock_swap_memory): SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) - expected = { - (("state", "used"),): Measurement._TYPE( - min=1 / 3, max=1 / 3, sum=1 / 3, count=1, last=1 / 3 - ), - (("state", "free"),): Measurement._TYPE( - min=2 / 3, max=2 / 3, sum=2 / 3, count=1, last=2 / 3 - ), - } + expected = [ + _SystemMetricsResult({"state": "used"}, 1/3), + _SystemMetricsResult({"state": "free"}, 2/3), + ] self._test_metrics("system.swap.utilization", expected) @mock.patch("psutil.disk_io_counters") @@ -321,24 +285,12 @@ def test_system_disk_io(self, mock_disk_io_counters): ), } - expected = { - ( - ("device", "sda"), - ("direction", "read"), - ): 3, - ( - ("device", "sda"), - ("direction", "write"), - ): 4, - ( - ("device", "sdb"), - ("direction", "read"), - ): 11, - ( - ("device", "sdb"), - ("direction", "write"), - ): 12, - } + expected = [ + _SystemMetricsResult({"device": "sda", "direction": "read"}, 3), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 4), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 11), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 12), + ] self._test_metrics("system.disk.io", expected) @mock.patch("psutil.disk_io_counters") @@ -379,24 +331,12 @@ def test_system_disk_operations(self, mock_disk_io_counters): ), } - expected = { - ( - ("device", "sda"), - ("direction", "read"), - ): 1, - ( - ("device", "sda"), - ("direction", "write"), - ): 2, - ( - ("device", "sdb"), - ("direction", "read"), - ): 9, - ( - ("device", "sdb"), - ("direction", "write"), - ): 10, - } + expected = [ + _SystemMetricsResult({"device": "sda", "direction": "read"}, 1), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 2), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 9), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 10), + ] self._test_metrics("system.disk.operations", expected) @mock.patch("psutil.disk_io_counters") @@ -437,88 +377,14 @@ def test_system_disk_time(self, mock_disk_io_counters): ), } - expected = { - ( - ("device", "sda"), - ("direction", "read"), - ): 5 - / 1000, - ( - ("device", "sda"), - ("direction", "write"), - ): 6 - / 1000, - ( - ("device", "sdb"), - ("direction", "read"), - ): 13 - / 1000, - ( - ("device", "sdb"), - ("direction", "write"), - ): 14 - / 1000, - } + expected = [ + _SystemMetricsResult({"device": "sda", "direction": "read"}, 5/ 1000), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 6/ 1000), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 13/ 1000), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 14/ 1000), + ] self._test_metrics("system.disk.time", expected) - @mock.patch("psutil.disk_io_counters") - def test_system_disk_merged(self, mock_disk_io_counters): - DiskIO = namedtuple( - "DiskIO", - [ - "read_count", - "write_count", - "read_bytes", - "write_bytes", - "read_time", - "write_time", - "read_merged_count", - "write_merged_count", - ], - ) - mock_disk_io_counters.return_value = { - "sda": DiskIO( - read_count=1, - write_count=2, - read_bytes=3, - write_bytes=4, - read_time=5, - write_time=6, - read_merged_count=7, - write_merged_count=8, - ), - "sdb": DiskIO( - read_count=9, - write_count=10, - read_bytes=11, - write_bytes=12, - read_time=13, - write_time=14, - read_merged_count=15, - write_merged_count=16, - ), - } - - expected = { - ( - ("device", "sda"), - ("direction", "read"), - ): 7, - ( - ("device", "sda"), - ("direction", "write"), - ): 8, - ( - ("device", "sdb"), - ("direction", "read"), - ): 15, - ( - ("device", "sdb"), - ("direction", "write"), - ): 16, - } - self._test_metrics("system.disk.merged", expected) - @mock.patch("psutil.net_io_counters") def test_system_network_dropped_packets(self, mock_net_io_counters): NetIO = namedtuple( @@ -557,24 +423,12 @@ def test_system_network_dropped_packets(self, mock_net_io_counters): ), } - expected = { - ( - ("device", "eth0"), - ("direction", "receive"), - ): 1, - ( - ("device", "eth0"), - ("direction", "transmit"), - ): 2, - ( - ("device", "eth1"), - ("direction", "receive"), - ): 9, - ( - ("device", "eth1"), - ("direction", "transmit"), - ): 10, - } + expected = [ + _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 1), + _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 2), + _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 9), + _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 10), + ] self._test_metrics("system.network.dropped_packets", expected) @mock.patch("psutil.net_io_counters") @@ -615,24 +469,12 @@ def test_system_network_packets(self, mock_net_io_counters): ), } - expected = { - ( - ("device", "eth0"), - ("direction", "receive"), - ): 4, - ( - ("device", "eth0"), - ("direction", "transmit"), - ): 3, - ( - ("device", "eth1"), - ("direction", "receive"), - ): 12, - ( - ("device", "eth1"), - ("direction", "transmit"), - ): 11, - } + expected = [ + _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 4), + _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 3), + _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 12), + _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 11), + ] self._test_metrics("system.network.packets", expected) @mock.patch("psutil.net_io_counters") @@ -673,24 +515,12 @@ def test_system_network_errors(self, mock_net_io_counters): ), } - expected = { - ( - ("device", "eth0"), - ("direction", "receive"), - ): 5, - ( - ("device", "eth0"), - ("direction", "transmit"), - ): 6, - ( - ("device", "eth1"), - ("direction", "receive"), - ): 13, - ( - ("device", "eth1"), - ("direction", "transmit"), - ): 14, - } + expected = [ + _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 5), + _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 6), + _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 13), + _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 14), + ] self._test_metrics("system.network.errors", expected) @mock.patch("psutil.net_io_counters") @@ -731,24 +561,12 @@ def test_system_network_io(self, mock_net_io_counters): ), } - expected = { - ( - ("device", "eth0"), - ("direction", "receive"), - ): 8, - ( - ("device", "eth0"), - ("direction", "transmit"), - ): 7, - ( - ("device", "eth1"), - ("direction", "receive"), - ): 16, - ( - ("device", "eth1"), - ("direction", "transmit"), - ): 15, - } + expected = [ + _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 8), + _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 7), + _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 16), + _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 15), + ] self._test_metrics("system.network.io", expected) @mock.patch("psutil.net_connections") @@ -770,20 +588,10 @@ def test_system_network_connections(self, mock_net_connections): ), ] - expected = { - ( - ("family", 1), - ("protocol", "udp"), - ("state", "ESTABLISHED"), - ("type", Type(value=2)), - ): 1, - ( - ("family", 1), - ("protocol", "tcp"), - ("state", "ESTABLISHED"), - ("type", Type(value=1)), - ): 1, - } + expected = [ + _SystemMetricsResult({"family": 1, "protocol": "udp", "state": "ESTABLISHED", "type": Type(value=2)}, 1), + _SystemMetricsResult({"family": 1, "protocol": "tcp", "state": "ESTABLISHED", "type": Type(value=1)}, 1), + ] self._test_metrics("system.network.connections", expected) @mock.patch("psutil.Process.memory_info") @@ -795,10 +603,10 @@ def test_runtime_memory(self, mock_process_memory_info): **{"return_value": PMem(rss=1, vms=2)} ) - expected = { - (("type", "rss"),): 1, - (("type", "vms"),): 2, - } + expected = [ + _SystemMetricsResult({"type": "rss"}, 1), + _SystemMetricsResult({"type": "vms"}, 2), + ] self._test_metrics( "runtime.{}.memory".format(self.implementation), expected ) @@ -812,10 +620,10 @@ def test_runtime_cpu_time(self, mock_process_cpu_times): **{"return_value": PCPUTimes(user=1.1, system=2.2)} ) - expected = { - (("type", "user"),): 1.1, - (("type", "system"),): 2.2, - } + expected = [ + _SystemMetricsResult({"type": "user"}, 1.1), + _SystemMetricsResult({"type": "system"}, 2.2), + ] self._test_metrics( "runtime.{}.cpu_time".format(self.implementation), expected ) @@ -825,11 +633,11 @@ def test_runtime_get_count(self, mock_gc_get_count): mock_gc_get_count.configure_mock(**{"return_value": (1, 2, 3)}) - expected = { - (("count", "0"),): 1, - (("count", "1"),): 2, - (("count", "2"),): 3, - } + expected = [ + _SystemMetricsResult({"count": "0"}, 1), + _SystemMetricsResult({"count": "1"}, 2), + _SystemMetricsResult({"count": "2"}, 3), + ] self._test_metrics( "runtime.{}.gc_count".format(self.implementation), expected ) From f105899027924f42c1fe6989caa4f8e490090fce Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 4 Apr 2022 11:07:13 -0700 Subject: [PATCH 04/15] update name to SystemMetricsInstrumentor --- .../setup.cfg | 4 ++++ .../instrumentation/system_metrics/__init__.py | 8 ++++---- .../tests/test_system_metrics.py | 8 ++++---- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg index 918a62c293..e54d7338a0 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg @@ -50,3 +50,7 @@ test = [options.packages.find] where = src + +[options.entry_points] +opentelemetry_instrumentor = + system_metrics = opentelemetry.instrumentation.system_metrics:SystemMetricsInstrumentor diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index d41ed6dcce..dd30e6227c 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -43,14 +43,14 @@ .. code:: python from opentelemetry._metrics import set_meter_provider - from opentelemetry.instrumentation.system_metrics import SystemMetrics + from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor from opentelemetry.sdk._metrics import MeterProvider from opentelemetry.sdk._metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader exporter = ConsoleMetricExporter() set_meter_provider(MeterProvider(PeriodicExportingMetricReader(exporter))) - SystemMetrics() + SystemMetricsInstrumentor().instrument() # metrics are collected asynchronously input("...") @@ -63,7 +63,7 @@ "runtime.memory": ["rss", "vms"], "runtime.cpu.time": ["user", "system"], } - SystemMetrics(config=configuration) + SystemMetricsInstrumentor(config=configuration).instrument() API --- @@ -86,7 +86,7 @@ from opentelemetry.sdk.util import get_dict_as_key -class SystemMetrics(BaseInstrumentor): +class SystemMetricsInstrumentor(BaseInstrumentor): # pylint: disable=too-many-statements def __init__( self, diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index 9b37bfeddb..60a96045bc 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -20,7 +20,7 @@ from opentelemetry._metrics import get_meter_provider from opentelemetry._metrics.measurement import Measurement -from opentelemetry.instrumentation.system_metrics import SystemMetrics +from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor from opentelemetry.sdk._metrics import MeterProvider from opentelemetry.sdk._metrics.export import InMemoryMetricReader from opentelemetry.test.test_base import TestBase @@ -59,12 +59,12 @@ def setUp(self): def tearDown(self): super().tearDown() self._patch_net_connections.stop() - SystemMetrics().uninstrument() + SystemMetricsInstrumentor().uninstrument() def test_system_metrics_instrument(self): reader = InMemoryMetricReader() meter_provider = MeterProvider(metric_readers=[reader]) - system_metrics = SystemMetrics() + system_metrics = SystemMetricsInstrumentor() system_metrics.instrument(meter_provider=meter_provider) metrics = reader.get_metrics() metric_names = list({x.name for x in metrics }) @@ -117,7 +117,7 @@ def _test_metrics(self, observer_name, expected): reader = InMemoryMetricReader() meter_provider = MeterProvider(metric_readers=[reader]) - system_metrics = SystemMetrics() + system_metrics = SystemMetricsInstrumentor() system_metrics.instrument(meter_provider=meter_provider) self._assert_metrics(observer_name, reader, expected) system_metrics.uninstrument() From 876e7f9d58d7ca4ae0d516192178895f162a6534 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 4 Apr 2022 11:11:51 -0700 Subject: [PATCH 05/15] fix lint --- .../system_metrics/__init__.py | 2 +- .../tests/test_system_metrics.py | 224 ++++++++++++------ 2 files changed, 154 insertions(+), 72 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index dd30e6227c..2bc5aa2136 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -320,7 +320,7 @@ def _instrument(self, **kwargs): ), unit="bytes", ) - + def _uninstrument(self, **__): pass diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index 60a96045bc..4f3bbabec4 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -20,7 +20,9 @@ from opentelemetry._metrics import get_meter_provider from opentelemetry._metrics.measurement import Measurement -from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor +from opentelemetry.instrumentation.system_metrics import ( + SystemMetricsInstrumentor, +) from opentelemetry.sdk._metrics import MeterProvider from opentelemetry.sdk._metrics.export import InMemoryMetricReader from opentelemetry.test.test_base import TestBase @@ -44,18 +46,22 @@ def _mock_netconnection(): ), ] -class _SystemMetricsResult(): + +class _SystemMetricsResult: def __init__(self, attributes, value) -> None: self.attributes = attributes self.value = value + class TestSystemMetrics(TestBase): def setUp(self): super().setUp() self.implementation = python_implementation().lower() - self._patch_net_connections = mock.patch("psutil.net_connections", _mock_netconnection) + self._patch_net_connections = mock.patch( + "psutil.net_connections", _mock_netconnection + ) self._patch_net_connections.start() - + def tearDown(self): super().tearDown() self._patch_net_connections.stop() @@ -67,7 +73,7 @@ def test_system_metrics_instrument(self): system_metrics = SystemMetricsInstrumentor() system_metrics.instrument(meter_provider=meter_provider) metrics = reader.get_metrics() - metric_names = list({x.name for x in metrics }) + metric_names = list({x.name for x in metrics}) self.assertEqual(len(metric_names), 17) observer_names = [ @@ -96,11 +102,7 @@ def test_system_metrics_instrument(self): def _assert_metrics(self, observer_name, reader, expected): assertions = 0 - for ( - metric - ) in ( - reader.get_metrics() # pylint: disable=protected-access - ): + for metric in reader.get_metrics(): # pylint: disable=protected-access for e in expected: if ( metric.attributes == e.attributes @@ -136,38 +138,62 @@ def test_system_cpu_time(self, mock_cpu_times, mock_cpu_times_percent): ] expected = [ - _SystemMetricsResult({ - "cpu": 1, - "state": "idle", - }, 1.2), - _SystemMetricsResult({ - "cpu": 1, - "state": "user", - }, 3.4), - _SystemMetricsResult({ - "cpu": 1, - "state": "system", - }, 5.6), - _SystemMetricsResult({ - "cpu": 1, - "state": "irq", - }, 7.8), - _SystemMetricsResult({ - "cpu": 2, - "state": "idle", - }, 1.2), - _SystemMetricsResult({ - "cpu": 2, - "state": "user", - }, 3.4), - _SystemMetricsResult({ - "cpu": 2, - "state": "system", - }, 5.6), - _SystemMetricsResult({ - "cpu": 2, - "state": "irq", - }, 7.8), + _SystemMetricsResult( + { + "cpu": 1, + "state": "idle", + }, + 1.2, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "user", + }, + 3.4, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "system", + }, + 5.6, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "irq", + }, + 7.8, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "idle", + }, + 1.2, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "user", + }, + 3.4, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "system", + }, + 5.6, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "irq", + }, + 7.8, + ), ] self._test_metrics("system.cpu.time", expected) @@ -183,7 +209,7 @@ def test_system_cpu_utilization(self, mock_cpu_times_percent): expected = [ _SystemMetricsResult({"cpu": 1, "state": "idle"}, 1.2 / 100), - _SystemMetricsResult({"cpu": 1, "state": "user"},3.4 / 100), + _SystemMetricsResult({"cpu": 1, "state": "user"}, 3.4 / 100), _SystemMetricsResult({"cpu": 1, "state": "system"}, 5.6 / 100), _SystemMetricsResult({"cpu": 1, "state": "irq"}, 7.8 / 100), _SystemMetricsResult({"cpu": 2, "state": "idle"}, 1.2 / 100), @@ -242,8 +268,8 @@ def test_system_swap_utilization(self, mock_swap_memory): mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) expected = [ - _SystemMetricsResult({"state": "used"}, 1/3), - _SystemMetricsResult({"state": "free"}, 2/3), + _SystemMetricsResult({"state": "used"}, 1 / 3), + _SystemMetricsResult({"state": "free"}, 2 / 3), ] self._test_metrics("system.swap.utilization", expected) @@ -332,10 +358,10 @@ def test_system_disk_operations(self, mock_disk_io_counters): } expected = [ - _SystemMetricsResult({"device": "sda", "direction": "read"}, 1), - _SystemMetricsResult({"device": "sda", "direction": "write"}, 2), - _SystemMetricsResult({"device": "sdb", "direction": "read"}, 9), - _SystemMetricsResult({"device": "sdb", "direction": "write"}, 10), + _SystemMetricsResult({"device": "sda", "direction": "read"}, 1), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 2), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 9), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 10), ] self._test_metrics("system.disk.operations", expected) @@ -378,10 +404,18 @@ def test_system_disk_time(self, mock_disk_io_counters): } expected = [ - _SystemMetricsResult({"device": "sda", "direction": "read"}, 5/ 1000), - _SystemMetricsResult({"device": "sda", "direction": "write"}, 6/ 1000), - _SystemMetricsResult({"device": "sdb", "direction": "read"}, 13/ 1000), - _SystemMetricsResult({"device": "sdb", "direction": "write"}, 14/ 1000), + _SystemMetricsResult( + {"device": "sda", "direction": "read"}, 5 / 1000 + ), + _SystemMetricsResult( + {"device": "sda", "direction": "write"}, 6 / 1000 + ), + _SystemMetricsResult( + {"device": "sdb", "direction": "read"}, 13 / 1000 + ), + _SystemMetricsResult( + {"device": "sdb", "direction": "write"}, 14 / 1000 + ), ] self._test_metrics("system.disk.time", expected) @@ -424,10 +458,18 @@ def test_system_network_dropped_packets(self, mock_net_io_counters): } expected = [ - _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 1), - _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 2), - _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 9), - _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 10), + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 1 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 2 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 9 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 10 + ), ] self._test_metrics("system.network.dropped_packets", expected) @@ -470,10 +512,18 @@ def test_system_network_packets(self, mock_net_io_counters): } expected = [ - _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 4), - _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 3), - _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 12), - _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 11), + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 4 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 3 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 12 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 11 + ), ] self._test_metrics("system.network.packets", expected) @@ -516,10 +566,18 @@ def test_system_network_errors(self, mock_net_io_counters): } expected = [ - _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 5), - _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 6), - _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 13), - _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 14), + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 5 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 6 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 13 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 14 + ), ] self._test_metrics("system.network.errors", expected) @@ -562,10 +620,18 @@ def test_system_network_io(self, mock_net_io_counters): } expected = [ - _SystemMetricsResult({"device": "eth0", "direction": "receive"}, 8), - _SystemMetricsResult({"device": "eth0", "direction": "transmit"}, 7), - _SystemMetricsResult({"device": "eth1", "direction": "receive"}, 16), - _SystemMetricsResult({"device": "eth1", "direction": "transmit"}, 15), + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 8 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 7 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 16 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 15 + ), ] self._test_metrics("system.network.io", expected) @@ -589,8 +655,24 @@ def test_system_network_connections(self, mock_net_connections): ] expected = [ - _SystemMetricsResult({"family": 1, "protocol": "udp", "state": "ESTABLISHED", "type": Type(value=2)}, 1), - _SystemMetricsResult({"family": 1, "protocol": "tcp", "state": "ESTABLISHED", "type": Type(value=1)}, 1), + _SystemMetricsResult( + { + "family": 1, + "protocol": "udp", + "state": "ESTABLISHED", + "type": Type(value=2), + }, + 1, + ), + _SystemMetricsResult( + { + "family": 1, + "protocol": "tcp", + "state": "ESTABLISHED", + "type": Type(value=1), + }, + 1, + ), ] self._test_metrics("system.network.connections", expected) From d0cb264a6519d11282b2f481ad389d78ad765cdb Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 4 Apr 2022 12:48:34 -0700 Subject: [PATCH 06/15] tox -e generate --- instrumentation/README.md | 1 + .../setup.py | 74 ++++++++++++++++++- .../system_metrics/__init__.py | 6 +- .../setup.cfg | 1 + .../instrumentation/bootstrap_gen.py | 4 + 5 files changed, 79 insertions(+), 7 deletions(-) diff --git a/instrumentation/README.md b/instrumentation/README.md index 1421653a70..a4e880d87a 100644 --- a/instrumentation/README.md +++ b/instrumentation/README.md @@ -33,6 +33,7 @@ | [opentelemetry-instrumentation-sqlalchemy](./opentelemetry-instrumentation-sqlalchemy) | sqlalchemy | | [opentelemetry-instrumentation-sqlite3](./opentelemetry-instrumentation-sqlite3) | sqlite3 | | [opentelemetry-instrumentation-starlette](./opentelemetry-instrumentation-starlette) | starlette ~= 0.13.0 | +| [opentelemetry-instrumentation-system-metrics](./opentelemetry-instrumentation-system-metrics) | psutil >= 5.7.3 | | [opentelemetry-instrumentation-tornado](./opentelemetry-instrumentation-tornado) | tornado >= 5.1.1 | | [opentelemetry-instrumentation-urllib](./opentelemetry-instrumentation-urllib) | urllib | | [opentelemetry-instrumentation-urllib3](./opentelemetry-instrumentation-urllib3) | urllib3 >= 1.0.0, < 2.0.0 | diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py index f0bbf9eff0..a3baaf78f9 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py @@ -11,11 +11,33 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + + +# DO NOT EDIT. THIS FILE WAS AUTOGENERATED FROM templates/instrumentation_setup.py.txt. +# RUN `python scripts/generate_setup.py` TO REGENERATE. + + +import distutils.cmd +import json import os +from configparser import ConfigParser import setuptools +config = ConfigParser() +config.read("setup.cfg") + +# We provide extras_require parameter to setuptools.setup later which +# overwrites the extras_require section from setup.cfg. To support extras_require +# section in setup.cfg, we load it here and merge it with the extras_require param. +extras_require = {} +if "options.extras_require" in config: + for key, value in config["options.extras_require"].items(): + extras_require[key] = [v for v in value.split("\n") if v.strip()] + BASE_DIR = os.path.dirname(__file__) +PACKAGE_INFO = {} + VERSION_FILENAME = os.path.join( BASE_DIR, "src", @@ -24,8 +46,54 @@ "system_metrics", "version.py", ) -PACKAGE_INFO = {} -with open(VERSION_FILENAME) as f: +with open(VERSION_FILENAME, encoding="utf-8") as f: exec(f.read(), PACKAGE_INFO) -setuptools.setup(version=PACKAGE_INFO["__version__"]) +PACKAGE_FILENAME = os.path.join( + BASE_DIR, + "src", + "opentelemetry", + "instrumentation", + "system_metrics", + "package.py", +) +with open(PACKAGE_FILENAME, encoding="utf-8") as f: + exec(f.read(), PACKAGE_INFO) + +# Mark any instruments/runtime dependencies as test dependencies as well. +extras_require["instruments"] = PACKAGE_INFO["_instruments"] +test_deps = extras_require.get("test", []) +for dep in extras_require["instruments"]: + test_deps.append(dep) + +extras_require["test"] = test_deps + + +class JSONMetadataCommand(distutils.cmd.Command): + + description = ( + "print out package metadata as JSON. This is used by OpenTelemetry dev scripts to ", + "auto-generate code in other places", + ) + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + metadata = { + "name": config["metadata"]["name"], + "version": PACKAGE_INFO["__version__"], + "instruments": PACKAGE_INFO["_instruments"], + } + print(json.dumps(metadata)) + + +setuptools.setup( + cmdclass={"meta": JSONMetadataCommand}, + version=PACKAGE_INFO["__version__"], + extras_require=extras_require, +) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index 2bc5aa2136..77311b7564 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -69,20 +69,18 @@ --- """ -from typing import Collection - import gc import os -from typing import Dict, List, Iterable, Optional from platform import python_implementation +from typing import Collection, Dict, Iterable, List, Optional import psutil from opentelemetry._metrics import get_meter from opentelemetry._metrics.measurement import Measurement from opentelemetry.instrumentation.instrumentor import BaseInstrumentor -from opentelemetry.instrumentation.system_metrics.version import __version__ from opentelemetry.instrumentation.system_metrics.package import _instruments +from opentelemetry.instrumentation.system_metrics.version import __version__ from opentelemetry.sdk.util import get_dict_as_key diff --git a/opentelemetry-contrib-instrumentations/setup.cfg b/opentelemetry-contrib-instrumentations/setup.cfg index 4fb943ec9a..730f1ca55e 100644 --- a/opentelemetry-contrib-instrumentations/setup.cfg +++ b/opentelemetry-contrib-instrumentations/setup.cfg @@ -60,6 +60,7 @@ install_requires = opentelemetry-instrumentation-sqlalchemy==0.29b0 opentelemetry-instrumentation-sqlite3==0.29b0 opentelemetry-instrumentation-starlette==0.29b0 + opentelemetry-instrumentation-system-metrics==0.29b0 opentelemetry-instrumentation-tornado==0.29b0 opentelemetry-instrumentation-urllib==0.29b0 opentelemetry-instrumentation-urllib3==0.29b0 diff --git a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py index a159d3bae9..ae8d86b08f 100644 --- a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py +++ b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py @@ -128,6 +128,10 @@ "library": "starlette ~= 0.13.0", "instrumentation": "opentelemetry-instrumentation-starlette==0.29b0", }, + "psutil": { + "library": "psutil >= 5.7.3", + "instrumentation": "opentelemetry-instrumentation-system-metrics==0.29b0", + }, "tornado": { "library": "tornado >= 5.1.1", "instrumentation": "opentelemetry-instrumentation-tornado==0.29b0", From 0398a0666e678367c5e3d72cf8260891c0990c84 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 4 Apr 2022 14:02:05 -0700 Subject: [PATCH 07/15] fix lint --- .../tests/test_system_metrics.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index 4f3bbabec4..987f4dafdb 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -18,8 +18,6 @@ from platform import python_implementation from unittest import mock -from opentelemetry._metrics import get_meter_provider -from opentelemetry._metrics.measurement import Measurement from opentelemetry.instrumentation.system_metrics import ( SystemMetricsInstrumentor, ) From 8f81b8cb3b7dc5027a36b8aee3c0e4c1f85d7dd5 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 5 Apr 2022 14:32:42 -0700 Subject: [PATCH 08/15] update sha --- .github/workflows/test.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index dc58ca283b..203deb8568 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -6,7 +6,7 @@ on: - 'release/*' pull_request: env: - CORE_REPO_SHA: eb98182acc54faeffc7440f9fbce8e71b4575fcc + CORE_REPO_SHA: f81381cf8aca64a707d934f20c6c27d40b949dce jobs: build: From 6c15fa0be78796e60258ceaf68ca332291e68aa5 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 5 Apr 2022 15:15:15 -0700 Subject: [PATCH 09/15] fix lint --- .../system_metrics/__init__.py | 74 +++++++++---------- .../tests/test_system_metrics.py | 19 +++-- 2 files changed, 43 insertions(+), 50 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index 77311b7564..0c1958ad80 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -119,17 +119,6 @@ def __init__( self._config = config self._labels = {} if labels is None else labels self._meter = None - - def instrumentation_dependencies(self) -> Collection[str]: - return _instruments - - def _instrument(self, **kwargs): - meter_provider = kwargs.get("meter_provider") - self._meter = get_meter( - "io.otel.python.instrumentation.system_metrics", - __version__, - meter_provider, - ) self._python_implementation = python_implementation().lower() self._proc = psutil.Process(os.getpid()) @@ -163,6 +152,17 @@ def _instrument(self, **kwargs): self._runtime_cpu_time_labels = self._labels.copy() self._runtime_gc_count_labels = self._labels.copy() + def instrumentation_dependencies(self) -> Collection[str]: + return _instruments + + def _instrument(self, **kwargs): + meter_provider = kwargs.get("meter_provider") + self._meter = get_meter( + __name__, + __version__, + meter_provider, + ) + self._meter.create_observable_counter( callback=self._get_system_cpu_time, name="system.cpu.time", @@ -294,28 +294,22 @@ def _instrument(self, **kwargs): self._meter.create_observable_counter( callback=self._get_runtime_memory, - name="runtime.{}.memory".format(self._python_implementation), - description="Runtime {} memory".format( - self._python_implementation - ), + name=f"runtime.{self._python_implementation}.memory", + description=f"Runtime {self._python_implementation} memory", unit="bytes", ) self._meter.create_observable_counter( callback=self._get_runtime_cpu_time, - name="runtime.{}.cpu_time".format(self._python_implementation), - description="Runtime {} CPU time".format( - self._python_implementation - ), + name=f"runtime.{self._python_implementation}.cpu_time", + description=f"Runtime {self._python_implementation} CPU time", unit="seconds", ) self._meter.create_observable_counter( callback=self._get_runtime_gc_count, - name="runtime.{}.gc_count".format(self._python_implementation), - description="Runtime {} GC count".format( - self._python_implementation - ), + name=f"runtime.{self._python_implementation}.gc_count", + description=f"Runtime {self._python_implementation} GC count", unit="bytes", ) @@ -430,11 +424,11 @@ def _get_system_disk_io(self) -> Iterable[Measurement]: """ for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.io"]: - if hasattr(counters, "{}_bytes".format(metric)): + if hasattr(counters, f"{metric}_bytes"): self._system_disk_io_labels["device"] = device self._system_disk_io_labels["direction"] = metric yield Measurement( - getattr(counters, "{}_bytes".format(metric)), + getattr(counters, f"{metric}_bytes"), self._system_disk_io_labels, ) @@ -446,11 +440,11 @@ def _get_system_disk_operations(self) -> Iterable[Measurement]: """ for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.operations"]: - if hasattr(counters, "{}_count".format(metric)): + if hasattr(counters, f"{metric}_count"): self._system_disk_operations_labels["device"] = device self._system_disk_operations_labels["direction"] = metric yield Measurement( - getattr(counters, "{}_count".format(metric)), + getattr(counters, f"{metric}_count"), self._system_disk_operations_labels, ) @@ -462,11 +456,11 @@ def _get_system_disk_time(self) -> Iterable[Measurement]: """ for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.time"]: - if hasattr(counters, "{}_time".format(metric)): + if hasattr(counters, f"{metric}_time"): self._system_disk_time_labels["device"] = device self._system_disk_time_labels["direction"] = metric yield Measurement( - getattr(counters, "{}_time".format(metric)) / 1000, + getattr(counters, f"{metric}_time") / 1000, self._system_disk_time_labels, ) @@ -482,11 +476,11 @@ def _get_system_disk_merged(self) -> Iterable[Measurement]: for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.time"]: - if hasattr(counters, "{}_merged_count".format(metric)): + if hasattr(counters, f"{metric}_merged_count"): self._system_disk_merged_labels["device"] = device self._system_disk_merged_labels["direction"] = metric yield Measurement( - getattr(counters, "{}_merged_count".format(metric)), + getattr(counters, f"{metric}_merged_count"), self._system_disk_merged_labels, ) @@ -505,7 +499,7 @@ def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: in_out = {"receive": "in", "transmit": "out"}[metric] - if hasattr(counters, "drop{}".format(in_out)): + if hasattr(counters, f"drop{in_out}"): self._system_network_dropped_packets_labels[ "device" ] = device @@ -513,7 +507,7 @@ def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: "direction" ] = metric yield Measurement( - getattr(counters, "drop{}".format(in_out)), + getattr(counters, f"drop{in_out}"), self._system_network_dropped_packets_labels, ) @@ -527,11 +521,11 @@ def _get_system_network_packets(self) -> Iterable[Measurement]: for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: recv_sent = {"receive": "recv", "transmit": "sent"}[metric] - if hasattr(counters, "packets_{}".format(recv_sent)): + if hasattr(counters, f"packets_{recv_sent}"): self._system_network_packets_labels["device"] = device self._system_network_packets_labels["direction"] = metric yield Measurement( - getattr(counters, "packets_{}".format(recv_sent)), + getattr(counters, f"packets_{recv_sent}"), self._system_network_packets_labels, ) @@ -544,11 +538,11 @@ def _get_system_network_errors(self) -> Iterable[Measurement]: for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.errors"]: in_out = {"receive": "in", "transmit": "out"}[metric] - if hasattr(counters, "err{}".format(in_out)): + if hasattr(counters, f"err{in_out}"): self._system_network_errors_labels["device"] = device self._system_network_errors_labels["direction"] = metric yield Measurement( - getattr(counters, "err{}".format(in_out)), + getattr(counters, f"err{in_out}"), self._system_network_errors_labels, ) @@ -562,11 +556,11 @@ def _get_system_network_io(self) -> Iterable[Measurement]: for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: recv_sent = {"receive": "recv", "transmit": "sent"}[metric] - if hasattr(counters, "bytes_{}".format(recv_sent)): + if hasattr(counters, f"bytes_{recv_sent}"): self._system_network_io_labels["device"] = device self._system_network_io_labels["direction"] = metric yield Measurement( - getattr(counters, "bytes_{}".format(recv_sent)), + getattr(counters, f"bytes_{recv_sent}"), self._system_network_io_labels, ) @@ -598,7 +592,7 @@ def _get_system_network_connections(self) -> Iterable[Measurement]: self._system_network_connections_labels ) - if connection_counters_key in connection_counters.keys(): + if connection_counters_key in connection_counters: connection_counters[connection_counters_key]["counter"] += 1 else: connection_counters[connection_counters_key] = { diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index 987f4dafdb..c28c79a9f7 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -89,9 +89,9 @@ def test_system_metrics_instrument(self): "system.network.errors", "system.network.io", "system.network.connections", - "runtime.{}.memory".format(self.implementation), - "runtime.{}.cpu_time".format(self.implementation), - "runtime.{}.gc_count".format(self.implementation), + f"runtime.{self.implementation}.memory", + f"runtime.{self.implementation}.cpu_time", + f"runtime.{self.implementation}.gc_count", ] for observer in metric_names: @@ -101,14 +101,14 @@ def test_system_metrics_instrument(self): def _assert_metrics(self, observer_name, reader, expected): assertions = 0 for metric in reader.get_metrics(): # pylint: disable=protected-access - for e in expected: + for expect in expected: if ( - metric.attributes == e.attributes + metric.attributes == expect.attributes and metric.name == observer_name ): self.assertEqual( metric.point.value, - e.value, + expect.value, ) assertions += 1 self.assertEqual(len(expected), assertions) @@ -120,7 +120,6 @@ def _test_metrics(self, observer_name, expected): system_metrics = SystemMetricsInstrumentor() system_metrics.instrument(meter_provider=meter_provider) self._assert_metrics(observer_name, reader, expected) - system_metrics.uninstrument() # When this test case is executed, _get_system_cpu_utilization gets run # too because of the controller thread which runs all observers. This patch @@ -688,7 +687,7 @@ def test_runtime_memory(self, mock_process_memory_info): _SystemMetricsResult({"type": "vms"}, 2), ] self._test_metrics( - "runtime.{}.memory".format(self.implementation), expected + f"runtime.{self.implementation}.memory", expected ) @mock.patch("psutil.Process.cpu_times") @@ -705,7 +704,7 @@ def test_runtime_cpu_time(self, mock_process_cpu_times): _SystemMetricsResult({"type": "system"}, 2.2), ] self._test_metrics( - "runtime.{}.cpu_time".format(self.implementation), expected + f"runtime.{self.implementation}.cpu_time", expected ) @mock.patch("gc.get_count") @@ -719,5 +718,5 @@ def test_runtime_get_count(self, mock_gc_get_count): _SystemMetricsResult({"count": "2"}, 3), ] self._test_metrics( - "runtime.{}.gc_count".format(self.implementation), expected + f"runtime.{self.implementation}.gc_count", expected ) From d4b54da0e4aad35f705e48e1f4992ae7fc86ffd9 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 5 Apr 2022 15:20:52 -0700 Subject: [PATCH 10/15] fix lint --- .../tests/test_system_metrics.py | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py index c28c79a9f7..aeee669f9b 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -121,9 +121,8 @@ def _test_metrics(self, observer_name, expected): system_metrics.instrument(meter_provider=meter_provider) self._assert_metrics(observer_name, reader, expected) - # When this test case is executed, _get_system_cpu_utilization gets run - # too because of the controller thread which runs all observers. This patch - # is added here to stop a warning that would otherwise be raised. + # This patch is added here to stop psutil from raising an exception + # because we're patching cpu_times # pylint: disable=unused-argument @mock.patch("psutil.cpu_times_percent") @mock.patch("psutil.cpu_times") @@ -686,9 +685,7 @@ def test_runtime_memory(self, mock_process_memory_info): _SystemMetricsResult({"type": "rss"}, 1), _SystemMetricsResult({"type": "vms"}, 2), ] - self._test_metrics( - f"runtime.{self.implementation}.memory", expected - ) + self._test_metrics(f"runtime.{self.implementation}.memory", expected) @mock.patch("psutil.Process.cpu_times") def test_runtime_cpu_time(self, mock_process_cpu_times): @@ -703,9 +700,7 @@ def test_runtime_cpu_time(self, mock_process_cpu_times): _SystemMetricsResult({"type": "user"}, 1.1), _SystemMetricsResult({"type": "system"}, 2.2), ] - self._test_metrics( - f"runtime.{self.implementation}.cpu_time", expected - ) + self._test_metrics(f"runtime.{self.implementation}.cpu_time", expected) @mock.patch("gc.get_count") def test_runtime_get_count(self, mock_gc_get_count): @@ -717,6 +712,4 @@ def test_runtime_get_count(self, mock_gc_get_count): _SystemMetricsResult({"count": "1"}, 2), _SystemMetricsResult({"count": "2"}, 3), ] - self._test_metrics( - f"runtime.{self.implementation}.gc_count", expected - ) + self._test_metrics(f"runtime.{self.implementation}.gc_count", expected) From d9e6dda134b3b30f91d7cccbb798eae60a0f085c Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 5 Apr 2022 15:34:09 -0700 Subject: [PATCH 11/15] forgot tox.ini --- tox.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/tox.ini b/tox.ini index 5f22749ba8..41d837b6da 100644 --- a/tox.ini +++ b/tox.ini @@ -468,6 +468,7 @@ commands_pre = python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-mysql[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-httpx[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aws-lambda[test] + python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics[test] python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-datadog[test] python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-richconsole[test] python -m pip install -e {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws[test] From 9445b90650a1fa6145b0b399d9083c90aa437556 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 6 Apr 2022 08:30:16 -0700 Subject: [PATCH 12/15] update tox to skip pypy3 tests --- .../opentelemetry-instrumentation-system-metrics/setup.cfg | 4 ++-- tox.ini | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg index e54d7338a0..1e4d52f4da 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg @@ -35,14 +35,14 @@ classifiers = Programming Language :: Python :: 3.10 [options] -python_requires = >=3.5 +python_requires = >=3.6 package_dir= =src packages=find_namespace: install_requires = opentelemetry-api ~= 1.10.0 opentelemetry-sdk ~= 1.10.0 - psutil ~= 5.7.0 + psutil >= 5.7.3 [options.extras_require] test = diff --git a/tox.ini b/tox.ini index 41d837b6da..2b45afe959 100644 --- a/tox.ini +++ b/tox.ini @@ -163,7 +163,7 @@ envlist = ; opentelemetry-instrumentation-system-metrics py3{6,7,8,9,10}-test-instrumentation-system-metrics - pypy3-test-instrumentation-system-metrics + ; instrumentation-system-metrics intentionally excluded from pypy3 ; opentelemetry-instrumentation-tornado py3{6,7,8,9,10}-test-instrumentation-tornado From 3943fc0186bbf87f453084eaa73477b584400f4e Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 6 Apr 2022 08:57:09 -0700 Subject: [PATCH 13/15] update deps --- docs-requirements.txt | 2 +- instrumentation/README.md | 2 +- .../opentelemetry-instrumentation-system-metrics/setup.cfg | 1 - .../src/opentelemetry/instrumentation/system_metrics/package.py | 2 +- .../src/opentelemetry/instrumentation/bootstrap_gen.py | 2 +- 5 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs-requirements.txt b/docs-requirements.txt index c7b3cb7df2..4efc0ca46e 100644 --- a/docs-requirements.txt +++ b/docs-requirements.txt @@ -12,7 +12,6 @@ sphinx-autodoc-typehints # Required by opentelemetry-instrumentation fastapi>=0.65.2 -psutil~=5.7.0 pymemcache~=1.3 # Required by conf @@ -29,6 +28,7 @@ flask~=2.0 falcon~=2.0 grpcio~=1.27 mysql-connector-python~=8.0 +psutil>=5 pymongo~=3.1 PyMySQL~=0.9.3 pyramid>=1.7 diff --git a/instrumentation/README.md b/instrumentation/README.md index a4e880d87a..9064a65a85 100644 --- a/instrumentation/README.md +++ b/instrumentation/README.md @@ -33,7 +33,7 @@ | [opentelemetry-instrumentation-sqlalchemy](./opentelemetry-instrumentation-sqlalchemy) | sqlalchemy | | [opentelemetry-instrumentation-sqlite3](./opentelemetry-instrumentation-sqlite3) | sqlite3 | | [opentelemetry-instrumentation-starlette](./opentelemetry-instrumentation-starlette) | starlette ~= 0.13.0 | -| [opentelemetry-instrumentation-system-metrics](./opentelemetry-instrumentation-system-metrics) | psutil >= 5.7.3 | +| [opentelemetry-instrumentation-system-metrics](./opentelemetry-instrumentation-system-metrics) | psutil >= 5 | | [opentelemetry-instrumentation-tornado](./opentelemetry-instrumentation-tornado) | tornado >= 5.1.1 | | [opentelemetry-instrumentation-urllib](./opentelemetry-instrumentation-urllib) | urllib | | [opentelemetry-instrumentation-urllib3](./opentelemetry-instrumentation-urllib3) | urllib3 >= 1.0.0, < 2.0.0 | diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg index 1e4d52f4da..84af307853 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg @@ -42,7 +42,6 @@ packages=find_namespace: install_requires = opentelemetry-api ~= 1.10.0 opentelemetry-sdk ~= 1.10.0 - psutil >= 5.7.3 [options.extras_require] test = diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py index 7eb5b17a99..570690e27f 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py @@ -13,4 +13,4 @@ # limitations under the License. -_instruments = ("psutil >= 5.7.3",) +_instruments = ("psutil >= 5",) diff --git a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py index ae8d86b08f..0a880b345f 100644 --- a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py +++ b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py @@ -129,7 +129,7 @@ "instrumentation": "opentelemetry-instrumentation-starlette==0.29b0", }, "psutil": { - "library": "psutil >= 5.7.3", + "library": "psutil >= 5", "instrumentation": "opentelemetry-instrumentation-system-metrics==0.29b0", }, "tornado": { From a1618702e102b9829f733b24a5b5056cb03b3413 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 6 Apr 2022 10:24:37 -0700 Subject: [PATCH 14/15] cleanup based on review --- .../system_metrics/__init__.py | 204 ++++++------------ 1 file changed, 63 insertions(+), 141 deletions(-) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py index 0c1958ad80..3317b3fc65 100644 --- a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -83,9 +83,27 @@ from opentelemetry.instrumentation.system_metrics.version import __version__ from opentelemetry.sdk.util import get_dict_as_key +_DEFAULT_CONFIG = { + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.cpu.utilization": ["idle", "user", "system", "irq"], + "system.memory.usage": ["used", "free", "cached"], + "system.memory.utilization": ["used", "free", "cached"], + "system.swap.usage": ["used", "free"], + "system.swap.utilization": ["used", "free"], + "system.disk.io": ["read", "write"], + "system.disk.operations": ["read", "write"], + "system.disk.time": ["read", "write"], + "system.network.dropped.packets": ["transmit", "receive"], + "system.network.packets": ["transmit", "receive"], + "system.network.errors": ["transmit", "receive"], + "system.network.io": ["trasmit", "receive"], + "system.network.connections": ["family", "type"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], +} + class SystemMetricsInstrumentor(BaseInstrumentor): - # pylint: disable=too-many-statements def __init__( self, labels: Optional[Dict[str, str]] = None, @@ -93,28 +111,7 @@ def __init__( ): super().__init__() if config is None: - self._config = { - "system.cpu.time": ["idle", "user", "system", "irq"], - "system.cpu.utilization": ["idle", "user", "system", "irq"], - "system.memory.usage": ["used", "free", "cached"], - "system.memory.utilization": ["used", "free", "cached"], - "system.swap.usage": ["used", "free"], - "system.swap.utilization": ["used", "free"], - # system.swap.page.faults: [], - # system.swap.page.operations: [], - "system.disk.io": ["read", "write"], - "system.disk.operations": ["read", "write"], - "system.disk.time": ["read", "write"], - # "system.filesystem.usage": [], - # "system.filesystem.utilization": [], - "system.network.dropped.packets": ["transmit", "receive"], - "system.network.packets": ["transmit", "receive"], - "system.network.errors": ["transmit", "receive"], - "system.network.io": ["trasmit", "receive"], - "system.network.connections": ["family", "type"], - "runtime.memory": ["rss", "vms"], - "runtime.cpu.time": ["user", "system"], - } + self._config = _DEFAULT_CONFIG else: self._config = config self._labels = {} if labels is None else labels @@ -131,17 +128,12 @@ def __init__( self._system_swap_usage_labels = self._labels.copy() self._system_swap_utilization_labels = self._labels.copy() - # self._system_swap_page_faults = self._labels.copy() - # self._system_swap_page_operations = self._labels.copy() self._system_disk_io_labels = self._labels.copy() self._system_disk_operations_labels = self._labels.copy() self._system_disk_time_labels = self._labels.copy() self._system_disk_merged_labels = self._labels.copy() - # self._system_filesystem_usage_labels = self._labels.copy() - # self._system_filesystem_utilization_labels = self._labels.copy() - self._system_network_dropped_packets_labels = self._labels.copy() self._system_network_packets_labels = self._labels.copy() self._system_network_errors_labels = self._labels.copy() @@ -205,21 +197,25 @@ def _instrument(self, **kwargs): unit="1", ) - # # self._meter.create_observable_counter( - # # callback=self._get_system_swap_page_faults, - # # name="system.swap.page_faults", - # # description="System swap page faults", - # # unit="faults", - # # value_type=int, - # # ) - - # # self._meter.create_observable_counter( - # # callback=self._get_system_swap_page_operations, - # # name="system.swap.page_operations", - # # description="System swap page operations", - # # unit="operations", - # # value_type=int, - # # ) + # TODO Add _get_system_swap_page_faults + + # self._meter.create_observable_counter( + # callback=self._get_system_swap_page_faults, + # name="system.swap.page_faults", + # description="System swap page faults", + # unit="faults", + # value_type=int, + # ) + + # TODO Add _get_system_swap_page_operations + # self._meter.create_observable_counter( + # callback=self._get_system_swap_page_operations, + # name="system.swap.page_operations", + # description="System swap page operations", + # unit="operations", + # value_type=int, + # ) + self._meter.create_observable_counter( callback=self._get_system_disk_io, name="system.disk.io", @@ -241,6 +237,8 @@ def _instrument(self, **kwargs): unit="seconds", ) + # TODO Add _get_system_filesystem_usage + # self.accumulator.register_valueobserver( # callback=self._get_system_filesystem_usage, # name="system.filesystem.usage", @@ -249,6 +247,7 @@ def _instrument(self, **kwargs): # value_type=int, # ) + # TODO Add _get_system_filesystem_utilization # self._meter.create_observable_gauge( # callback=self._get_system_filesystem_utilization, # name="system.filesystem.utilization", @@ -257,6 +256,9 @@ def _instrument(self, **kwargs): # value_type=float, # ) + # TODO Filesystem information can be obtained with os.statvfs in Unix-like + # OSs, how to do the same in Windows? + self._meter.create_observable_counter( callback=self._get_system_network_dropped_packets, name="system.network.dropped_packets", @@ -317,11 +319,7 @@ def _uninstrument(self, **__): pass def _get_system_cpu_time(self) -> Iterable[Measurement]: - """Observer callback for system CPU time - - Args: - observer: the observer to update - """ + """Observer callback for system CPU time""" for cpu, times in enumerate(psutil.cpu_times(percpu=True)): for metric in self._config["system.cpu.time"]: if hasattr(times, metric): @@ -332,11 +330,7 @@ def _get_system_cpu_time(self) -> Iterable[Measurement]: ) def _get_system_cpu_utilization(self) -> Iterable[Measurement]: - """Observer callback for system CPU utilization - - Args: - observer: the observer to update - """ + """Observer callback for system CPU utilization""" for cpu, times_percent in enumerate( psutil.cpu_times_percent(percpu=True) @@ -351,11 +345,7 @@ def _get_system_cpu_utilization(self) -> Iterable[Measurement]: ) def _get_system_memory_usage(self) -> Iterable[Measurement]: - """Observer callback for memory usage - - Args: - observer: the observer to update - """ + """Observer callback for memory usage""" virtual_memory = psutil.virtual_memory() for metric in self._config["system.memory.usage"]: self._system_memory_usage_labels["state"] = metric @@ -366,11 +356,7 @@ def _get_system_memory_usage(self) -> Iterable[Measurement]: ) def _get_system_memory_utilization(self) -> Iterable[Measurement]: - """Observer callback for memory utilization - - Args: - observer: the observer to update - """ + """Observer callback for memory utilization""" system_memory = psutil.virtual_memory() for metric in self._config["system.memory.utilization"]: @@ -382,11 +368,7 @@ def _get_system_memory_utilization(self) -> Iterable[Measurement]: ) def _get_system_swap_usage(self) -> Iterable[Measurement]: - """Observer callback for swap usage - - Args: - observer: the observer to update - """ + """Observer callback for swap usage""" system_swap = psutil.swap_memory() for metric in self._config["system.swap.usage"]: @@ -398,11 +380,7 @@ def _get_system_swap_usage(self) -> Iterable[Measurement]: ) def _get_system_swap_utilization(self) -> Iterable[Measurement]: - """Observer callback for swap utilization - - Args: - observer: the observer to update - """ + """Observer callback for swap utilization""" system_swap = psutil.swap_memory() for metric in self._config["system.swap.utilization"]: @@ -413,15 +391,8 @@ def _get_system_swap_utilization(self) -> Iterable[Measurement]: self._system_swap_utilization_labels, ) - # TODO Add _get_system_swap_page_faults - # TODO Add _get_system_swap_page_operations - def _get_system_disk_io(self) -> Iterable[Measurement]: - """Observer callback for disk IO - - Args: - observer: the observer to update - """ + """Observer callback for disk IO""" for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.io"]: if hasattr(counters, f"{metric}_bytes"): @@ -433,11 +404,7 @@ def _get_system_disk_io(self) -> Iterable[Measurement]: ) def _get_system_disk_operations(self) -> Iterable[Measurement]: - """Observer callback for disk operations - - Args: - observer: the observer to update - """ + """Observer callback for disk operations""" for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.operations"]: if hasattr(counters, f"{metric}_count"): @@ -449,11 +416,7 @@ def _get_system_disk_operations(self) -> Iterable[Measurement]: ) def _get_system_disk_time(self) -> Iterable[Measurement]: - """Observer callback for disk time - - Args: - observer: the observer to update - """ + """Observer callback for disk time""" for device, counters in psutil.disk_io_counters(perdisk=True).items(): for metric in self._config["system.disk.time"]: if hasattr(counters, f"{metric}_time"): @@ -465,11 +428,7 @@ def _get_system_disk_time(self) -> Iterable[Measurement]: ) def _get_system_disk_merged(self) -> Iterable[Measurement]: - """Observer callback for disk merged operations - - Args: - observer: the observer to update - """ + """Observer callback for disk merged operations""" # FIXME The units in the spec is 1, it seems like it should be # operations or the value type should be Double @@ -484,17 +443,8 @@ def _get_system_disk_merged(self) -> Iterable[Measurement]: self._system_disk_merged_labels, ) - # TODO Add _get_system_filesystem_usage - # TODO Add _get_system_filesystem_utilization - # TODO Filesystem information can be obtained with os.statvfs in Unix-like - # OSs, how to do the same in Windows? - def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: - """Observer callback for network dropped packets - - Args: - observer: the observer to update - """ + """Observer callback for network dropped packets""" for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: @@ -512,11 +462,7 @@ def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: ) def _get_system_network_packets(self) -> Iterable[Measurement]: - """Observer callback for network packets - - Args: - observer: the observer to update - """ + """Observer callback for network packets""" for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: @@ -530,11 +476,7 @@ def _get_system_network_packets(self) -> Iterable[Measurement]: ) def _get_system_network_errors(self) -> Iterable[Measurement]: - """Observer callback for network errors - - Args: - observer: the observer to update - """ + """Observer callback for network errors""" for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.errors"]: in_out = {"receive": "in", "transmit": "out"}[metric] @@ -547,11 +489,7 @@ def _get_system_network_errors(self) -> Iterable[Measurement]: ) def _get_system_network_io(self) -> Iterable[Measurement]: - """Observer callback for network IO - - Args: - observer: the observer to update - """ + """Observer callback for network IO""" for device, counters in psutil.net_io_counters(pernic=True).items(): for metric in self._config["system.network.dropped.packets"]: @@ -565,11 +503,7 @@ def _get_system_network_io(self) -> Iterable[Measurement]: ) def _get_system_network_connections(self) -> Iterable[Measurement]: - """Observer callback for network connections - - Args: - observer: the observer to update - """ + """Observer callback for network connections""" # TODO How to find the device identifier for a particular # connection? @@ -607,11 +541,7 @@ def _get_system_network_connections(self) -> Iterable[Measurement]: ) def _get_runtime_memory(self) -> Iterable[Measurement]: - """Observer callback for runtime memory - - Args: - observer: the observer to update - """ + """Observer callback for runtime memory""" proc_memory = self._proc.memory_info() for metric in self._config["runtime.memory"]: if hasattr(proc_memory, metric): @@ -622,11 +552,7 @@ def _get_runtime_memory(self) -> Iterable[Measurement]: ) def _get_runtime_cpu_time(self) -> Iterable[Measurement]: - """Observer callback for runtime CPU time - - Args: - observer: the observer to update - """ + """Observer callback for runtime CPU time""" proc_cpu = self._proc.cpu_times() for metric in self._config["runtime.cpu.time"]: if hasattr(proc_cpu, metric): @@ -637,11 +563,7 @@ def _get_runtime_cpu_time(self) -> Iterable[Measurement]: ) def _get_runtime_gc_count(self) -> Iterable[Measurement]: - """Observer callback for garbage collection - - Args: - observer: the observer to update - """ + """Observer callback for garbage collection""" for index, count in enumerate(gc.get_count()): self._runtime_gc_count_labels["count"] = str(index) yield Measurement(count, self._runtime_gc_count_labels) From dd2803de5501b84a07b34238c80379bb2b8b850d Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 6 Apr 2022 10:26:44 -0700 Subject: [PATCH 15/15] Update CHANGELOG.md --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cb863f9a2b..d7597561a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,8 +13,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) - `opentelemetry-instrumentation-tornado` Fix non-recording span bug ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) -- `opentelemetry-instrumentation-pyramid` Handle non-HTTPException exceptions - ([#1001](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1001)) ### Added @@ -28,6 +26,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#1003])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1003) - `opentelemetry-instrumentation-elasticsearch` no longer creates unique span names by including search target, replaces them with `` and puts the value in attribute `elasticsearch.target` ([#1018](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1018)) +- `opentelemetry-instrumentation-pyramid` Handle non-HTTPException exceptions + ([#1001](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1001)) - `opentelemetry-instrumentation-system-metrics` restore `SystemMetrics` instrumentation as `SystemMetricsInstrumentor` ([#1012](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1012))