TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon · 2024-12-06T14:46:46Z

Which issue(s) this PR addresses:

Closes #

openshift-ci · 2024-12-06T14:46:50Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-12-06T14:47:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [copejon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ggiguash · 2024-12-09T07:17:36Z

packaging/rpm/microshift.spec

+Requires: opentelemetry-collector
+
+%description observability
+Demo otel-col config geared for microshift. Client certificates are generated by microshift on start, then placed in


Suggested change

Demo otel-col config geared for microshift. Client certificates are generated by microshift on start, then placed in

Demo OpenTelemetry configuration geared for MicroShift. MicroShift OpenTelemetry client certificates are copied to /etc/pki/microshift-observability to facilitate OpenTelemetry collector service connection to MicroShift.

ggiguash · 2024-12-09T07:18:27Z

packaging/observability/opentelemetry-collector.yaml

+    ca_file: /etc/pki/microshift-opentelemetry-collector-client/client-ca.crt
+    key_file: /etc/pki/microshift-opentelemetry-collector-client/client.key
+    cert_file: /etc/pki/microshift-opentelemetry-collector-client/client.crt


Suggested change

ca_file: /etc/pki/microshift-opentelemetry-collector-client/client-ca.crt

key_file: /etc/pki/microshift-opentelemetry-collector-client/client.key

cert_file: /etc/pki/microshift-opentelemetry-collector-client/client.crt

ca_file: /etc/pki/microshift-observability/client-ca.crt

key_file: /etc/pki/microshift-observability/client.key

cert_file: /etc/pki/microshift-observability/client.crt

ggiguash · 2024-12-09T07:19:03Z

packaging/observability/opentelemetry-collector.yaml

+#  filelog/workload:
+#    Path: /….
+#  filelog/kube-system:
+#    Path: /….


If we'd like to keep comments in the file, can we add an explanation why?

ggiguash · 2024-12-09T07:19:39Z

packaging/observability/opentelemetry-collector.yaml

+  journald:
+    units:
+    - microshift
+    - crio


Do we need to monitor microshift-etcd unit too?

ggiguash · 2024-12-09T07:20:20Z

packaging/rpm/microshift.spec

@@ -482,6 +492,10 @@ cat assets/optional/gateway-api/kustomization.x86_64.yaml >> %{buildroot}/%{_pre
 mkdir -p -m755 %{buildroot}%{_datadir}/microshift/release
 install -p -m644 assets/optional/gateway-api/release-gateway-api-{x86_64,aarch64}.json %{buildroot}%{_datadir}/microshift/release/

+#observability


Suggested change

#observability

# observability

ggiguash · 2024-12-09T07:21:24Z

packaging/rpm/microshift.spec

@@ -645,6 +662,9 @@ fi
 %files gateway-api-release-info
 %{_datadir}/microshift/release/release-gateway-api-{x86_64,aarch64}.json

+%files observability
+%config %{_sysconfdir}/microshift/opentelemetry-collector.yaml
+%config %{_unitdir}/microshift-observability.service


How can we undo the certificate file copy on the package removal?

ggiguash · 2024-12-09T07:27:08Z

packaging/observability/microshift-observability.service

+
+# It takes a bit for the certs to be created. This service will reach it's burst limit almost immediately, pretty much
+# guaranteeing that it will reach the restart limit before it can possibly succeed.
+RestartSec=200ms


We need to optimize this to avoid service failures / restarts. See the unit section comment.

ggiguash · 2024-12-09T07:36:56Z

packaging/observability/microshift-observability.service

+AssertPathExists=/var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key
+AssertPathExists=/var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.crt


I'm not sure why this is necessary. We declare the current unit After MicroShift, so it should only start when MicroShift declares it's running, right? This means, certificates are created for sure.
What am I missing?

ggiguash · 2024-12-09T07:37:27Z

packaging/observability/microshift-observability.service

@@ -0,0 +1,27 @@
+[Unit]
+Description=MicroShift Observability
+BindsTo=microshift.service


Do we want to run the collector even when MicroShift fails?

ggiguash · 2024-12-09T07:41:49Z

packaging/observability/microshift-observability.service

+ExecStartPre=+/bin/mkdir -p /etc/pki/microshift-observability/
+ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key /etc/pki/microshift-observability/
+ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key /etc/pki/microshift-observability/
+ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.crt /etc/pki/microshift-observability/
+ExecStartPre=+/bin/chown -R observability /etc/pki/microshift-observability
+ExecStartPre=+/bin/chmod -R 600 /etc/pki/microshift-observability/
+ExecStartPre=+/bin/chmod 755 /etc/pki/microshift-observability


Suggested change

ExecStartPre=+/bin/mkdir -p /etc/pki/microshift-observability/

ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key /etc/pki/microshift-observability/

ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key /etc/pki/microshift-observability/

ExecStartPre=+/bin/cp /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.crt /etc/pki/microshift-observability/

ExecStartPre=+/bin/chown -R observability /etc/pki/microshift-observability

ExecStartPre=+/bin/chmod -R 600 /etc/pki/microshift-observability/

ExecStartPre=+/bin/chmod 755 /etc/pki/microshift-observability

ExecStartPre=+/bin/mkdir -p -m 755 /etc/pki/microshift-observability/

ExecStartPre=+/bin/install -o observability -m 600 /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key /etc/pki/microshift-observability/client.key

ExecStartPre=+/bin/install -o observability -m 600 /var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.crt /etc/pki/microshift-observability/client.crt

ggiguash · 2024-12-09T07:49:41Z

/retitle NO-ISSUE: OpenTelemetry certificates and service for MicroShift

openshift-ci-robot · 2024-12-09T07:49:47Z

@copejon: This pull request explicitly references no jira issue.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-12-10T09:12:01Z

@copejon: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

pavolloffay · 2024-12-12T14:01:16Z

packaging/observability/opentelemetry-collector.yaml

+      # specifies how frequently the conditions for compaction are being checked
+      check_interval: 5s # Default
+
+service:


Docs for internal collector metrics https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-metrics

it would be good to send them as well so users can monitor the collector.

openshift-ci-robot · 2024-12-12T20:47:24Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

copejon · 2024-12-12T20:47:28Z

/jira refresh

openshift-ci-robot · 2024-12-12T20:47:31Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

implemented opentelemetry-collector in packaging pipeline. the otel binary is unable to see the cert files, but the config paths are correct and the files exist. file permissions for the observability user have been checked, but are good. WIP Signed-off-by: Jon Cope <[email protected]>

ensure the otelcol process creates necessary dirs added firewall port handling for otel-col exporter Signed-off-by: Jon Cope <[email protected]>

… on the localhost Signed-off-by: Jon Cope <[email protected]>

Signed-off-by: Jon Cope <[email protected]>

… to allow the collection of pod logs from the host filesystem Signed-off-by: Jon Cope <[email protected]>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2024

ggiguash reviewed Dec 9, 2024

View reviewed changes

openshift-ci bot changed the title ~~No issue generate otel cert~~ NO-ISSUE: OpenTelemetry certificates and service for MicroShift Dec 9, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 9, 2024

pavolloffay reviewed Dec 12, 2024

View reviewed changes

copejon changed the title ~~NO-ISSUE: OpenTelemetry certificates and service for MicroShift~~ TRACING-4752: Add OpenTelemetry-Collector as optional sub-package Dec 12, 2024

copejon added 5 commits December 12, 2024 14:48

flakey, but less so

ef409e7

ensure the otelcol process creates necessary dirs added firewall port handling for otel-col exporter Signed-off-by: Jon Cope <[email protected]>

removed hardcoded IP, firewall-cmds. only relevant to running backend…

59cf929

… on the localhost Signed-off-by: Jon Cope <[email protected]>

shorten observability handle throughout

fa5d98d

Signed-off-by: Jon Cope <[email protected]>

remove hack to support user "observability" in favor of root. this is…

fede276

… to allow the collection of pod logs from the host filesystem Signed-off-by: Jon Cope <[email protected]>

copejon force-pushed the no-issue-generate-otel-cert branch from fa4f579 to fede276 Compare December 12, 2024 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash Dec 9, 2024

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci bot commented Dec 10, 2024

pavolloffay Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

	Demo otel-col config geared for microshift. Client certificates are generated by microshift on start, then placed in
	Demo OpenTelemetry configuration geared for MicroShift. MicroShift OpenTelemetry client certificates are copied to /etc/pki/microshift-observability to facilitate OpenTelemetry collector service connection to MicroShift.

		AssertPathExists=/var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.key
		AssertPathExists=/var/lib/microshift/certs/kube-apiserver-localhost-signer/observability/client.crt

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Are you sure you want to change the base?

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Conversation

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci bot commented Dec 10, 2024

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading