Send pipeline logs to OpenTelemetry endpoint #304

cyrille-leclerc · 2022-01-25T22:47:04Z

Support sending the Pipeline logs to an OpenTelemetry endpoint.
- When pipeline logs are sent to Elastic through the OpenTelemetry instrumentation, support visualization of logs through Jenkins pipeline logs screens in addition to supporting visualization of logs through Elastic / Kibana
Add instrumentation of Jenkins Stapler

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
- Collect pipelines and build logs #245
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

# Conflicts: # pom.xml

* Move Elasticsearch code under backend package * Move Elasticsearch code under backend package

…nt (#315) * Implement logs retriever pagination using new Elasticsearch Java client * WIP * WIP * extract integration tests configuration from code * WIP * WIP * WIP * WIP * fix findbugs * better docs * WIP

…/ Stapler framework

…stead of the new elasticsearch-java waiting for the fix elastic/elasticsearch-java#163

jglick

Any general feedback on how well JEP-210 worked for your use case?

jglick · 2022-07-18T19:07:16Z

src/main/java/io/jenkins/plugins/opentelemetry/servlet/OpenTelemetryServletFilter.java

+            if (rootPath.equals("job")) {
+                // e.g /job/my-war/job/master/lastBuild/console
+                // e.g /job/my-war/job/master/2/console
+                ParsedJobUrl parsedJobUrl = parseJobUrl(pathInfoTokens);


This all seems like a maintenance burden. Did you consider using a supported API like https://javadoc.jenkins.io/component/stapler/org/kohsuke/stapler/StaplerRequest.html#getAncestors()?

Thanks I got trapped. I started adding HTTP request tracing to understand how the progressive rendering of logs worked in Jenkins and ended up this insane if then else.

I'll add a comment in the code to use this Stapler API.

adding HTTP request tracing to understand how the progressive rendering of logs worked in Jenkins

BTW -Dstapler.trace, automatic with mvn hpi:run.

jglick · 2022-07-18T19:10:58Z

.../java/io/jenkins/plugins/opentelemetry/backend/elastic/ElasticsearchLogStorageRetriever.java

+
+/**
+ * Use the old `org.elasticsearch.client:elasticsearch-rest-high-level-client` waiting for
+ * `co.elastic.clients:elasticsearch-java` to fix https://github.com/elastic/elasticsearch-java/issues/163


elastic/elasticsearch-java#163 supposedly shipped in https://github.com/elastic/elasticsearch-java/releases/tag/v8.2.0?

Code has been refactored and cleaned up since then.

jglick · 2022-07-18T19:12:41Z

...n/java/io/jenkins/plugins/opentelemetry/backend/elastic/ElasticsearchLogStorageRetriever.txt

+ * Retrieve the logs from Elasticsearch.
+ * FIXME graceful shutdown
+ */
+public class ElasticsearchLogStorageRetriever implements LogStorageRetriever<ElasticsearchLogsQueryContext>, Closeable {


Deliberately committed as a *.txt?

I was struggling understanding the Jenkins progressive logs rendering APIs and kept temporarily some code as .txt, it has been removed since then.

jglick · 2022-07-18T19:13:00Z

...main/java/io/jenkins/plugins/opentelemetry/backend/elastic/ElasticsearchLogsQueryContext.txt

+
+import java.util.Objects;
+
+public class ElasticsearchLogsQueryContext implements LogsQueryContext {


ditto, removed since then

jglick · 2022-07-18T19:15:14Z

src/main/java/io/jenkins/plugins/opentelemetry/job/log/ConsoleNotes.java

+
+/**
+ * Utilities for extracting and reinserting {@link ConsoleNote}s.
+ * copied from https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin


…and then modified from there. Candidate for a convenience API in workflow-api?

That could be an idea. The helper in the Jenkins Otel Plugin changed since then but it can probably be useful for other. implementations of JEP 210

jglick · 2022-07-18T19:17:20Z

src/main/java/io/jenkins/plugins/opentelemetry/job/log/OtelLogOutputStream.java

+        ConsoleNotes.TextAndAnnotations textAndAnnotations = ConsoleNotes.parse(bytes, len);
+        String plainLogLine = textAndAnnotations.text;
+        if (plainLogLine == null || plainLogLine.isEmpty()) {
+            LOGGER.log(Level.FINER, () -> buildInfo + " - skip empty log line");


Could be confusing if you actually had some tool run as part of a sh step which deliberately emitted blank lines as a visual separator.

Valid point, I should have added a comment to explain why I ended removing these line breaks, it was probably caused by challenges I faced understanding Jenkins progressive logs rendering APIs

jglick · 2022-07-18T19:23:33Z

src/main/java/io/jenkins/plugins/opentelemetry/job/log/OtelLogSenderBuildListener.java

+                this.clock = Clock.getDefault();
+            } else {
+                long instantInNanosOnJenkinsAgentAtDeserialization = Clock.getDefault().now();
+                long offsetInNanosOnJenkinsAgent = instantInNanosOnJenkinsControllerBeforeSerialization - instantInNanosOnJenkinsAgentAtDeserialization;


Does this assume that the serialization, communication over Remoting, and deserialization are nearly instantaneous?

Not instantaneous, I wanted monotonicity of the timestamps even when the clocks of the controller and of the build agents are different.
Don't you do something similar with the CloudWatch storage?

See https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-2.9.0-rc1/src/main/java/io/jenkins/plugins/opentelemetry/job/log/OtelLogSenderBuildListener.java#L50

I wanted monotonicity of the timestamps

My point: let us say the two hardware clocks are actually perfectly synchronized (up to relativistic constraints, whatever). But the Remoting channel is extremely sluggish today for whatever reason, and it takes ten seconds to send a Command. Then offsetInNanosOnJenkinsAgent will be 10_000_000_000 or so. But in fact the correct behavior in this case would be to use zero offset, if you are trying to ensure that messages delivered to the external storage are accurately threaded.

In other words, if you want to calculate clock skew, you need to send a round-trip packet and adjust by half the time required to do so, or something a bit more complex like that. I think https://github.com/jenkinsci/jenkins/blob/38d4e86a80d123d50c966f4584a7e795b6bcb227/core/src/main/java/hudson/model/Slave.java#L732 does this.

Don't you do something similar with the CloudWatch storage?

No, io.jenkins.plugins.pipeline_cloudwatch_logs.TimestampTracker solves different problems:

ensuring that we do not mark the build log as complete until all messages known to have been sent from agents have actually arrived in the cloud and are available for reading (otherwise you would often lose the last few lines of a build, which tend to be the most critical if there was a failure)

making sure output from multiple steps running on a given agent is logically ordered (no attempt is made to order messages between the controller and an agent, or between two agents, in the face of clock skew)

jglick · 2022-07-18T19:29:35Z

src/test/java/io/jenkins/plugins/opentelemetry/job/log/OpenTelemetryLogToElasticsearchIT.txt

+import java.util.Properties;
+import java.util.Random;
+
+public class OpenTelemetryLogToElasticsearchIT {


*.txt rather than just @Ignore like the rest of the ITs?

I have to cleanup all the test container based ITs.

jglick · 2022-07-18T19:31:55Z

.../java/io/jenkins/plugins/opentelemetry/backend/elastic/PipelineElasticsearchBackendTest.java

+
+import static org.junit.Assume.assumeTrue;
+
+@Ignore


Would this be the only IT of the JEP-210 implementation, if enabled?

yes, all tests are manual for the moment. It's not great but I didn't have the bandwidth to fix the integrated tests.

jglick · 2022-07-18T19:36:59Z

src/main/java/io/jenkins/plugins/opentelemetry/job/log/OtelLogSenderBuildListener.java

+        private Object writeReplace() throws IOException {
+            logger.log(Level.FINEST, () -> "writeReplace()");
+            JenkinsJVM.checkJenkinsJVM();
+            return new OtelLogSenderBuildListenerOnAgent(buildInfo, otelConfigProperties, otelResourceAttributes);


Is there any need to pass minimal credentials to the agent replacement as the CloudWatch Logs implementation does, or does the telemetry endpoint somewhat automatically verify the origin of a message so that a malicious build would not be able to spoof output from unrelated jobs?

otelConfigProperties contain the credentials of the OTLP endpoint. Does it answer your question?

It answers my question by affirming that this implementation is indeed insecure for some configurations. https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin#security

cyrille-leclerc · 2022-07-18T21:11:50Z

THANK YOU very much for your code review @jglick .

Any general feedback on how well JEP-210 worked for your use case?

JEP-210 was quite hard to implement but to our knowledge, it works well.

The most difficult things were to understand:

The role of the org.kohsuke.stapler.framework.io.ByteBuffer in the hudson.console.AnnotatedLargeText produced by org.jenkinsci.plugins.workflow.log.LogStorage#overallLog . I struggled to not create an OOM
The way the progressive rendering of logs worked in Jenkins

jglick · 2022-07-19T20:06:30Z

AnnotatedLargeText is indeed a low-level API, and it is not well suited to the needs of JEP-210; workflow-api and/or pipeline-cloudwatch-logs wound up working around some of its limitations. I cannot recall details at this point but it would be better to make some changes in Stapler to better support non-local storage implementations.

Rendering of logs for running builds is not complete in JEP-210. Currently Stapler (and by extension Jenkins core, and also Blue Ocean) use long as a byte offset, which can only be efficiently implemented in a file-based implementation. Thus CloudWatchRetriever has to stream the complete text to a buffer, before potentially throwing away 99% of it (whatever long start says); since rendering is stateless, each HTTP request has to repeat the process, making for O(n²) work rendering a running build (unless you cache the log in a temporary file or something). A proper API would allow the implementation to use a String offset, which for CWL would be a FilterLogEventsResult.nextToken.

cyrille-leclerc added 9 commits January 25, 2022 11:00

Send Pipeline Logs to OTel

7e3af1f

WIP

fdf81db

Disable OtelLogStorage if Otel SDK is not configured to export logs

cc45c2c

Add trace context

f319d24

WIP

ddb2fec

WIP

1224500

Simplify code

8063fdb

Simplify

fd2d0cf

Fix dependency injection problems

c4c0ea9

kuisathaverat force-pushed the send-build-logs-to-otel branch from 5da6599 to c4c0ea9 Compare January 27, 2022 12:27

cyrille-leclerc mentioned this pull request Jan 31, 2022

POC [Maven Extension] Capture Maven build logs using Otel Logs open-telemetry/opentelemetry-java-contrib#221

Closed

kuisathaverat and others added 10 commits February 6, 2022 11:37

Add Elasticsearch log retriever

f398b1d

Merge branch 'master' into send-build-logs-to-otel

4a35569

# Conflicts: # pom.xml

Extract more elasticsearch code (#310)

6978007

* Move Elasticsearch code under backend package * Move Elasticsearch code under backend package

Extract logs retriever API (#311)

ed1594d

Implement logs retriever pagination using new Elasticsearch Java clie…

6af398d

…nt (#315) * Implement logs retriever pagination using new Elasticsearch Java client * WIP * WIP * extract integration tests configuration from code * WIP * WIP * WIP * WIP * fix findbugs * better docs * WIP

Skip empty log messages

a2e058a

Fix log rendering

a517bf2

WIP

db37a26

WIP

4b1e4db

WIP

933f4d7

kuisathaverat approved these changes Feb 17, 2022

View reviewed changes

cyrille-leclerc added 8 commits February 17, 2022 16:18

Code cleanup

8260804

Cleanup

fbb798a

Prefix root span name of build executions by "BUILD"

530967c

Clock adjustment between the Jenkins Controller and the Jenkins Agents

2d62535

WIP

52d63fd

Fix unit tests

1919856

WIP

4cbe00f

WUP

9231afa

cyrille-leclerc added 3 commits February 19, 2022 15:54

WIP

647d84a

Support disabling the instrumentation with traces of the Jenkins Web …

15c286d

…/ Stapler framework

Cleaner disabling of unit tests

8227106

cyrille-leclerc marked this pull request as ready for review February 19, 2022 16:22

cyrille-leclerc added 3 commits February 19, 2022 18:49

WIP

db5f908

WIP

988f40c

fix "NoClassDefFoundError: org/parboiled/common/StringUtils"

56403dc

cyrille-leclerc mentioned this pull request Feb 19, 2022

ElasticsearchLogsRetriever: ClassCastException: class org.glassfish.json.JsonProviderImpl cannot be cast to class jakarta.json.spi.JsonProvider #322

Closed

cyrille-leclerc added 3 commits February 20, 2022 14:33

Cleanup

16af0ed

fix "NoClassDefFoundError: org/parboiled/common/StringUtils"

73cba42

Switch back to the deprecated elasticsearch-rest-high-level-client in…

6b32491

…stead of the new elasticsearch-java waiting for the fix elastic/elasticsearch-java#163

cyrille-leclerc merged commit 0397b31 into master Feb 20, 2022

cyrille-leclerc deleted the send-build-logs-to-otel branch February 20, 2022 15:26

jglick mentioned this pull request Jul 18, 2022

Missing shell output in Kubernetes based environment #482

Closed

jglick reviewed Jul 18, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send pipeline logs to OpenTelemetry endpoint #304

Send pipeline logs to OpenTelemetry endpoint #304

cyrille-leclerc commented Jan 25, 2022 •

edited

Loading

jglick left a comment

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 19, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 19, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 18, 2022

cyrille-leclerc Jul 18, 2022

jglick Jul 19, 2022

cyrille-leclerc commented Jul 18, 2022

jglick commented Jul 19, 2022


		import java.util.Objects;

		public class ElasticsearchLogsQueryContext implements LogsQueryContext {

Send pipeline logs to OpenTelemetry endpoint #304

Send pipeline logs to OpenTelemetry endpoint #304

Conversation

cyrille-leclerc commented Jan 25, 2022 • edited Loading

jglick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyrille-leclerc commented Jul 18, 2022

jglick commented Jul 19, 2022

cyrille-leclerc commented Jan 25, 2022 •

edited

Loading