[meta] [fleet] Java agent auto attaching

[jalvz]

The goal is to enable users to instrument their Java applications with zero configuration.

This requires a few steps:

• Bundle the Java agent attacher in apm-server ([fleet] Bundle the java agent attacher #4824)
• Add settings in the fleet policy editor for the attached
  • https://github.com/elastic/observability-design/issues/77
  • [APM] Fleet: Auto attachment for java agent beta kibana#108513
• Support java attacher settings via apm-server.yml ([fleet] Support java attacher settings via apm-server.yml  #4829)
• Support --download-agent-version option [fleet] Add download agent version option to Java agent attacher #5853
• Download the appropriate version of the Java agent. APM Server asks the Elastic Agent to download the Java Agent binaries in a specific version (the version from the apm-server.yml). Elastic Agent downloads the binaries and returns the path to the binaries to the APM Server. ([fleet] Trigger Java Agent binary download via Elastic Agent #5477)
• APM Server runs the improved Java agent attacher (prefer $JAVA_HOME/bin/java over java), which takes care of attaching the binary to all matching JVMs on the same host ([fleet] Run the Java agent attacher to attach binaries to all matching JVMs #5478)
• Process attacher logs (as part of [fleet] Run the Java agent attacher to attach binaries to all matching JVMs #5478)
• limit Java agent auto attachment to not be available when running in ECE/ESS environments ([fleet] Only support java agent attachment when not running in cloud env #5479)

[felixbarny]

Some additional steps:

• Handling of version updates
  • Download new version
  • Stop attacher
  • Start attacher with path to updated --agent-jar
  • Delete old agent jar?
• Handling of config changes (ex include/exclude rules)
  • Stop attacher
  • Restart with new config

[simitt]

@felixbarny could you help us define more details for this so we can start working on it?

[jalvz]

@simitt we should add this to #5388, right?

[simitt]

good catch @jalvz , added it.

[jalvz]

@felixbarny how should we handle java logs in the end?
will apm-server just log them as if it were their own? should we handle them separately and have extra settings for it (including rotation / etc)?

[simitt]

@felixbarny related to #4830 (comment) - does that mean that the java agent attacher should not be packaged with the apm-server but that the apm-server needs to check for the latest attacher version and then download that?

[felixbarny]

how should we handle java logs in the end?

My first idea was that APM Server would just forward the standard out from the attacher to it's own log file. But I'm not sure if that's feasible with the logging framework that APM Server uses. It might be tricky to use the same file lock as the underlying log framework to avoid server and attacher logs to be interleaved.

In that case, it's probably best to have a separate log file for the attacher. See the --log-file and --log-level options of the attacher on how to do that.

Probably not the best alternative but another option would be to write the std. out of the attacher into it's own log file and APM Server would handle the log file rotation itself.

does that mean that the java agent attacher should not be packaged with the apm-server

No. The attacher version should always be a fixed one for an APM Server version.

There are two components that are involved. The attacher (apm-agent-attach-cli-slim.jar), and the Java agent binaries (elastic-apm-agent.jar).

The APM Server should bundle the attacher. The Java agent binaries are resolved using a different mechanism. It's not 100% clear what that mechanism is. Maybe we'll build those capabilities into the attacher for now and switch to Fleet packages later.

[stuartnelson3]

Looking at these two issues:

• [fleet] Add download agent version option to Java agent attacher #5853
• [fleet] Trigger Java Agent binary download via Elastic Agent #5477

Do we want both of these? It seems like we would be fine doing one of the other -- either apm-server requests the binary download from agent, or the attacher handles it on its own.

Since this behavior is already available in the attacher, it seems like that might be the simpler way to go?

[axw]

I see Felix closed the Elastic Agent method issue. Going with #5853 SGTM too. We may want to revise this later to support airgapped environments, but we don't need to worry about that just now.

[felixbarny]

The Fleet/Agent Team wasn’t keen on exposing an API to APM Server to explicitly download a binary. They want Agent to handle dependency management. The long term plan is to have subpackages but that’ll take a while. For the experimental version works via maven central.

…

On Tue 24. Aug 2021 at 04:22 Andrew Wilkins ***@***.***> wrote: I see Felix closed the Elastic Agent method issue. Going with #5853 <#5853> SGTM too. We may want to revise this later to support airgapped environments, but we don't need to worry about that just now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4830 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQQGCDNOJGJYJRZTKOVHW3T6L677ANCNFSM4X37YRTA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[simitt]

@stuartnelson3 it looks like all the checkboxes can be ticked, can this be closed?

[stuartnelson3]

yep!