Attacher improvements

[felixbarny]

What does this PR do?

From changelog.asciidoc:

• Overhaul of the <<setup-attach-cli,attacher cli>> application that allows to attach the agent to running JVMS
  • The artifact of the standalone cli applicaiton is now called apm-agent-attach-cli. The attacher API is still called apm-agent-attach.
  • There is also a slim version of the cli application that does not bundle the Java agent.
    It requires the --agent-jar option to be set.
  • Improved logging
    The application uses {ecs-logging-java-ref}/intro.html[Java ECS logging] to emit JSON logs.
    The log level can be configured with the --log-level option.
    By default, the program is logging to the console but using the --log-file option, it can also log to a file.
  • Attach to JVMs running under a different user (unix only)
    The JVM requires the attacher to be running under the same user as the target VM (the attachee).
    The apm-agent-attach-standalone.jar can now be run with a user that has permissions to switch to the user that runs the target VM.
    On Windows, the attacher can still only attach to JVMs that are running with under the same user.
  • New include/exclude discovery rules
    • --include-all: Attach to all discovered JVMs. If no matchers are provided, it will not attach to any JVMs.
    • --include-user/--exclude-user: Attach to all JVMs of a given operating system user.
    • --include-main/--exclude-main: Attach to all JVMs that whose main class/jar name, or system properties match the provided regex.
    • --include-vmargs/--exclude-vmargs: Attach to all JVMs that whose main class/jar name, or system properties match the provided regex.
  • Removal of options
    • The deprecated --arg option has been removed.
    • The -i/--include, -e/exclude options have been removed in favor of the --<include|exclude>-<main|vmargs> options.
    • The -p/--pid options have been removed in favor of the --include-pid option.
  • Changed behavior of the -l/--list option
    The option now only lists JVMs that match the include/exclude discovery rules.
    Thus, it can be used to do a dry-run of the matchers without actually performing an attachment.
    It even works in combination with --continuous now.
    By default, the VM arguments are not printed, but only when the -a/--list-vmargs option is set.
  • Remove dependency on jps
    Even when matching on the main class name or on system properties,
  • Checks the Java version before attaching to avoid attachment on unsupported JVMs.

Checklist

• This is an enhancement of existing features, or a new feature in existing plugins
  • I have updated CHANGELOG.asciidoc
  • I have added tests that prove my fix is effective or that my feature works
  • Added an API method or config option? Document in which version this will be introduced
  • I have made corresponding changes to the documentation
• This is a bugfix
  • I have updated CHANGELOG.asciidoc
  • I have added tests that would fail without this fix
• This is a new plugin
  • I have updated CHANGELOG.asciidoc
  • My code follows the style guidelines of this project
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated supported-technologies.asciidoc
  • Added an API method or config option? Document in which version this will be introduced
  • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
• This is something else
  • I have updated CHANGELOG.asciidoc

TODOs

• The apm-agent-attach.jar (attacher API) now has a compile dependency to log4j2 and log4j2-ecs-layout. Maybe we'll have to create different modules for the standalone cli attacher and the attacher API.
• allow to log to a file
• Allow specifying the path to the agent jar. Consider publishing an attacher that doesn't bundle the agent jar.
• Rethink include/exclude rule logic. Currently, all conditions are applied with an OR. Example: --include-cmd foo --exclude-cmd bar would match foo-bar.jar. Should a given JVM pass the filter if it matches any include rule AND not matches any exclude rule?
  For Elastic Agent, they have recently added a similar rule logic. The following only allows APM to run:

constraints:
- rule: allow
  input: apm
- rule: deny
  input: "*"

Related issues

Closes #1688

[codecov-io]

Codecov Report

Merging #1667 (dfff7d9) into master (ffbaaa0) will increase coverage by 3.95%.
The diff coverage is 39.05%.

@@             Coverage Diff              @@
##             master    #1667      +/-   ##
============================================
+ Coverage     58.77%   62.73%   +3.95%     
- Complexity       92     3531    +3439     
============================================
  Files           403      364      -39     
  Lines         18475    17457    -1018     
  Branches       2570     2395     -175     
============================================
+ Hits          10858    10951      +93     
+ Misses         6847     5731    -1116     
- Partials        770      775       +5     

Impacted Files	Coverage Δ	Complexity Δ
.../main/java/co/elastic/apm/attach/UserRegistry.java	20.51% <20.51%> (ø)	6.00 <6.00> (?)
...main/java/co/elastic/apm/attach/AgentAttacher.java	28.28% <28.28%> (ø)	0.00 <0.00> (?)
...java/co/elastic/apm/attach/GetAgentProperties.java	36.66% <36.66%> (ø)	2.00 <2.00> (?)
...main/java/co/elastic/apm/attach/JvmDiscoverer.java	39.74% <39.74%> (ø)	0.00 <0.00> (?)
...i/src/main/java/co/elastic/apm/attach/JvmInfo.java	72.22% <72.22%> (ø)	12.00 <12.00> (?)
...ain/java/co/elastic/apm/attach/DiscoveryRules.java	80.23% <80.23%> (ø)	23.00 <23.00> (?)
...t/kafka/helper/KafkaInstrumentationHelperImpl.java
.../helper/KafkaInstrumentationHeadersHelperImpl.java
.../opentracing/impl/ScopeManagerInstrumentation.java
...agent/bci/IndyBootstrapDispatcherModuleSetter.java
... and 47 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffbaaa0...dfff7d9. Read the comment docs.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #1667 updated

• Start Time: 2021-03-24T09:19:58.038+0000

• Duration: 52 min 1 sec

• Commit: 687a341

Test stats 🧪

Test	Results
Failed	0
Passed	1912
Skipped	19
Total	1931

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1912
Skipped	19
Total	1931

[SylvainJuge]

--pid is deprecated in favor of --include-pid

Seems to contradict

The -p/--pid options have been removed in favor of the --include-pid option.

Given the number of changes here, breaking compatibility with the --pid option could be a relevant option to ensure that other changes are not silently ignored on upgrade.

Also, most of the support cases we had in the past with --include/--exclude were due to the fact that it was relying on jps output parsing which isn't the most straightforward.

What exactly would be the use-case of the attacher without bundled agent in practice ? Having them bundled together tends to simplify deployment and compatibility checks.

[felixbarny]

--pid is deprecated in favor of --include-pid

That's outdated. I removed -p/--pid.

What exactly would be the use-case of the attacher without bundled agent in practice ? Having them bundled together tends to simplify deployment and compatibility checks.

I think that this is the mode we'll be using when integrating APM Server. While we want to be able to update the Java agent independently from APM Server, there is a strong dependency between the UI, the server and the attacher-cli. These three components have to have a known set of features. If we were to implement a new type of include/exclude matcher in the attacher-cli in the future, we couldn't easily expose it in the UI if different versions of the attacher-cli may be used.

So, we'd bundle the attacher-cli-slim into APM Server and download the agent jar on demand, based on what's configured in the Fleet UI.

[felixbarny]

Note that there's still the non-slim version of the attacher-cli that does ship with a bundled agent jar.

[SylvainJuge]

LGTM, only few comments/questions.
Will definitely make attachment easier, thus a breaking change is welcome for once !