Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Migrate es-restclient to indy plugin #1356

Conversation

videnkz
Copy link
Contributor

@videnkz videnkz commented Aug 26, 2020

relates to #1337
migration of es-restclient to indy plugin

@@ -77,27 +71,25 @@ public ElasticsearchClientAsyncInstrumentation(ElasticApmTracer tracer) {


public static class ElasticsearchRestClientAsyncAdvice {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixbarny after running tests, I got next exception,
this condition classLoader instanceof ExternalPluginClassLoader in ElasticApmAgent#validateAdvice return false:

java.lang.IllegalStateException: Indy-dispatched advice class must be at the root of the instrumentation plugin.

	at co.elastic.apm.agent.bci.ElasticApmAgent.validateAdvice(ElasticApmAgent.java:461)
	at co.elastic.apm.agent.bci.ElasticApmAgent.getTransformer(ElasticApmAgent.java:397)
	at co.elastic.apm.agent.bci.ElasticApmAgent.applyAdvice(ElasticApmAgent.java:359)
	at co.elastic.apm.agent.bci.ElasticApmAgent.initAgentBuilder(ElasticApmAgent.java:282)
	at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:230)
	at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:150)
	at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:141)
	at co.elastic.apm.agent.AbstractInstrumentationTest.beforeAll(AbstractInstrumentationTest.java:60)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll have to rename the package to co.elastic.apm.agent.es.restclient.v5_6 co.elastic.apm.agent.esrestclient5_6, or similar.

The reason is that we rely on package scanning to determine which classes to load from the IndyPluginClassLoader. Once we have completed the migration, we might be able to package each plugin in its own jar and inject the whole jar instead of scanning a package.

I just noticed another issue: it's not easy to migrate this plugin as apm-es-restclient-plugin-common obviously has a different package name than apm-es-restclient-plugin-5_6 and thus the package scanning would fail. This means the common plugin would still be loaded form the bootstrap classloader. This would result in NoClassDefFoundErrors for org.elasticsearch.client.Response, for example.

I don't have a clear answer how to fix it. One approach could be to remove the dependency on the ES client in apm-es-restclient-plugin-common. Not sure if that's feasible though.

This is a tricky plugin to migrate.

Copy link
Contributor

@eyalkoren eyalkoren Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be OK if you remove the v5_6 and v6_4 packages entirely and use the versions in the Instrumentation and Advice class names instead. If all advices are in the co.elastic.apm.agent.es.restclient it should work.

@videnkz videnkz changed the title moved helper to advice classes. Migrate es-restclient to indy plugin WIP: moved helper to advice classes. Migrate es-restclient to indy plugin Aug 26, 2020
@apmmachine
Copy link
Contributor

apmmachine commented Aug 26, 2020

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Reason: The PR is not allowed to run in the CI yet

  • Start Time: 2020-11-21T11:39:23.479+0000

  • Duration: 4 min 50 sec

  • Commit: 59ef6ba

Steps errors 1

Expand to view the steps failures

Error signal

  • Took 0 min 0 sec . View more details on here
  • Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permission

Log output

Expand to view the last 100 lines of log output

[2020-11-21T11:42:46.825Z] Cloning repository https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:42:46.873Z]  > git init /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1356 # timeout=10
[2020-11-21T11:42:47.735Z] Pruning obsolete local branches
[2020-11-21T11:42:46.935Z] Using reference repository: /var/lib/jenkins/.git-references/apm-agent-java.git
[2020-11-21T11:42:46.942Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:42:46.943Z]  > git --version # timeout=10
[2020-11-21T11:42:46.963Z]  > git --version # 'git version 2.17.1'
[2020-11-21T11:42:46.963Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2020-11-21T11:42:46.965Z]  > git fetch --no-tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2020-11-21T11:42:47.706Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2020-11-21T11:42:47.717Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2020-11-21T11:42:47.726Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2020-11-21T11:42:47.740Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:42:47.740Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2020-11-21T11:42:47.741Z]  > git fetch --no-tags --progress --prune -- https://github.com/elastic/apm-agent-java.git +refs/pull/1356/head:refs/remotes/origin/PR-1356 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2020-11-21T11:42:48.030Z] Merging remotes/origin/master commit a98d6caa448f66a9e563e438fe554de66f5dcd30 into PR head commit 59ef6ba54293510fb35a74553c6e2c8329443389
[2020-11-21T11:42:48.771Z] Merge succeeded, producing 9d45dacd8b1326d20f934fae3d363af196d264e9
[2020-11-21T11:42:48.771Z] Checking out Revision 9d45dacd8b1326d20f934fae3d363af196d264e9 (PR-1356)
[2020-11-21T11:42:48.039Z]  > git config core.sparsecheckout # timeout=10
[2020-11-21T11:42:48.051Z]  > git checkout -f 59ef6ba54293510fb35a74553c6e2c8329443389 # timeout=15
[2020-11-21T11:42:48.240Z]  > git remote # timeout=10
[2020-11-21T11:42:48.247Z]  > git config --get remote.origin.url # timeout=10
[2020-11-21T11:42:48.260Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2020-11-21T11:42:48.261Z]  > git merge a98d6caa448f66a9e563e438fe554de66f5dcd30 # timeout=10
[2020-11-21T11:42:48.756Z]  > git rev-parse HEAD^{commit} # timeout=10
[2020-11-21T11:42:48.775Z]  > git config core.sparsecheckout # timeout=10
[2020-11-21T11:42:48.782Z]  > git checkout -f 9d45dacd8b1326d20f934fae3d363af196d264e9 # timeout=15
[2020-11-21T11:42:52.588Z] Commit message: "Merge commit 'a98d6caa448f66a9e563e438fe554de66f5dcd30' into HEAD"
[2020-11-21T11:42:52.618Z] First time build. Skipping changelog.
[2020-11-21T11:42:52.593Z]  > git rev-list --no-walk b03283790627518304ebc8605e425f40cef3eae3 # timeout=10
[2020-11-21T11:42:53.373Z] Masking supported pattern matches of $JOB_GCS_BUCKET or $NOTIFY_TO
[2020-11-21T11:42:53.420Z] Timeout set to expire in 1 hr 0 min
[2020-11-21T11:42:53.433Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2020-11-21T11:42:53.664Z] [INFO] Number of builds to be searched 10
[2020-11-21T11:42:53.949Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2020-11-21T11:42:53.965Z] Running in /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1356/src/github.com/elastic/apm-agent-java
[2020-11-21T11:42:53.981Z] [INFO] gitCheckout: Checkout SCM PR-1356 with some customisation.
[2020-11-21T11:42:54.004Z] [INFO] Override default checkout
[2020-11-21T11:42:54.042Z] Sleeping for 10 sec
[2020-11-21T11:43:04.160Z] using credential 2a9602aa-ab9f-4e52-baf3-b71ca88469c7-UserAndToken
[2020-11-21T11:43:04.221Z] Cloning the remote Git repository
[2020-11-21T11:43:04.237Z] Cloning repository https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:43:04.257Z]  > git init /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1356/src/github.com/elastic/apm-agent-java # timeout=10
[2020-11-21T11:43:04.265Z] Using reference repository: /var/lib/jenkins/.git-references/apm-agent-java.git
[2020-11-21T11:43:04.265Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:43:04.265Z]  > git --version # timeout=10
[2020-11-21T11:43:04.270Z]  > git --version # 'git version 2.17.1'
[2020-11-21T11:43:04.271Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2020-11-21T11:43:04.272Z]  > git fetch --tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2020-11-21T11:43:04.784Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2020-11-21T11:43:04.795Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2020-11-21T11:43:04.806Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2020-11-21T11:43:04.818Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2020-11-21T11:43:04.819Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2020-11-21T11:43:04.820Z]  > git fetch --tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/pull/1356/head:refs/remotes/origin/PR-1356 +refs/heads/master:refs/remotes/origin/master # timeout=10
[2020-11-21T11:43:05.126Z] Checking out Revision 59ef6ba54293510fb35a74553c6e2c8329443389 (origin/PR-1356)
[2020-11-21T11:43:05.287Z] Commit message: "renamed packages"
[2020-11-21T11:43:05.287Z] First time build. Skipping changelog.
[2020-11-21T11:43:05.122Z]  > git rev-parse origin/PR-1356^{commit} # timeout=10
[2020-11-21T11:43:05.129Z]  > git config core.sparsecheckout # timeout=10
[2020-11-21T11:43:05.132Z]  > git checkout -f 59ef6ba54293510fb35a74553c6e2c8329443389 # timeout=10
[2020-11-21T11:43:06.133Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2020-11-21T11:43:06.770Z] + git fetch https://****:****@github.com/elastic/apm-agent-java.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2020-11-21T11:43:07.802Z] Archiving artifacts
[2020-11-21T11:43:08.450Z] + git rev-parse HEAD
[2020-11-21T11:43:08.791Z] + git rev-parse HEAD
[2020-11-21T11:43:09.090Z] + git rev-parse origin/pr/1356
[2020-11-21T11:43:09.137Z] [INFO] githubEnv: Found Git Build Cause: pr
[2020-11-21T11:43:09.464Z] Masking supported pattern matches of $GITHUB_TOKEN
[2020-11-21T11:43:10.544Z] [INFO] githubPrCheckApproved: Title: WIP:  Migrate es-restclient to indy plugin - User: kananindzya - Author Association: CONTRIBUTOR
[2020-11-21T11:43:10.822Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2020-11-21T11:43:10.822Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2020-11-21T11:43:10.861Z] [INFO] Let's stop build #15. The PR is not allowed to run in the CI yet
[2020-11-21T11:43:10.876Z] Sleeping for 5 sec
[2020-11-21T11:43:11.860Z] Stage "Build" skipped due to earlier failure(s)
[2020-11-21T11:43:11.938Z] Stage "Tests" skipped due to earlier failure(s)
[2020-11-21T11:43:11.992Z] Stage "Unit Tests" skipped due to earlier failure(s)
[2020-11-21T11:43:11.993Z] Stage "Smoke Tests 01" skipped due to earlier failure(s)
[2020-11-21T11:43:11.994Z] Stage "Smoke Tests 02" skipped due to earlier failure(s)
[2020-11-21T11:43:11.995Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2020-11-21T11:43:11.996Z] Stage "Javadoc" skipped due to earlier failure(s)
[2020-11-21T11:43:12.051Z] Failed in branch Unit Tests
[2020-11-21T11:43:12.052Z] Failed in branch Smoke Tests 01
[2020-11-21T11:43:12.053Z] Failed in branch Smoke Tests 02
[2020-11-21T11:43:12.054Z] Failed in branch Benchmarks
[2020-11-21T11:43:12.054Z] Failed in branch Javadoc
[2020-11-21T11:43:12.129Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2020-11-21T11:43:12.163Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2020-11-21T11:43:12.186Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2020-11-21T11:43:12.440Z] Running on worker-1095690 in /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1356
[2020-11-21T11:43:12.524Z] [INFO] getVaultSecret: Getting secrets
[2020-11-21T11:43:12.871Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-11-21T11:43:14.868Z] + chmod 755 generate-build-data.sh
[2020-11-21T11:43:14.868Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1356/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1356/runs/15 ABORTED 229989
[2020-11-21T11:43:14.868Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1356/runs/15/steps/?limit=10000 -o steps-info.json
[2020-11-21T11:43:16.322Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1356/runs/15/tests/?status=FAILED -o tests-errors.json
[2020-11-21T11:43:17.025Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2020-11-21T11:43:19.296Z] Retry 2/3 exited 22, retrying in 2 seconds...
[2020-11-21T11:43:21.563Z] Retry 3/3 exited 22, no more retries left.
[2020-11-21T11:43:21.564Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1356/runs/15/log/ -o pipeline-log.txt

@videnkz videnkz changed the title WIP: moved helper to advice classes. Migrate es-restclient to indy plugin WIP: Migrate es-restclient to indy pluginhttps://github.com/kananindzya Aug 26, 2020
@videnkz videnkz changed the title WIP: Migrate es-restclient to indy pluginhttps://github.com/kananindzya WIP: Migrate es-restclient to indy plugin Aug 26, 2020
@videnkz videnkz force-pushed the issue-1337-migrate-indy-es-restclient-plugin branch from a8684d3 to 59ef6ba Compare August 29, 2020 20:16
@felixbarny felixbarny mentioned this pull request Aug 31, 2020
37 tasks
@AlexanderWert AlexanderWert added the technical-debt Technical debt label Mar 1, 2021
@felixbarny
Copy link
Member

Unfortunately, the instrumentations have changed too much in order to meaningfully rebase. I guess we'll have to start over.

@videnkz videnkz closed this Apr 15, 2021
@felixbarny
Copy link
Member

Here's a new version of it: #2088

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants