cmd/pprofessor: introduce pprof-based tool

[axw]

Motivation/summary

pprofessor is based on the pprof tool, using an alternative
"Fetcher" implementation that queries Elasticsearch to
aggregate continuous profiling samples recorded by APM Server.

I haven't gone through all of the pprof flags to make sure
they work, so some may not make sense. There are a couple of
flags added specifically for our use case:

• -service, which defines the service name to filter on.
• -start, which defines the start timestamp (or date math)
  from which to start aggregating. If specified along with
  -seconds, the latter controls the end time. If only -start
  is specified, then the end time is now. If only -seconds is
  specified, then the end time is now and the start time is duration
  seconds before now.

We record the number of profiles and documents aggregated in
profile comments. All metrics (currently: cpu, inuse/allocated
heap objects/space) are aggregated into a single profile.

To aggregate duration correctly, we introduce the profile.id
field, which is a unique ID generated per profile, shared by
all sample docs derived from a profile.

No automated tests for now, and comes with no warranty or support.

Note that there is a known issue with how the Go Agent reports "alloc"
samples: it reports ever-increasing counters, rather than deltas. We
need to change this, so we can visualise the allocations within a define
time range. See elastic/apm-agent-go#708

Checklist

• I have signed the Contributor License Agreement.
  - [ ] I have updated CHANGELOG.asciidoc

I have considered changes for:
- [ ] documentation
- [ ] logging (add log lines, choose appropriate log selector, etc.)
- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)

• automated tests (add tests for the code changes, all unit tests pass locally)
  - [ ] telemetry
  - [ ] Elasticsearch Service (https://cloud.elastic.co)
  - [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
  - [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)

How to test these changes

• Run APM Server with CPU and Heap profiling enabled in APM Server:

apm-server -E apm-server.instrumentation.enabled=true -E apm-server.instrumentation.profiling.cpu.enabled=true -E apm-server.instrumentation.profiling.cpu.interval=10s -E apm-server.instrumentation.profiling.heap.enabled=true

• Run pprofessor:

go run ./cmd/pprofessor -start=now-1h -service=apm-server --http=:6060 http://admin:changeme@localhost:9200

Demo

Related issues

Closes #3828

[apmmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #4017 updated]

• Start Time: 2020-08-05T03:52:09.483+0000

• Duration: 46 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3223
Skipped	153
Total	3376

Steps errors

Expand to view the steps failures

• Name: Compress

  • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

  • Duration: 0 min 0 sec

  • Start Time: 2020-08-05T04:05:23.112+0000

  • log

• Name: Compress

  • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

  • Duration: 0 min 0 sec

  • Start Time: 2020-08-05T04:29:41.230+0000

  • log

• Name: Test Sync

  • Description: ./script/jenkins/sync.sh

  • Duration: 3 min 48 sec

  • Start Time: 2020-08-05T04:02:19.840+0000

  • log

[jalvz]

This doesn't seem to work for me with the 7.8 stack:

http://admin:changeme@localhost:9200: [400 Bad Request] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:103] [composite] after doesn't support values of type: VALUE_NULL"}],"type":"x_content_parse_exception","reason":"[1:103] [composite] after doesn't support values of type: VALUE_NULL"},"status":400}

A few other notes/questions:

• Non obvious requirements for running this should be documented (min stack version, Graphviz, etc)
• The command semantics are not so intuitive to me:
  Since this is mostly for us, I'd expect service to be hardcoded to apm-server, or at least default to it.
  The behaviour of specifying only seconds is conventional? it doesn't look so useful..
  I think the Elasticsearch URL should have its own flag, as other required arguments have it too.
• While not tested, I'd remove all non-essential code, like tls_ca handling.
• The expression "unique profile ID shared by all samples" is not so intuitive, consider profileKey, profileGroupID, or similar.
• It would be more coherent to use types for both Elasticsearch responses and queries or none, but not only responses.
• I think Elasticsearch types should live on their own package, or at least their own file.

[axw]

@jalvz thanks for the review!

http://admin:changeme@localhost:9200: [400 Bad Request] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:103] [composite] after doesn't support values of type: VALUE_NULL"}],"type":"x_content_parse_exception","reason":"[1:103] [composite] after doesn't support values of type: VALUE_NULL"},"status":400}

That's probably because there's a composite aggregation on profile.id, which is introduced in this PR. You would need to build and run apm-server to use the tool. (Part of the reason for creating this tool is to surface missed data/requirements such as this.)

Non obvious requirements for running this should be documented (min stack version, Graphviz, etc)

I can add a basic README. Until then: my intention was for the tool to be used with the same (branch) stack & server version. Otherwise, no additional dependencies apart from pprof's (which includes Graphviz).

The command semantics are not so intuitive to me:
Since this is mostly for us, I'd expect service to be hardcoded to apm-server, or at least default to it.

I'll set "apm-server" as the default.

The behaviour of specifying only seconds is conventional? it doesn't look so useful..

That's one of the standard pprof flags. If you run go tool pprof -seconds=60 http://service:1234/debug/pprof/profile it'll profile for 60 seconds.

I think the Elasticsearch URL should have its own flag, as other required arguments have it too.

This is part of the pprof tool: the "source" (i.e. Elasticsearch URL in this case) is a positional argument.

While not tested, I'd remove all non-essential code, like tls_ca handling.

Sure, I'll remove it until it's needed.

The expression "unique profile ID shared by all samples" is not so intuitive, consider profileKey, profileGroupID, or similar.

Are you referring to the wording in fields.yml? I'll try to clarify the description if that's what you mean. I don't think we should change the field name. It is the ID of the profile to which the samples belong. Parent/child (1:n) relationship.

It would be more coherent to use types for both Elasticsearch responses and queries or none, but not only responses.

The reason for having struct types for responses is because we need to operate on the results. The requests are just JSON marshalled, so it doesn't really matter if they're structs or maps. I did what's simplest.

I think Elasticsearch types should live on their own package, or at least their own file.

-1 on separate package, see https://dave.cheney.net/practical-go/presentations/qcon-china.html#_consider_fewer_larger_packages

If you think it's really worthwhile I can create a separate file, but I don't think it is. The types are used in exactly one place.

[jalvz]

You would need to build and run apm-server to use the tool

Don't follow. Did make and make update, then ran with relevant flags. Anything else needed?

no additional dependencies apart from pprof's (which includes Graphviz).

Hmm, I actually had to apt-install it...

[axw]

You would need to build and run apm-server to use the tool

Don't follow. Did make and make update, then ran with relevant flags. Anything else needed?

Looking at the error message again, maybe I introduced a bug in between testing and proposing this PR. I'll look into it and come back.

no additional dependencies apart from pprof's (which includes Graphviz).

Hmm, I actually had to apt-install it...

Sorry, what I meant was "no additional dependencies other than pprof's dependencies, among which is Graphviz" -- not that it's included within the pprof binary.

[axw]

Looking at the error message again, maybe I introduced a bug in between testing and proposing this PR. I'll look into it and come back.

I hadn't tested under the scenario where there was an existing index/template without the profile.id field. If you remove your index and overwrite the template then it should work. Either way, I've added a bug fix which will avoid the error noted.

I've added a README and (hopefully) clarified the meaning of profile.id.

[jalvz]

thanks for the readme and changes!

[axw]

Merging without a changelog entry, as this is not intended for end users.