WIP on intake api v2

[roncohen]

This is a WIP/POC that introduces a new endpoint /v2/intake which accepts data in the ndjson format.

It shows how we can have both v1 and v2 until we deprecate v1. There is still quite some work to do on this before i would consider it merge ready, but it should allow us to start testing agents against a v2 intake APM Server. Crucially, we need to test that the performance improvements we expected in the agent and server actually exist.

Each entity (error, transaction, span) should be on its own line. They need to have a _type that is one of error, transaction, span. There's also the header type, which should be the first object in the stream. See the example payload.

You can test it by running:

$ curl --data-binary @tests/data/valid/transaction/payload-v2.ndjson localhost:8200/v2/intake  -H "Content-Type: application/ndjson"

there a quite a few changes in here. Reading this list should help mentally decode the PR:

• introduce TransformContext. This structure holds things like service, process etc.. Things that are shared between transformations. The implementation is not pretty (caching the transformation result) and needs some love, but i think it's a step in the right direction.
• rename config.Config to config.TransformConfig. TransformConfig depends on the the endpoint, apm-server config etc. In the future we should consider if it makes sense to merge TransformConfig and TransformContext.
• felt like we had two different concepts called "decoders":
  1. decode from map[string]interface{} to a struct (for example DecodeProcess)
  2. decode from http.Request to map[string]interface{} (DecodeUserData)
     I've renamed (ii) to extractors and separated the augmentation into a separate thing.
• decode functions (i) for transactions, spans and errors now all returns []Transformable which can be transformed asynchronously. I needed to separate transformation from the payload to implement the stream-like v2 intake
• removed health-check processor at it was not doing anything. Generally, i think we should get rid of the concept of the processors entirely as they don't really serve any purpose at the moment.
• Transform functions for errors, transactions and spans return a beat.Event directly now. They used to return just a mapStr.
• there are now individual generated schemas for each entity. I have also moved them to a sub-directory so they don't just sit next to the regular code (processor/error/generated-schemas/error.go for example). I needed to separate the generated schemas out from the payloads so entities could be validated independently.
• spans can now include a timestamp and a transaction.id. This was required when they no longer are nested under transactions in the API. Still missing: validating the presence of these fields when using intake v2
• lots of tests still need to be written or adopted

[roncohen]

Closing for now. Working on a production oriented version instead.