Add pipeline support to Ingest API

[Bargs]

Requires #6027 and a build of Elasticsearch with the ingest functionality (at time of writing, a build of the master branch)

This PR adds pipeline support to the ingest API.

If you like Postman, import this collection with example requests to play around with.

An array of processors can be included under a pipeline key in the JSON payload provided to the POST endpoint. The Kibana ingest API will use this list of processors to create an ES ingest pipeline. As a result, you'll need to test this PR against a build from ES master (or clone the ES repo and run gradle run).

I've created a ticket to add API tests once Kibana master has been updated to track ES master (#6035).

[jbudz]

is the postman collection up to date? I see a delete and create.

[jbudz]

I might need a walkthrough on this, these are always supposed to throw errors?

[Bargs]

Yeah, so the API creates an index pattern, an index template, and a pipeline. It should be an atomic operation, so if any one item fails, we need to rollback the previous steps. This patternRollback function gets called when, for instance, the template creation fails but the index pattern was already created successfully. If the index pattern rollback succeeds, patternRollback throws the original template creation error so that it gets returned to the client and they can fix their error. If the rollback fails, patternRollback throws an error which lets the user know that things have been left in a partially completed state, along with the original template creation error so they know why a rollback was attempted in the first place.

Let me know if that helps at all. I'd be happy to hop on zoom tomorrow and talk through it some more.

[Bargs]

The API only supports DELETE and POST right now, so you should only see those two examples in Postman. It should be up to date, does the POST body have an index_pattern and pipeline key?

[Bargs]

Note, tests are only failing because it depends on this fix #6027 getting merged

[jbudz]

@Bargs no pipeline property: id, title, time_field_name, fields array. updated November 25

[Bargs]

@jbudz ugh, the Postman sync doesn't seem to work very well. I've had to fiddle with it in the past to get it to re-sync. I'll touch base with you on Monday to try to get it working.

[Bargs]

@jbudz could you try to re-import the collection now? I think I finally got it to re-sync.

[jbudz]

Got the updates, thanks. Having some issues with the requests:
Running delete gives me:

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "An internal server error occurred"
}

Running create for the first time after deleting gives me a stack trace on the server and responds with:

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "TypeError: Cannot read property 'debug' of undefined"
}

Can we replace the message with less code level information/something more user friendly?

The second create gives me a 409 conflict

[Bargs]

@jbudz I just realized even the existing tests and Postman examples aren't going to work without running against a build of elasticsearch master. So this PR really has #5656 as a hard requirement. I think I'm going to recreate this PR against the feature/ingest branch so I can get it hooked up to the ingest UI without breaking master. For now I'll assign this back to myself.

Regarding the error messages, I'll have to give it some thought. Generally unknown errors will return a 500 to the client with a generic error message, and a detailed stack trace in the server logs. I added TypeError as an exception to that rule, because the elasticsearch.js client was throwing a TypeError during template creation when the user sent some bad request payload (can't remember the exact details since it was a couple months ago...). But it looks like returning a 400 for all TypeErrors is too general a rule, so I'll have to figure out something that works better.