HTTP API for Logstash Centralized Configuration Management

[elasticmachine]

Original comment by @ycombinator:

Currently, if users want to programatically use Logstash Centralized Configuration Management, they must directly read/write from the .logstash index in Elasticsearch. This is obviously not ideal for at least a couple of reasons:

• Such users end up getting tied to the exact structure of the .logstash index, so any changes we might make to it will break such users.

• As the .logstash index grows in structure to accommodate different aspects of centralized configuration management, users could easily misuse it and cause unintended side effects / breakages in their Logstash deployments.

A safer alternative is to provide an HTTP API instead as a more abstract and constrained programmatic interface. Chatting with @epixa, since the .logstash index doesn't actually get used by any processes within Elasticsearch itself (unlike, say, the Watcher or Security indices), it makes sense for the Logstash Centralized Configuration HTTP API to be exposed from Kibana.

As part of creating this HTTP API, we'll also want to document it as part of our usual docs (i.e. the ones that show up on elastic.co) and create API integration tests for it.

[elasticmachine]

Original comment by @ycombinator:

/cc @acchen97 @andrewvc @exekias @brandonmensing

[elasticmachine]

Original comment by @ycombinator:

@tylersmalley Court mentioned that you had put some thought/effort into docs for publicizing a saved objects API. Care to share what you had in mind? Thanks.

[elasticmachine]

Original comment by @ycombinator:

For existing Logstash Centralized Management functionality, I'm thinking we need the following APIs:

• Create a new pipeline / update an existing pipeline — PUT /api/logstash/pipeline/{id}
• Delete an existing pipeline — DELETE /api/logstash/pipeline/{id}
• Retrieve an existing pipeline — GET /api/logstash/pipeline/{id}
• List existing pipelines — GET /api/logstash/pipelines

A few notes about the above set:

• I'm not married to the routes; happy to discuss alternatives.
• I'd punt on a more sophisticated listing/search API for now. Lets build this one when we hear or otherwise develop the need for it.
• Yes, for now, these APIs will be very thin wrappers around corresponding ES APIs on the .logstash index. That's fine; we're mainly just laying a foundation here for safe programmatic access. In the future I can imagine this API surface growing with additional Centralized Configuration Management features such as pipeline settings, ancillary configs, and groups/tags.

[elasticmachine]

Original comment by @ycombinator:

We might also want to mark these API endpoints as "beta" (or similar) for the first couple of releases, get some customers using it, massage its shape if necessary, and then remove the "beta" label on we feel the shape has stabilized.

[elasticmachine]

Original comment by @ycombinator:

Steal doc format from LINK REDACTED which builds on top of elastic/docs#309.

[elasticmachine]

Original comment by @brandonmensing:

I'll be a beta dissenter (as usual) and ask that we consider doing enough in advance (alpha testing with customer, etc) to avoid a beta label. If we find that we want a slightly different API or even a whole new concept, then perhaps we could add more APIs in a minor, deprecating others toward removal in 7.0?

Putting this into beta as a whole would make sense if we didn't know how the feature would behave for a real customer, but here it feels like it's more about being able to make changes. Meanwhile, beta means we're not recommending it for production and won't take sev1/2. So this is an improvement to a paid feature but not one that our paying customers should use?

Beside the beta matter, I'm on board with APIs for LS central config :)

[elasticmachine]

Original comment by @ycombinator:

So this is an improvement to a paid feature but not one that our paying customers should use?

That's a good point, @brandonmensing. I'd be okay with building this API and pointing customers who're asking for it to it. Once we get some usage feedback from them (and make any necessary changes), we can publish documentation for it, taking it directly to GA. Thoughts, @epixa?

[elasticmachine]

Original comment by @epixa:

I don't have strong opinions here about whether this is marked as a beta/experimental when it ships. It really comes down the ingest team's confidence in it and their commitment to maintaining backwards compatibility between minors and reasonably upgrade paths in majors.

There isn't a middleground here though - BC considerations begin the moment we ship a feature as GA. If your goal is to ship a feature and only pointing some customers to it to gather feedback and whatnot, then the only option you have is experimental or beta.

[elasticmachine]

Original comment by @ycombinator:

@acchen97 @brandonmensing @andrewvc When the API being proposed in this issue is implemented, one major change will be required from existing (and new) users of Logstash Centralized Management. Instead of setting xpack.management.elasticsearch.* in their logstash.yml, we'd want them to setup the analogous xpack.management.kibana.* settings instead. Obviously, the Logstash code working off these settings will need to change as well.

Given this, is this API something we necessarily ought to introduce soon, in a 6.x release, and support either set of settings, deprecating the xpack.management.elasticsearch.* ones and warning about them? Then in 7.0 we can fail hard if those settings are found. Thoughts?

[elasticmachine]

Original comment by @brandonmensing:

One way to address the question is the supported versions. The LS table in our support matrix pattern of being very compatible - allowing any 6.x to work with any 6.x so far (between LS and Monitoring). So I think your statement of supporting both old and new fits that precedent for 6.x and then deprecating in 7.0 makes sense.

[elasticmachine]

Original comment by @ycombinator:

After consulting with docs folks, I've started the API documentation work here: #17706

[elasticmachine]

Original comment by @ycombinator:

Here are the steps I see in resolving this issue:

1. Tweak the existing APIs used by the UI code to line up with the proposal - LINK REDACTED

2. Document APIs for public consumption - Document Logstash Centralized Configuration Management HTTP APIs #17706

3. Update LS core code to start using the proposed APIs, including using xpack.management.kibana.* settings and deprecating the xpack.management.elasticsearch.* settings - [Centralized Management] Accept xpack.management.kibana.* settings logstash#9477

[elasticmachine]

Original comment by @andrewvc:

@ycombinator @brandonmensing yeah, this is a bit weird. So, we'd have to maintain both code paths for 6.x? A kibana one, and an ES one?

I think that's actually not going to be too bad in terms of complexity, it just isn't ideal.

Is there a specific 6.x version you all are targeting? I assume 6.4?

[elasticmachine]

Original comment by @ycombinator:

Thanks @andrewvc. Yes, I was targeting 6.4, but that will come down to whether we can get the x-pack LS changes made by then. The UI/doc changes are already in progress, as you can see in the comment two comments before this one.

[ycombinator]

The Kibana end of this work is done so I'm going to close this issue. What remains is for Logstash core to use the Kibana HTTP APIs instead of talking directly to ES. That work is being tracked in this Logstash issue: elastic/logstash#9477.