[Ingest] Editable ingest pipelines

[Bargs]

Update: After folks had a chance to play with the Add Data wizards without edit capabilities, it was decided that we at least need the ability to edit pipelines. It's very possible that a user will make mistakes when building a pipeline the first time. It's important that we allow the user to go back and correct those mistakes without having to throw away all their work and restart from the beginning.

During this "construction" phase, users probably don't care if they have to blow away any data they've indexed, they're still just playing around with things at this point. Editing pipelines is still a potentially dangerous operation that could cause conflicts if the user has already begun indexing production data. We're not trying to build a bullet proof solution for the production use case though. We'll stick a big warning at the top of the page discussing the risks.

Unanswered questions:

• How will the user navigate to this page and how will they select their pipeline (the answer will probably depend on the management redesign)
• Do we make any attempt to automatically update the index pattern and index template? Maybe we just destroy and re-create them?
• Do we give them a UI driven way to throw away the current indices, index pattern and template? Court was against this, but I think it's going to be a huge pain point for users unfamiliar with the ES request syntax if we don't provide something.

Original comment: Add data wizards create an "ingest configuration" which is made up of an index pattern, index template, and ingest pipeline. Users will want to edit these things, but there are some sharp edges we need to think about. Editing any of these can have an impact on mappings with all of the caveats that entails. It gets even more complicated considering Kibana works with a normalized set of the mappings across all of the indices that its index pattern matches.

I'd like to gather some feedback and use cases on this ticket to figure out exactly how we should expose this functionality.

[Bargs]

@epixa @BigFunger @rashidkpc @tbragin I've updated this ticket with everything I remember from our discussions. Let me know if I've missed anything or misspoke. Let's also use this ticket to track any further thoughts on the topic so we all stay on the same page with what needs to be built.

[epixa]

Great stuff.

[w33ble]

Do we give them a UI driven way to throw away the current indices, index pattern and template?

It makes sense to offer this, as I agree with you that users starting out with this will end up with a fair bit of trial and error, and they won't care about the data they indexed during their initial missteps. It's not unlike the logstash use case.

Also, this may be a bit off-topic, but has there been any discussion about importing/exporting the pipeline configuration? I know we are creating Filebeat output, but can a user import that config file to create another pipeline configuration?

[epixa]

My concern with deletion is that we're now crossing into a new frontier for Kibana and there is no mechanism now or in the short term to lessen the effect. Up until now, anyone using Kibana has the ability to perform destructive actions on Kibana itself, but that's a far cry from deleting data that kibana doesn't own. I know we're trying to make the first-time usage easier for new people, but we do need to strike a balance. Personally, I'd rather it not be easy to delete data that Kibana doesn't own just for the sake of convenience, if only to avoid that business-crushing middle-of-the-night support request when a user mistakenly deleted a few hundred TB of data from production indices instead of the development index they thought they were modifying.

[w33ble]

That's fair, but the Management app itself is sort of a new frontier, and I think for that reason. As you say, until now, Kibana has been read-only. The apps and functionality we're planning to add to Management changes that though, and turns Kibana into an administration tool as well (implying writes).

I assume there will be some access controls around this, and also some sort of confirmation before modifying/destroying existing data. Or, maybe we can make it a 2-step process, where the user has to explicitly delete/reset stuff before they re-create the pipeline? This strikes as more of a UX thing than anything - we need to make sure users don't accidentally do destructive things.

[Bargs]

if only to avoid that business-crushing middle-of-the-night support request when a user mistakenly deleted a few hundred TB of data from production indices instead of the development index they thought they were modifying

Hopefully if someone has a hundred TB cluster, they're using shield or some other security mechanism and they're not giving delete permissions to random Kibana users. If not, I think they have even bigger problems on their hands.

We could use the same argument against implementing editable pipelines at all. What if someone is indexing TBs of data though a pipeline each day, and along comes a user who edits that pipeline in such a way that it causes all documents to fail to index due to a mapping conflict? Hopefully they've set up permissions to prevent that from happening.

[epixa]

Yeah, you're right. As @w33ble mentioned, fundamentally this is more of a question around UX than anything else. So long as we give people info and make it difficult to delete data by accident, then I think we've done everything we need to. To that end, it would be awesome if we could tell people how many indices they'd be deleting at least.

[w33ble]

To that end, it would be awesome if we could tell people how many indices they'd be deleting at least.

That'd be really cool: "You're about to permanently delete X GB of data, across Y indicies and Z documents. Are you sure you want to continue?" It may also be nice to have a threshold after which we put a timeout on the prompt, or change it somehow - like, if there are more than 100 documents, maybe we can assume the data they indexed wasn't just them playing around, and we should make it harder/slower to delete things.

[Bargs]

Will there by an indices management section of the management app? If so does this functionality belong there instead, and we could link to it? That might serve as another cue that they're done with the pipeline, and now they're potentially modifying data by taking them to a section designed for that purpose.

[epixa]

We still want to do a pipelines UI, but it would almost certainly be going out in x-pack, so I'm going to close this so people don't get the wrong idea.