CSV Upload: Translate indexing errors into plain english

[Bargs]

CSV Upload currently returns the raw error message from Elasticsearch when there's an issue indexing a document. Based on error type, we can probably translate these errors into something more user friendly. For instance when we get a mapper_parsing_exception we could display a message like:

"Line 8: Failed to parse field @timestamp as type date"

Based on convo with @gmoskovicz in #7620

[bahaaldine]

To follow up, in the example I gave about date parsing exception: I think I've seen this error message in Elasticsearch when trying to index with Logstash, the message was something like:

"could not index field '2016-01-01 11:11:11' with date pattern 'yyyy-dd-mm hh:MM:ssSSS' ...."

So anything that we can pull out Elasticsearch response message that helps to understand why the parsing error occurred would help here.

Another point; I'm not sure we need to summarize or to filter Elasticsearch output even if it's dense, if we we switch from plain list to textarea as explained in #7648

The user will be able to have the complete log to debug; and that's always better.

[gmoskovicz]

Agree in every point.

[tbragin]

@Bargs One of the problems I've seen with relying on Elasticsearch errors is that when a big CSV file is uploaded, user does not see the errors until the whole file is processed. Meanwhile Elasticsearch is spewing out a stream of errors for a long time. Not sure much can be done about this until editable pipelines, but figured I'd see if you have any clever ideas.

[Bargs]

@tbragin It should be fairly easy to update the UI to show indexing results in real time as they're streaming back from ES. However, I imagine the user would want to take some action based on this information, like cancel the request if they see a truck load of errors? This might be a tad bit more complicated, but I'm sure we could do it. Would it be useful at all to see realtime results without the ability to cancel a request?

[bahaaldine]

If real time logs are shown to the user, I think that a cancel action should be implemented, at least the user can stop the ingestion process and if some data has been indexed, the index should be removed.

In terms of experience, if we could simulate the indexation of a sample to see if there are potential errors, the user will know before processing the whole file what are the tweaks that he should apply, which would cause less frustration.

As far as I understood, the import data UI will at some point leverage the ingest node API and thus we could use the relative simulate API and have real time logs there.

[Bargs]

As far as I understood, the import data UI will at some point leverage the ingest node API and thus we could use the relative simulate API and have real time logs there.

This is true, however the pipeline simulate functionality doesn't take existing mappings into account so it won't catch those errors, and I suspect mapping errors are the most common.

[tbragin]

@Bargs That's a good point... probably not. Elasticsearch has a task management API, would that be something that could be used to track the progress of and cancel a request? https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html

[Bargs]

We'd have to implement it on the Kibana side, the CSV actually gets ingested in mini 100 doc bulk requests, so we'd need a way to tell Kibana to stop sending those bulk requests. Now that I think about it, it might be enough to simply kill the request from the client side, because Kibana sends those bulk requests as it receives the data from the client in a stream. I'll have to play around with it, might turn out to be easier than I thought.

[epixa]

I'm going to close this since CSV upload was pulled. We can always refer back to this issue if/when we revisit the feature.