Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Allow "no timestamp" to be specified as the timestamp format in file upload #146700

Closed
droberts195 opened this issue Nov 30, 2022 · 3 comments · Fixed by #147504
Closed

[ML] Allow "no timestamp" to be specified as the timestamp format in file upload #146700

droberts195 opened this issue Nov 30, 2022 · 3 comments · Fixed by #147504
Assignees
Labels
enhancement New value added to drive a business result Feature:File and Index Data Viz ML file and index data visualizer Feature:File Upload :ml v8.7.0

Comments

@droberts195
Copy link
Contributor

In elastic/elasticsearch#90764 we added the ability to specify null as the timestamp format override in the file structure finder endpoint.

The file data visualizer/file upload in Kibana can now be enhanced to make this option available. If a user chooses to add overrides then specifying "there is no timestamp" should be an alternative to overriding the timestamp format and if the user selects this then timestamp_format=null should be passed as an argument the next time the file structure finder endpoint is called.

Relates to #63526

@droberts195 droberts195 added enhancement New value added to drive a business result :ml Feature:File and Index Data Viz ML file and index data visualizer Feature:File Upload labels Nov 30, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@jgowdyelastic
Copy link
Member

jgowdyelastic commented Dec 14, 2022

@droberts195

When analysing a semi structured text file with no timestamps, I get an error saying no timestamp could be found. If I then run analysis again, setting timestamp_format to null it works correctly.
The original error is

[illegal_argument_exception: [illegal_argument_exception] Reason: Could not find a timestamp in the sample provided]: Could not find a timestamp in the sample provided
which makes me wonder why the original analysis can't perform a check to see if using timestamp_format: null produces good results.

As it is now, the UI user has to manually run the analysis again. The UI could look out for this error and attempt a run with timestamp_format: null automatically, but it would be faster if es did this instead.

@droberts195
Copy link
Contributor Author

The UI could look out for this error and attempt a run with timestamp_format: null automatically, but it would be faster if es did this instead.

The problem is, 9 times out of 10 a semi-structured log file will have a timestamp but we just didn't detect it. Assuming each line in the file is a separate message will always work, but will look very stupid to a user who can see the timestamp in a format that makes complete sense to them but was missed by our timestamp detection.

jgowdyelastic added a commit that referenced this issue Dec 16, 2022
Adds a checkbox to the overrides flyout to allow the user to tell the
find structure endpoint that the data does not contain a time field.


![image](https://user-images.githubusercontent.com/22172091/207582663-a510083e-be75-4ab8-bf7e-c91d72734c6f.png)

If the original analysis of the data does not find a time field, this
checkbox is unchecked by default.

Closes #146700

Also removes the jest snapshot test for the override flyout as it isn't
very useful.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Feature:File and Index Data Viz ML file and index data visualizer Feature:File Upload :ml v8.7.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants