[ML] Adding better error reporting for reading and importing data #24269

jgowdyelastic · 2018-10-19T15:12:23Z

Adding error handling for the import data steps.
Errors are now displayed below the steps.

When importing the data, if the import endpoint fails for a chunk of data, it will attempt then upload again 5 times.
After that it will abort the upload altogether.

elasticmachine · 2018-10-19T15:12:25Z

Pinging @elastic/ml-ui

elasticmachine · 2018-10-19T16:58:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

jgowdyelastic · 2018-10-19T19:36:46Z

retest

elasticmachine · 2018-10-19T21:19:43Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

jgowdyelastic · 2018-10-22T08:37:49Z

retest

jgowdyelastic · 2018-10-22T08:40:15Z

retest

peteharverson

A typo in an error message, but otherwise LGTM

peteharverson · 2018-10-22T08:44:30Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/importer/importer.js

+    if (!id || !index) {
+      return {
+        success: false,
+        error: 'no id ot index supplied'


Typo - should be no id or index supplied or even no ID or index supplied ?

peteharverson · 2018-10-22T08:44:43Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/importer/importer.js

+    if (!id || !index) {
+      return {
+        success: false,
+        error: 'no id ot index supplied'


Typo - should be no id or index supplied or even no ID or index supplied ?

peteharverson · 2018-10-22T08:46:11Z

x-pack/plugins/ml/public/file_datavisualizer/components/summary/summary.js

-      title: 'Charset',
-      description: results.charset,
-    }
+    // {


Is this meant to be commented out? Not sure how it relates to this PR?

there is an outstanding issue with the charset of the document being sent to the endpoint always being UTF-8 due to the way it's read by the FileReader.
A future enhancement will be to read the file as an ArrayBuffer and then send it to the endpoint as it to preserve the encoding.
For now we should not list the charset because it is inaccurate.

walterra

LGTM, just one question/suggestions: Are the retries fired without any delay? If not, maybe something worth adding with a future update

alvarezmelissa87 · 2018-10-22T09:29:22Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/import_view.js

-            </React.Fragment>
+            <EuiButton
+              onClick={() => this.clickReset()}
+            >


Just curious why the onClick gets passed in as an arrow function here instead of just the clickReset function. Might be missing something here on what EuiButton is expecting.

alvarezmelissa87 · 2018-10-22T09:30:15Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/import_view.js

-            </React.Fragment>
+            <EuiButton
+              onClick={() => this.clickReset()}
+            >


Just curious why the onClick gets passed in as an arrow function here instead of just the clickReset function. Might be missing something here on what EuiButton is expecting.

alvarezmelissa87 · 2018-10-22T09:31:41Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/import_view.js

-            </React.Fragment>
+            <EuiButton
+              onClick={() => this.clickReset()}
+            >


Just curious why the onClick gets passed in as an arrow function here instead of just the clickReset function. I might be missing something here on what EuiButton is expecting, though.

alvarezmelissa87 · 2018-10-22T09:39:54Z

x-pack/plugins/ml/public/file_datavisualizer/components/import_view/import_view.js

-            </React.Fragment>
+            <EuiButton
+              onClick={() => this.clickReset()}
+            >


Just curious why the onClick gets passed in as an arrow function here instead of just the clickReset function. Same for the clickImport above on line 291.
I might be missing something here on what EuiButton is expecting, though.

elasticmachine · 2018-10-22T10:32:29Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

jgowdyelastic · 2018-10-22T10:47:04Z

retest

jgowdyelastic · 2018-10-22T10:49:57Z

retest

alvarezmelissa87

LGTM - sorry about the extra comments - looks like it was updating after all. Woops!

jgowdyelastic · 2018-10-22T11:34:14Z

retest

jgowdyelastic · 2018-10-22T11:41:05Z

retest

* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

* [ML] File datavisualizer initial commit (elastic#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (elastic#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (elastic#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (elastic#23500) * [ML] Changes to timestamp formats (elastic#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (elastic#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (elastic#23763) * [ML] Adding license check to datavisualizer landing page (elastic#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (elastic#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (elastic#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (elastic#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (elastic#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (elastic#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (elastic#24387) * [ML] Adding ability to override various settings when importing data (elastic#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

jgowdyelastic added 5 commits October 17, 2018 22:21

[ML] Adding better error reporting for reading and importing data

2c124be

changes to endpoint errors

c93df7d

displaying errors

b3390d2

step logic refactor

057c9f5

removing log statements

5a5232a

jgowdyelastic added review v7.0.0 :ml labels Oct 19, 2018

jgowdyelastic self-assigned this Oct 19, 2018

jgowdyelastic requested review from walterra, alvarezmelissa87 and peteharverson October 19, 2018 15:12

peteharverson approved these changes Oct 22, 2018

View reviewed changes

walterra approved these changes Oct 22, 2018

View reviewed changes

alvarezmelissa87 reviewed Oct 22, 2018

View reviewed changes

alvarezmelissa87 approved these changes Oct 22, 2018

View reviewed changes

jgowdyelastic merged commit 4df3ee8 into elastic:feature/file-datavisualizer Oct 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adding better error reporting for reading and importing data #24269

[ML] Adding better error reporting for reading and importing data #24269

jgowdyelastic commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

jgowdyelastic commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

peteharverson left a comment

peteharverson Oct 22, 2018

peteharverson Oct 22, 2018

peteharverson Oct 22, 2018

jgowdyelastic Oct 22, 2018

walterra left a comment

alvarezmelissa87 Oct 22, 2018

alvarezmelissa87 Oct 22, 2018

alvarezmelissa87 Oct 22, 2018

alvarezmelissa87 Oct 22, 2018

elasticmachine commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

alvarezmelissa87 left a comment

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

[ML] Adding better error reporting for reading and importing data #24269

[ML] Adding better error reporting for reading and importing data #24269

Conversation

jgowdyelastic commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

💔 Build Failed

jgowdyelastic commented Oct 19, 2018

elasticmachine commented Oct 19, 2018

💔 Build Failed

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 22, 2018

💔 Build Failed

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

jgowdyelastic commented Oct 22, 2018

jgowdyelastic commented Oct 22, 2018