[ML] File datavisualizer initial commit #22828

jgowdyelastic · 2018-09-07T17:00:18Z

Initial commit of the File Datavisualizer.
This allows the user to upload a file which is passed to the new find_file_structure endpoint.
The results are then displayed on the page.

The page can be found here:
<kibana address>/app/ml#/filedatavisualizer
The default max upload size is pretty small. I added this to my kibana.dev.yml file:
server.maxPayloadBytes: 104857600 # 100MB

elasticmachine · 2018-09-07T17:08:52Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-07T17:19:53Z

Pinging @elastic/ml-ui

elasticmachine · 2018-09-10T08:19:38Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-10T16:29:08Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-10T17:08:26Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-11T10:25:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-11T10:34:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

peteharverson

Initial comments. Looks good so far!

peteharverson · 2018-09-11T11:09:52Z

x-pack/plugins/ml/public/file_datavisualizer/components/fields_stats/_field_stats_card.scss

@@ -0,0 +1,88 @@
+.card-container {


Could any of these settings be used in the existing index based datavisualizer once that is converted to sass? Just wondering if some of these should be moved to a higher level scss file?

yes, i imagine there will be common rules which can be moved to a shared location once the big sass conversion has happened

peteharverson · 2018-09-11T11:11:08Z

x-pack/plugins/ml/public/file_datavisualizer/components/fields_stats/field_stats_card.js

+          <div className="card-contents">
+            <div className="stats">
+              <div className="stat">
+                <i className="fa fa-files-o" aria-hidden="true" /> {field.count} documents ({percent}%)


Should add a check for 1 document here and switch documents to document

peteharverson · 2018-09-11T11:11:53Z

x-pack/plugins/ml/public/file_datavisualizer/components/fields_stats/field_stats_card.js

+                <i className="fa fa-files-o" aria-hidden="true" /> {field.count} documents ({percent}%)
+              </div>
+              <div className="stat">
+                <i className="fa fa-cubes" aria-hidden="true" /> {field.cardinality} distinct values


As above - check cardinality and output values or value

peteharverson · 2018-09-11T11:14:36Z

x-pack/plugins/ml/public/file_datavisualizer/components/fields_stats/field_stats_card.js

+  );
+}
+
+function getTypeColor(type) {


Can this be done in css instead, using the same styles as used in /ml/public/components/field_data_card/styles/main.less ?

yes it can and makes the code simpler.

peteharverson · 2018-09-11T11:15:05Z

x-pack/plugins/ml/public/file_datavisualizer/components/fields_stats/fields_stats.js

+    super(props);
+
+    this.fields = createFields(this.props.results);
+    console.log(this.fields);


Did you mean to leave this console.log in?

yes, i'm leaving logging in as it's a work in progress

peteharverson · 2018-09-11T11:34:33Z

x-pack/plugins/ml/public/file_datavisualizer/components/summary/summary.js

+function createDisplayItems(results) {
+  const items = [
+    {
+      title: 'Number of lines analysed',


Should be analyzed!

peteharverson · 2018-09-11T11:35:44Z

x-pack/plugins/ml/server/models/file_data_visualizer/file_data_visualizer.js

+import fs from 'fs';
+import os from 'os';
+const util = require('util');
+// const readFile = util.promisify(fs.readFile);


Is this commented out line still needed?

i'm leaving this commented code in as this is a work in progress and will eventually be used.

peteharverson · 2018-09-11T11:36:09Z

x-pack/plugins/ml/server/models/file_data_visualizer/file_data_visualizer.js

+const writeFile = util.promisify(fs.writeFile);
+
+export function fileDataVisualizerProvider(callWithRequest) {
+  async function analyseFile(data) {


Should be analyzeFile for consistency with DataVisualizer I think, and the route which is analyzeFile

peteharverson · 2018-09-11T11:36:58Z

x-pack/plugins/ml/server/models/file_data_visualizer/file_data_visualizer.js

+    }
+  }
+
+  // async function cacheData2(data) {


Just checking you want to keep this commented out code for now?

peteharverson · 2018-09-11T11:38:36Z

x-pack/plugins/ml/server/routes/file_data_visualizer.js

+import { wrapError } from '../client/errors';
+import { fileDataVisualizerProvider } from '../models/file_data_visualizer';
+
+function analyseFiles(callWithRequest, data) {


To analyzeFiles ?

This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6.

elasticmachine · 2018-09-11T12:20:15Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

walterra · 2018-09-11T12:32:06Z

...l/public/file_datavisualizer/components/file_datavisualizer_view/file_datavisualizer_view.js

+
+      if (data !== null) {
+        try {
+          const resp = await  ml.analyseFile(data);


total nit, double space

walterra

LGTM at this stage - further on might be worth looking into reducing redundancy with original data visualizer.

elasticmachine · 2018-09-11T13:20:16Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

peteharverson

Couple of minor comments. Otherwise all your changes LGTM.

peteharverson · 2018-09-11T14:03:33Z

...l/public/file_datavisualizer/components/file_datavisualizer_view/file_datavisualizer_view.js

+        <div style={{ textAlign: 'center' }} >
+          <EuiFilePicker
+            id="filePicker"
+            initialPromptText="Select or drag and drop a log file"


I wonder if this should just say 'Select or drag and drop a file'? Do we need the word 'log' in there?

peteharverson · 2018-09-11T14:05:25Z

x-pack/plugins/ml/public/jobs/jobs_list/components/ml_job_editor/ml_job_editor.js

-export function MLJobEditor({ value, height = '500px', width = '100%', mode = 'json', readOnly = false, onChange = () => {} }) {
+export const EDITOR_MODE = { TEXT: 'text', JSON: 'json' };
+
+export function MLJobEditor({ value, height = '500px', width = '100%', mode = EDITOR_MODE.JSON, readOnly = false, onChange = () => {} }) {


These should be split over multiple lines to help readability I think

elasticmachine · 2018-09-11T14:49:06Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

jgowdyelastic · 2018-09-11T16:19:46Z

retest

elasticmachine · 2018-09-11T17:56:24Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

* [ML] File datavisualizer initial commit (elastic#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (elastic#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (elastic#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (elastic#23500) * [ML] Changes to timestamp formats (elastic#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (elastic#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (elastic#23763) * [ML] Adding license check to datavisualizer landing page (elastic#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (elastic#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (elastic#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (elastic#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (elastic#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (elastic#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (elastic#24387) * [ML] Adding ability to override various settings when importing data (elastic#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test

sophiec20 added the :ml label Sep 7, 2018

jgowdyelastic requested review from peteharverson and walterra September 11, 2018 10:28

jgowdyelastic self-assigned this Sep 11, 2018

jgowdyelastic added the review label Sep 11, 2018

peteharverson reviewed Sep 11, 2018

View reviewed changes

jgowdyelastic added 8 commits September 11, 2018 13:11

[ML] File datavisualizer initial commit

3c008ec

removing mocked data and adding initial stats

57a1fd4

adding card styling to fields

3c0c69c

58da202

Revert "". accidentally added with no commit message

4abbeed

This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6.

adding date type to timestamp field

ccfe844

renaming FileStats to FieldsStats

28231d9

code clean up

9bd18a7

jgowdyelastic force-pushed the file-datavisualizer-initial-commit branch from 1ca213b to 9bd18a7 Compare September 11, 2018 12:12

jgowdyelastic changed the base branch from feature/file-datavisualizer to master September 11, 2018 12:25

jgowdyelastic changed the base branch from master to feature/file-datavisualizer September 11, 2018 12:25

walterra reviewed Sep 11, 2018

View reviewed changes

walterra approved these changes Sep 11, 2018

View reviewed changes

changes based on review

f704f48

peteharverson approved these changes Sep 11, 2018

View reviewed changes

changes to error handling

bfea643

jgowdyelastic merged commit 0ffb032 into elastic:feature/file-datavisualizer Sep 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] File datavisualizer initial commit #22828

[ML] File datavisualizer initial commit #22828

jgowdyelastic commented Sep 7, 2018 •

edited

Loading

elasticmachine commented Sep 7, 2018

elasticmachine commented Sep 7, 2018

elasticmachine commented Sep 10, 2018

elasticmachine commented Sep 10, 2018

elasticmachine commented Sep 10, 2018

elasticmachine commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

peteharverson left a comment

peteharverson Sep 11, 2018

jgowdyelastic Sep 11, 2018

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

jgowdyelastic Sep 11, 2018

peteharverson Sep 11, 2018

jgowdyelastic Sep 11, 2018

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

jgowdyelastic Sep 11, 2018

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

elasticmachine commented Sep 11, 2018

walterra Sep 11, 2018

walterra left a comment

elasticmachine commented Sep 11, 2018

peteharverson left a comment

peteharverson Sep 11, 2018

peteharverson Sep 11, 2018

elasticmachine commented Sep 11, 2018

jgowdyelastic commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

[ML] File datavisualizer initial commit #22828

[ML] File datavisualizer initial commit #22828

Conversation

jgowdyelastic commented Sep 7, 2018 • edited Loading

elasticmachine commented Sep 7, 2018

💔 Build Failed

elasticmachine commented Sep 7, 2018

elasticmachine commented Sep 10, 2018

💔 Build Failed

elasticmachine commented Sep 10, 2018

💔 Build Failed

elasticmachine commented Sep 10, 2018

💔 Build Failed

elasticmachine commented Sep 11, 2018

💔 Build Failed

elasticmachine commented Sep 11, 2018

💔 Build Failed

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 11, 2018

💔 Build Failed

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 11, 2018

💔 Build Failed

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 11, 2018

💔 Build Failed

jgowdyelastic commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

💚 Build Succeeded

jgowdyelastic commented Sep 7, 2018 •

edited

Loading