rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB

docs/management/advanced-options.asciidoc

@@ -83,6 +83,10 @@ specific dashboard, application, or saved object as they enter each space.
 [[fields-popularlimit]]`fields:popularLimit`::
 The top N most popular fields to show.
 
+[[fileupload-maxfilesize]]`fileUpload:maxFileSize`::
+Sets the file size limit when importing files. The default
+value is `100MB`. The highest supported value for this setting is `1GB`.
+
 [[filtereditor-suggestvalues]]`filterEditor:suggestValues`::
 Set this property to `false` to prevent the filter editor from suggesting values
 for fields.
@@ -257,7 +261,7 @@ Hides the "Time" column in *Discover* and in all saved searches on dashboards.
 Highlights results in *Discover* and saved searches on dashboards. Highlighting
 slows requests when working on big documents.
 
-[[doctable-legacy]]`doc_table:legacy`:: 
+[[doctable-legacy]]`doc_table:legacy`::
 Controls the way the document table looks and works. Set this property to `true` to revert to the legacy implementation.
 
 [[discover-searchFieldsFromSource]]`discover:searchFieldsFromSource`::
@@ -281,10 +285,6 @@ must contain `from` and `to` values (see
 {ref}/common-options.html#date-math[accepted formats]). It is ignored unless
 `ml:anomalyDetection:results:enableTimeDefaults` is enabled.
 
-[[ml-filedatavisualizermaxfilesize]]`ml:fileDataVisualizerMaxFileSize`::
-Sets the file size limit when importing data in the {data-viz}. The default
-value is `100MB`. The highest supported value for this setting is `1GB`.
-
 
 [float]
 [[kibana-notification-settings]]

docs/user/ml/index.asciidoc

@@ -20,10 +20,10 @@ image::user/ml/images/ml-data-visualizer-sample.jpg[{data-viz} for sample flight
 experimental[] You can also upload a CSV, NDJSON, or log file. The *{data-viz}*
 identifies the file format and field mappings. You can then optionally import
 that data into an {es} index. To change the default file size limit, see
-<<kibana-ml-settings,Machine learning advanced settings>>.
+<<kibana-general-settings, fileUpload:maxFileSize advanced settings>>.
 
 If {stack-security-features} are enabled, users must have the necessary
-privileges to use {ml-features}. Refer to 
+privileges to use {ml-features}. Refer to
 {ml-docs}/setup.html#setup-privileges[Set up {ml-features}].
 
 NOTE: There are limitations in {ml-features} that affect {kib}. For more information, refer to {ml-docs}/ml-limitations.html[Machine learning].
@@ -40,15 +40,15 @@ false positives. {anomaly-detect-cap} runs in and scales with {es}, and
 includes an intuitive UI on the {kib} *Machine Learning* page for creating
 {anomaly-jobs} and understanding results.
 
-If you have a license that includes the {ml-features}, you can 
+If you have a license that includes the {ml-features}, you can
 create {anomaly-jobs} and manage jobs and {dfeeds} from the *Job Management*
-pane: 
+pane:
 
 [role="screenshot"]
 image::user/ml/images/ml-job-management.png[Job Management]
 
-You can use the *Settings* pane to create and edit 
-{ml-docs}/ml-calendars.html[calendars] and the filters that are used in 
+You can use the *Settings* pane to create and edit
+{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
 {ml-docs}/ml-rules.html[custom rules]:
 
 [role="screenshot"]
@@ -69,13 +69,13 @@ occurring in your operational environment at that time:
 image::user/ml/images/ml-annotations-list.png[Single Metric Viewer with annotations]
 
 In some circumstances, annotations are also added automatically. For example, if
-the {anomaly-job} detects that there is missing data, it annotates the affected 
-time period. For more information, see 
-{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The 
+the {anomaly-job} detects that there is missing data, it annotates the affected
+time period. For more information, see
+{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
 *Job Management* pane shows the full list of annotations for each job.
 
-NOTE: The {kib} {ml-features} use pop-ups. You must configure your web 
-browser so that it does not block pop-up windows or create an exception for your 
+NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
+browser so that it does not block pop-up windows or create an exception for your
 {kib} URL.
 
 For more information about the {anomaly-detect} feature, see
@@ -89,7 +89,7 @@ experimental[]
 
 The Elastic {ml} {dfanalytics} feature enables you to analyze your data using
 {classification}, {oldetection}, and {regression} algorithms and generate new
-indices that contain the results alongside your source data. 
+indices that contain the results alongside your source data.
 
 If you have a license that includes the {ml-features}, you can create
 {dfanalytics-jobs} and view their results on the *Data Frame Analytics* page in
@@ -98,5 +98,5 @@ If you have a license that includes the {ml-features}, you can create
 [role="screenshot"]
 image::user/ml/images/outliers.png[{oldetection-cap} results in {kib}]
 
-For more information about the {dfanalytics} feature, see 
-{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
+For more information about the {dfanalytics} feature, see
+{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].

src/core/server/ui_settings/saved_objects/migrations.test.ts

@@ -44,3 +44,37 @@ describe('ui_settings 7.9.0 migrations', () => {
     });
   });
 });
+
+describe('ui_settings 7.13.0 migrations', () => {
+  const migration = migrations['7.13.0'];
+
+  test('returns doc on empty object', () => {
+    expect(migration({} as SavedObjectUnsanitizedDoc)).toEqual({
+      references: [],
+    });
+  });
+  test('properly renames ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize', () => {
+    const doc = {
+      type: 'config',
+      id: '8.0.0',
+      attributes: {
+        buildNum: 9007199254740991,
+        'ml:fileDataVisualizerMaxFileSize': '250MB',
+      },
+      references: [],
+      updated_at: '2020-06-09T20:18:20.349Z',
+      migrationVersion: {},
+    };
+    expect(migration(doc)).toEqual({
+      type: 'config',
+      id: '8.0.0',
+      attributes: {
+        buildNum: 9007199254740991,
+        'fileUpload:maxFileSize': '250MB',
+      },
+      references: [],
+      updated_at: '2020-06-09T20:18:20.349Z',
+      migrationVersion: {},
+    });
+  });
+});

src/core/server/ui_settings/saved_objects/migrations.ts

@@ -28,4 +28,23 @@ export const migrations = {
     }),
     references: doc.references || [],
   }),
+  '7.13.0': (doc: SavedObjectUnsanitizedDoc<any>): SavedObjectSanitizedDoc<any> => ({
+    ...doc,
+    ...(doc.attributes && {
+      attributes: Object.keys(doc.attributes).reduce(
+        (acc, key) =>
+          key === 'ml:fileDataVisualizerMaxFileSize'
+            ? {
+                ...acc,
+                ['fileUpload:maxFileSize']: doc.attributes[key],
+              }
+            : {
+                ...acc,
+                [key]: doc.attributes[key],
+              },
+        {}
+      ),
+    }),
+    references: doc.references || [],
+  }),
 };

src/plugins/kibana_usage_collection/server/collectors/management/schema.ts

@@ -73,6 +73,7 @@ export const stackManagementSchema: MakeSchemaFrom<UsageStats> = {
   'discover:sort:defaultOrder': { type: 'keyword' },
   'context:step': { type: 'long' },
   'accessibility:disableAnimations': { type: 'boolean' },
+  'fileUpload:maxFileSize': { type: 'keyword' },
   'ml:fileDataVisualizerMaxFileSize': { type: 'keyword' },
   'ml:anomalyDetection:results:enableTimeDefaults': { type: 'boolean' },
   'ml:anomalyDetection:results:timeDefaults': { type: 'keyword' },

src/plugins/kibana_usage_collection/server/collectors/management/types.ts

@@ -73,6 +73,7 @@ export interface UsageStats {
   'discover:sort:defaultOrder': string;
   'context:step': number;
   'accessibility:disableAnimations': boolean;
+  'fileUpload:maxFileSize': string;
   'ml:fileDataVisualizerMaxFileSize': string;
   'ml:anomalyDetection:results:enableTimeDefaults': boolean;
   'ml:anomalyDetection:results:timeDefaults': string;

src/plugins/telemetry/schema/oss_plugins.json

@@ -4235,6 +4235,9 @@
         "accessibility:disableAnimations": {
           "type": "boolean"
         },
+        "fileUpload:maxFileSize": {
+          "type": "keyword"
+        },
         "ml:fileDataVisualizerMaxFileSize": {
           "type": "keyword"
         },

x-pack/plugins/file_upload/common/constants.ts

@@ -5,6 +5,8 @@
  * 2.0.
  */
 
+export const UI_SETTING_MAX_FILE_SIZE = 'fileUpload:maxFileSize';
+
 export const MB = Math.pow(2, 20);
 export const MAX_FILE_SIZE = '100MB';
 export const MAX_FILE_SIZE_BYTES = 104857600; // 100MB

x-pack/plugins/file_upload/kibana.json

@@ -4,5 +4,6 @@
   "kibanaVersion": "kibana",
   "server": true,
   "ui": true,
-  "requiredPlugins": ["data", "usageCollection"]
+  "requiredPlugins": ["data", "usageCollection"],
+  "requiredBundles": ["kibanaReact"]
 }

x-pack/plugins/file_upload/public/api/index.ts

@@ -12,6 +12,8 @@ import type { IImporter, ImportFactoryOptions } from '../importer';
 export interface FileUploadStartApi {
   getFileUploadComponent(): Promise<React.ComponentType<FileUploadComponentProps>>;
   importerFactory(format: string, options: ImportFactoryOptions): Promise<IImporter | undefined>;
+  getMaxBytes(): number;
+  getMaxBytesFormatted(): string;
 }
 
 export async function getFileUploadComponent(): Promise<

x-pack/plugins/file_upload/public/components/geojson_file_picker.tsx

@@ -0,0 +1,160 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React, { Component } from 'react';
+import { Feature } from 'geojson';
+import { EuiFilePicker, EuiFormRow } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+import { MB } from '../../common';
+import { getMaxBytesFormatted } from '../get_max_bytes';
+import { validateFile } from '../importer';
+import { GeoJsonImporter, GEOJSON_FILE_TYPES } from '../importer/geojson_importer';
+
+interface Props {
+  onSelect: ({
+    features,
+    geoFieldTypes,
+    importer,
+    indexName,
+  }: {
+    features: Feature[];
+    indexName: string;
+    importer: GeoJsonImporter;
+    geoFieldTypes: string[];
+  }) => void;
+  onClear: () => void;
+}
+
+interface State {
+  error: string | null;
+  isLoadingPreview: boolean;
+  previewSummary: string | null;
+}
+
+export class GeoJsonFilePicker extends Component<Props, State> {
+  private _isMounted = false;
+
+  state: State = {
+    error: null,
+    isLoadingPreview: false,
+    previewSummary: null,
+  };
+
+  async componentDidMount() {
+    this._isMounted = true;
+  }
+
+  componentWillUnmount() {
+    this._isMounted = false;
+  }
+
+  _onFileSelect = (files: FileList | null) => {
+    this.props.onClear();
+
+    this.setState({
+      error: null,
+      isLoadingPreview: false,
+      previewSummary: null,
+    });
+
+    if (files && files.length) {
+      this._loadFilePreview(files[0]);
+    }
+  };
+
+  async _loadFilePreview(file: File) {
+    this.setState({ isLoadingPreview: true });
+
+    let importer: GeoJsonImporter | null = null;
+    let previewError: string | null = null;
+    let preview: {
+      features: Feature[];
+      geoFieldTypes: string[];
+      previewCoverage: number;
+    } | null = null;
+    try {
+      validateFile(file, GEOJSON_FILE_TYPES);
+      importer = new GeoJsonImporter(file);
+      preview = await importer.previewFile(10000, MB * 3);
+      if (preview.features.length === 0) {
+        previewError = i18n.translate('xpack.fileUpload.geojsonFilePicker.noFeaturesDetected', {
+          defaultMessage: 'No GeoJson features found in selected file.',
+        });
+      }
+    } catch (error) {
+      previewError = error.message;
+    }
+
+    if (!this._isMounted) {
+      return;
+    }
+
+    this.setState({
+      error: previewError,
+      isLoadingPreview: false,
+      previewSummary:
+        !previewError && preview
+          ? i18n.translate('xpack.fileUpload.geojsonFilePicker.previewSummary', {
+              defaultMessage: 'Previewing {numFeatures} features, {previewCoverage}% of file.',
+              values: {
+                numFeatures: preview.features.length,
+                previewCoverage: preview.previewCoverage,
+              },
+            })
+          : null,
+    });
+
+    if (importer && preview) {
+      this.props.onSelect({
+        ...preview,
+        importer,
+        indexName: file.name.split('.')[0],
+      });
+    }
+  }
+
+  _renderHelpText() {
+    return this.state.previewSummary !== null ? (
+      this.state.previewSummary
+    ) : (
+      <span>
+        {i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedFormats', {
+          defaultMessage: 'Formats accepted: {fileTypes}',
+          values: { fileTypes: GEOJSON_FILE_TYPES.join(', ') },
+        })}
+        <br />
+        {i18n.translate('xpack.fileUpload.geojsonFilePicker.maxSize', {
+          defaultMessage: 'Max size: {maxFileSize}',
+          values: { maxFileSize: getMaxBytesFormatted() },
+        })}
+        <br />
+        {i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedCoordinateSystem', {
+          defaultMessage: 'Coordinates must be in EPSG:4326 coordinate reference system.',
+        })}
+      </span>
+    );
+  }
+
+  render() {
+    return (
+      <EuiFormRow
+        isInvalid={!!this.state.error}
+        error={!!this.state.error ? [this.state.error] : []}
+        helpText={this._renderHelpText()}
+      >
+        <EuiFilePicker
+          initialPromptText={i18n.translate('xpack.fileUpload.geojsonFilePicker.filePicker', {
+            defaultMessage: 'Select or drag and drop a file',
+          })}
+          onChange={this._onFileSelect}
+          accept={GEOJSON_FILE_TYPES.join(',')}
+          isLoading={this.state.isLoadingPreview}
+        />
+      </EuiFormRow>
+    );
+  }
+}