-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620
Conversation
Pinging @elastic/kibana-gis (Team:Geo) |
Pinging @elastic/ml-ui (:ml) |
@elasticmachine merge upstream |
#92504 is resolved by changes made to this PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
x-pack/plugins/file_upload/public/components/import_complete_view.tsx
Outdated
Show resolved
Hide resolved
x-pack/plugins/file_upload/public/importer/geojson_importer/geojson_importer.ts
Outdated
Show resolved
Hide resolved
@nreese Overall working really smoothly. Great to see GeoJSON Upload handling large files! A few questions/comments: Does this PR need a more recent merged version of master or is the number doubling still an issue? Note the Looks like we're handling files with mixed types differently than before (albeit one of them always failed before). Looks like it defaults incorrectly to geo point now? I've attached the file used below for testing. In a perfect world we should get a point, a zig-zag line and a square. Or if the square really is invalid, then just the first two. Old left, new right: We were previously lowercasing the index & index pattern name to save users the hassle of having to do it when ingesting a file with mixed casing in the name. Might be worth doing a |
Needs a more recent version of master. That has been fixed but is not in this branch yet.
Have you tried indexing the file as geo_shape? |
Sorry, I should've been more clear. I'm just referring to the default type-detection settings which determine if the file is geo point or geo shape upfront. The picture above shows how each are ingested by default without the user overriding any settings. |
ok, I have updated the logic for auto selecting geo field type. If a file contains both points and shapes then geo_shape is selected. If a file contains just points then geo_point is selected. With this logic in place, the file loads the same as in previous versions, without the user having to configure geo field type. |
I'm unable to upload the attached file. The progress indicator stays at 0 until it eventually times out. Following the timeout, it progresses to the results screen showing a bad gateway error but also allowing the user to still Then it ends up in a strange state where there are no source details but there is a layer. ne_10m_urban_areas_landscan.zip FWIW, I tested the file in mapshaper.org and it does appear to be valid but with a lot of intersections. Regardless though, we might want to handle it with something other than a 502 if possible and ensure the user can't add it as a layer. |
Discussed offline with @aaronjcaldwell about ne_10m_urban_areas_landscan.zip. This file is causing loaders.js problems. Loaders can not cleanly iterator over file without problems. Fixing loaders.js should be done outside of this PR. There was a problem identified where when in an error state, the wizard incorrectly allowed users to advance to adding the layer. This issue has been resolved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing feedback. This is a long overdue improvement to Maps' file upload capabilities. Also great to have alignment with ML so we can hopefully all benefit from each other's work from here forward. Very excited to see where we can take it from here! 🍾
LGTM w/ green CI!
- code review
- tested locally in chrome
…d:maxFileSize and increase max geojson upload size to 1GB (elastic#92620) * rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize * add saved object migration * file preview * importing status * remove console statement * import complete view * fix geojson_importer test * tslint * i18n fixes * cleanup * update documenation for advanced setting rename * advanced settings usage_collection * remove ml:fileDataVisualizerMaxFileSize from schemas and types * add copy buttons for import response and fix geojson upload functional tests * tslint * remove clipboard-read check * return early if env does not support reading from clipboard * fix reporting tests * review feedback * update GeoJsonFileSource to support showing results trimmed icon and tooltip * add fileUpload to useMlKibana context and replace dependencyCache with useMlKibana * tslint * review feedback * lower case file name * default to selecting geo_shape when file contains both points and shapes * fix wizard onError callback to not advance to next step Co-authored-by: Kibana Machine <[email protected]>
…d:maxFileSize and increase max geojson upload size to 1GB (#92620) (#93358) * rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize * add saved object migration * file preview * importing status * remove console statement * import complete view * fix geojson_importer test * tslint * i18n fixes * cleanup * update documenation for advanced setting rename * advanced settings usage_collection * remove ml:fileDataVisualizerMaxFileSize from schemas and types * add copy buttons for import response and fix geojson upload functional tests * tslint * remove clipboard-read check * return early if env does not support reading from clipboard * fix reporting tests * review feedback * update GeoJsonFileSource to support showing results trimmed icon and tooltip * add fileUpload to useMlKibana context and replace dependencyCache with useMlKibana * tslint * review feedback * lower case file name * default to selecting geo_shape when file contains both points and shapes * fix wizard onError callback to not advance to next step Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>
* master: (45 commits) Add outcome of node scripts/build_api_docs (elastic#93399) [Lens] fix long field name on field stats panel doesn't wrap (elastic#93279) [Bug] Fix filter creation for numeric scripted fields in Discover (elastic#93224) [uptime] Fix anomaly alert edit (elastic#93025) Consolidate @babel/* packages and use latest compatible version (elastic#93264) [Search Embeddable] Add highlighting when searching (elastic#93178) [APM] Add missing bottom border to header (elastic#93179) [CI] No longer collect APM span stack traces (elastic#93263) [XY Chart] Fix "No data to display" error when using IP range aggregation to split series (elastic#93024) update generated public api docs API DOCS Step 3/3 (elastic#92929) chore(NA): look for bazel packages on npm_module folder during distributable build (elastic#93262) rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB (elastic#92620) [kbn/optimizer] allow customizing the limits path from the script (elastic#93153) [Alerting][Docs] Adding template for documenting alert and action types (elastic#92830) [jenkins] convert baseline capture job to use tasks (elastic#93288) removing the linked issue in comments from PR (elastic#93303) chore(NA): do not include fs within a storybook build (elastic#93294) [Maps] Update Map extent queries to use bounding box logic for both point and shape queries (elastic#93156) Add searchDuration to EQL and Threshold rules (elastic#93149) ...
💚 Build SucceededMetrics [docs]Module Count
Async chunks
Page load bundle
History
To update your PR or re-run it, just comment with: |
fixes #46376, #60710, #92504
This PR moves advanced setting ml:fileDataVisualizerMaxFileSize to the file_upload plugin and renames the setting to fileUp\load:maxFileSize. There is a migration script to migrate ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize. The setting is now displayed in the general section.
This PR updates geojson upload from a hard coded 50MB to using fileUpload:maxFileSize. This means that geojson upload size will increase from 50MB to 100MB and upto 1GB. Because of this increased file size, geojson importing had to be re-worked so that the entire file is never held in memory at a single time. Chunks of the file are read and indexed in Elasticsearch. The biggest change for users is that they will no longer get to preview the entire file. Instead, they get to preview only the first 10000 features or 3MB, whichever is less.