[Reporting/Discover] include the document's entire set of fields #92730

tsullivan · 2021-02-24T22:42:11Z

Summary

In 7.12, Discover may return an empty set as the list of fields selected, whereas in previous versions we would get an array with _source as a single element.

This PR revives the expected behavior when Discover triggers a CSV export and there are no columns selected for the search.

During development of this PR and testing Download CSV from a saved search Dashboard panel (different app):

A set of fields from the index pattern is not available on the server side. If no fields are selected, Download CSV would only include the auto-populated timeField column. Or if there is no timeField, it could possibly return an empty file as the CSV export
Meta fields (such as keyword sub-values of text fields) are not available when downloading the CSV from a dashboard panel.

The above issues will be fixed in 7.13 with #88303

New functional tests were added to verify the expected behavior in Discover. These tests were simply copied over from #88303

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

… as columns in the saved search

elasticmachine · 2021-02-24T22:43:14Z

Pinging @elastic/kibana-app-services (Team:AppServices)

tsullivan · 2021-02-24T22:44:33Z

src/plugins/discover/public/application/helpers/get_sharing_data.ts

@@ -19,7 +19,10 @@ const getSharingDataFields = async (
  timeFieldName: string,
  hideTimeColumn: boolean
 ) => {
-  if (selectedFields.length === 1 && selectedFields[0] === '_source') {
+  if (
+    selectedFields.length === 0 ||


This fixes the bug that was immediately noticed: only the date field is populated when exporting a new search (if no other fields are selected as columns)

tsullivan · 2021-02-24T22:44:48Z

x-pack/plugins/reporting/server/export_types/csv_from_savedobject/execute_job.ts


-    logger.debug(`Execute job generating [${visType}] csv`);
+    logger.debug(`Execute job generating saved search CSV`);


logging cleanup: the visType variable was undefined

tsullivan · 2021-02-24T22:45:32Z

x-pack/plugins/reporting/server/export_types/csv_from_savedobject/lib/get_csv_job.ts

@@ -75,6 +77,14 @@ export const getGenerateCsvParams = async (
    fields: indexPatternFields,
  } = indexPatternSavedObject;

+  if (!indexPatternFields || indexPatternFields.length === 0) {


logging addition for the open issues with Download CSV from a dashboard panel (to be fixed in #88303)

tsullivan · 2021-02-24T22:46:10Z

x-pack/test/functional/page_objects/reporting_page.ts

-      const fromTime = 'Sep 19, 2015 @ 06:31:44.000';
-      const toTime = 'Sep 19, 2015 @ 18:01:44.000';
+      const fromTime = 'Apr 27, 2019 @ 23:56:51.374';
+      const toTime = 'Aug 23, 2019 @ 16:18:51.821';


These values have been wrong for a long time, but in this PR it matters because the content of the CSV is actually checked against a snapshot.

tsullivan · 2021-02-24T22:58:53Z

@elasticmachine merge upstream

wylieconlon

I tested this change and found this fixes the issue and doesn't affect _source fetching. The new tests look safe to me.

wylieconlon · 2021-02-24T23:02:55Z

x-pack/test/functional/apps/discover/reporting.ts

@@ -69,14 +69,83 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
        await PageObjects.reporting.setTimepickerInDataRange();
        await PageObjects.discover.saveSearch('my search - with data - expectReportCanBeCreated');
        await PageObjects.reporting.openCsvReportingPanel();
-        expect(await PageObjects.reporting.canReportBeCreated()).to.be(true);


Got it, so we were previously testing that reporting is available, but not the contents of the reports.

tsullivan · 2021-02-25T02:01:27Z

@elasticmachine merge upstream

tsullivan · 2021-02-25T16:54:09Z

@elasticmachine merge upstream

wylieconlon · 2021-02-25T19:00:02Z

@tsullivan you have actual type errors in the code, it's not flaky

kibanamachine · 2021-02-25T21:45:19Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	401.0KB	401.0KB	+27.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…stic#92730) * fix get_sharing_data to provide a list of fields if none are selected as columns in the saved search * add logging for download CSV that fields must be selected as columns * add more functional test coverage * fix ts in test Co-authored-by: Kibana Machine <[email protected]>

) (#92922) * fix get_sharing_data to provide a list of fields if none are selected as columns in the saved search * add logging for download CSV that fields must be selected as columns * add more functional test coverage * fix ts in test Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

) (#92921) * fix get_sharing_data to provide a list of fields if none are selected as columns in the saved search * add logging for download CSV that fields must be selected as columns * add more functional test coverage * fix ts in test Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

…bana into task-manager/docs-monitoring * 'task-manager/docs-monitoring' of github.com:gmmorris/kibana: (40 commits) [Security Solution][Case][Bug] Improve case logging (elastic#91924) [Alerts][Doc] Added README documentation for alerts plugin status and framework health checks configuration options. (elastic#92761) Add warning for EQL and Threshold rules if exception list contains value list items (elastic#92914) [Security Solution][Case] Fix subcases bugs on detections and case view (elastic#91836) [APM] Always allow access to Profiling via URL (elastic#92889) [Vega] Allow image loading without CORS policy by changing the default to crossOrigin=null (elastic#91991) skip flaky suite (elastic#92114) [APM] Fix for default fields in correlations view (elastic#91868) (elastic#92090) chore(NA): bump bazelisk to v1.7.5 (elastic#92905) [Maps] fix selecting EMS basemap does not populate input (elastic#92711) API docs (elastic#92827) [kbn/test] add import/export support to KbnClient (elastic#92526) Test fix management scripted field filter functional test and unskip it (elastic#92756) [App Search] Create Curation view/functionality (elastic#92560) [Reporting/Discover] include the document's entire set of fields (elastic#92730) [Fleet] Add new index to fleet for artifacts being served out of fleet-server (elastic#92860) [Alerts][Doc] Added README documentation for API key invalidation configuration options. (elastic#92757) [Discover][docs] Add search for relevance (elastic#90611) [Alerts][Docs] Extended README.md and the user docs with the licensing information. (elastic#92564) [7.12][Telemetry] Security telemetry allowlist fix. (elastic#92850) ...

tsullivan added 3 commits February 24, 2021 15:21

fix get_sharing_data to provide a list of fields if none are selected…

1265ecd

… as columns in the saved search

add logging for download CSV that fields must be selected as columns

32d542c

add more functional test coverage

f520f50

tsullivan requested review from a team and wylieconlon February 24, 2021 22:42

tsullivan added v7.12.0 v7.13.0 v8.0.0 (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes Team:AppServices labels Feb 24, 2021

tsullivan commented Feb 24, 2021

View reviewed changes

Merge branch 'master' into fix/92712

b1206b6

wylieconlon approved these changes Feb 24, 2021

View reviewed changes

Merge branch 'master' into fix/92712

6d57fd4

Merge branch 'master' into fix/92712

9a96ae3

tsullivan added 2 commits February 25, 2021 12:47

Merge branch 'master' into fix/92712

fbaa0b9

fix ts in test

c97a14e

tsullivan merged commit fd348d3 into elastic:master Feb 25, 2021

tsullivan deleted the fix/92712 branch February 25, 2021 21:47

tsullivan mentioned this pull request Feb 25, 2021

[7.x] [Reporting/Discover] include the document's entire set of fields (#92730) #92921

Merged

tsullivan mentioned this pull request Feb 25, 2021

[7.12] [Reporting/Discover] include the document's entire set of fields (#92730) #92922

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting/Discover] include the document's entire set of fields #92730

[Reporting/Discover] include the document's entire set of fields #92730

tsullivan commented Feb 24, 2021 •

edited

Loading

elasticmachine commented Feb 24, 2021

tsullivan Feb 24, 2021

tsullivan Feb 24, 2021

tsullivan Feb 24, 2021

tsullivan Feb 24, 2021

tsullivan commented Feb 24, 2021

wylieconlon left a comment

wylieconlon Feb 24, 2021

tsullivan commented Feb 25, 2021

tsullivan commented Feb 25, 2021

wylieconlon commented Feb 25, 2021

kibanamachine commented Feb 25, 2021


		logger.debug(`Execute job generating [${visType}] csv`);
		logger.debug(`Execute job generating saved search CSV`);

[Reporting/Discover] include the document's entire set of fields #92730

[Reporting/Discover] include the document's entire set of fields #92730

Conversation

tsullivan commented Feb 24, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Feb 24, 2021

tsullivan Feb 24, 2021

Choose a reason for hiding this comment

tsullivan Feb 24, 2021

Choose a reason for hiding this comment

tsullivan Feb 24, 2021

Choose a reason for hiding this comment

tsullivan Feb 24, 2021

Choose a reason for hiding this comment

tsullivan commented Feb 24, 2021

wylieconlon left a comment

Choose a reason for hiding this comment

wylieconlon Feb 24, 2021

Choose a reason for hiding this comment

tsullivan commented Feb 25, 2021

tsullivan commented Feb 25, 2021

wylieconlon commented Feb 25, 2021

kibanamachine commented Feb 25, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

tsullivan commented Feb 24, 2021 •

edited

Loading