[Reporting] Full data table content must be contained in the pdf generated

[elasticmachine]

Original comment by @tbragin:

Right now only the preview of the data table shows up in the PDF report, because we essentially take a screenshot of the first page of the table. We should consider how to improve this.

Possibilities to discuss:
- Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
- Option to export a full data table as a formatted PDF table (limited to N results, IBM Qradar does max of 500)

This is similar to the problem we have with saved search export LINK REDACTED

[elasticmachine]

Original comment by @tbragin:

cc: @kobelb

[elasticmachine]

Original comment by @tbragin:

We discussed this live and decided on the following approach in the immediate future:

• Export a full data table as a formatted PDF table, limited to N results.
• 500 seems like a reasonable default for N.
• For now, do not offer the option of making N configurable.
• Put these data tables at the end of the PDF, after the charts.

Note, this is what Splunk does as well

Reason for not going with the other option:

• Adding an option to export CSV along with PDF adds the complication of dealing with multiple files (we can zip them up, of course)

[mbarretta]

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

[stacey-gammon]

We discussed this a bit in the Sharing team planning week. Here is some of the notes from that session:

Saved Search and Data Tables: treat them the same.

• Problems:
  • Panel is cut off so page 1 might not show all the rows on the first page.
  • Table might have 10 pages, but only first page. Even if panel shows all rows of the first page, users want to have all 10 pages of data.
• Possible Solutions
  • Show just page 1 table in the pdf, but bundle data table information as .csv
  • Show just page 1 table in the pdf but add tables at the bottom of the pdf, maybe as extra pages.
  • [NO] auto expand rows so all rows show in a screenshot.
    • We don’t want this, it could be way too much information
  • [Best option] Offer an option in report PDF:
  • [Yes/No]: Include zip of csvs, one for each table. Tablename.csv
    • Add dashboard level option, export raw data as csv. rawDataForVis1.csv

[tsullivan]

@mbarretta

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

For saved searches, there is a CSV export option in the "Share" menu when viewing the search in Discover.

There's a gap right now on being able to export CSV from a saved search when viewing it in the Dashboard context though, and we need to fix it.

Providing more convenient access to exporting CSV, for saved search and data table, is a general theme where we could see a lot of wins.

1. Have a POST "Generation URL" to allow exporting data table (aggregated) visualization data to CSV
2. Be able to export CSV from a saved search when viewing it in the Dashboard context

[mbarretta]

@tsullivan I appreciate that the CSV export allows saved search records to be exported, but not exporting everything seen in a dashboard breaks the "promise".

I've seen many dashboards that are essentially text-only: the preference is for raw data over graphics so that they can see everything in one place and avoid the extra click required to go from a visualization to the underlying data. In those cases, the dashboard is really more like a few Discover views glued together.

So yes, users could export the various data, import them into something like Excel, and then export to PDF from there, but I think it's a fair expectation for them to assume if there are 20 rows of records on a dashboard, they will be visible when that dashboard is exported.

[Eniqmatic]

+1 is there any more progress on this or when we will likely see it? We have a dashboard that is data tables with split tables based on how many security attacks have occured. But our PDF reports only show 1 page. Is there any temporary workarounds for this?

[cmagdalena-racowireless]

+1

[tsullivan]

@Eniqmatic The App team is actively working on doing a lot of heavy lifting around how the visualizations get their data, and the fruit of that effort will be that we can get data from a saved visualization on the server-side. That will lead into options on how to implement both of the items from the description:

• Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
• Option to export a full data table as a formatted PDF table

There is a workaround to export CSV from an aggregated table visualization. It's from the UI, so unfortunately it would be hard to automate: you can click the Inspector for the visualization to see the underlying data. On that screen are export controls. See: https://www.elastic.co/guide/en/kibana/6.6/vis-inspector.html

[Eniqmatic]

@tsullivan Excellent sounds like what I am looking for. Hopefully the guys make good progress with that! Thanks for your response.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[mikeh688]

no problem and many thanks

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[tsullivan]

I chatted with @vadimkibana a bit about this today. Ultimately, the dashboard application will need to render itself to support a feature where tables are fully visible when a screenshot is taken, or when we get the PDF from Chromium. There needs to be two enhancements:

1. In "preserve layout" mode (default for generating a PDF in dashboard), we can include additional pages at the end of the PDF for each table panel on the dashboard. Multiple pages would result in the PDF by having the Dashboard app provide multiple "locator params" - one for each page in the PDF. The concept of using multiple locator params in a report job is already used in Kibana for the Canvas application.
2. In "print layout" mode, the custom CSS for screenshot layout, which is under the Dashboard domain, should have a selector for panels with tables inside of them, and ensure the height of those panels is not constrained and the panel doesn't show a scrollbar.

This outline is just to give context on why Vadim and I felt we need to tag the Presentation team on this issue

[ThomThomson]

@tsullivan I'm not sure it's feasible for us to build functionality into the "preserve layout" mode that appends pages onto the report for the contents of each table. When printing a dashboard, that mode is really more of a WYSIWYG sort of experience, where the layout of the dashboard grid is reproduced. I'm not convinced that we should introduce print specific changes to it.

The idea of ensuring the entirety of each panel is shown with no scroll bars in print mode is appealing, and something we could attempt.

[scottf9]

Highly highly request this feature! So many times I get requests to schedule/build pdf reports and this severely limits what we can do.

[mrin9san]

Hi, we are platinum customers of Elastic and request this feature. Its not practical to simply open up elastic account in presentations and show reports. Anyways a report with screenshot of the first page is a useless feature.

[gord-lyr]

+1 on this enhancement. Moving from a competing SIEM product, and can't provide the same level of reporting to our customers.

[petrklapka]

Looping @osmanis, @kevinsweet and @rayafratkina - This feature ask has percolated up several times now here.

[mikeh688]

we'd still like to see this feature; the requirement hasn't gone away (see earlier responses).

[tsullivan]

The feature added in #174511 (8.13) makes it possible to create a CSV report using data from an ES|QL query. It doesn't entirely satisfy these requirements, but it does for the first time make it possible to get a report of an aggregated search.

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[tsullivan]

@elastic/kibana-presentation Hi, this issue pertains to the layout engine of the dashboard, so wouldn't really fall under the SharedUX team, which maintains the service of reporting. What I'd like to do next is add this to a new meta issue that collects other issues involving reporting where dashboard layout is factor of making the feature work better.

Update: created #191532

[ThomThomson]

This is a really interesting one. It's definitely the layout of Dashboard, but it's also the pagination of the table, etc. For instance, even if we limit to 500 rows - how do we do this? Are we calculating the exact pixel size of each row, rendering all rows in one page and spitting out a massive PDF of the Dashboard?

To be honest, I think this issue could be closed. Firing off a CSV export is likely a much better way to get all of the data from a table.

[tsullivan]

To be honest, I think this issue could be closed. Firing off a CSV export is likely a much better way to get all of the data from a table.

I agree with you, @ThomThomson. There is a ton of complexity under this and many open questions about how this could be implemented. If we close this we could instead focus on #30982, which about capturing aggregated table data (not a saved search, which is currently supported) in a CSV report.

[mikeh688]

From a user's perspective, this is the wrong mentality. yep, the data could be pulled as a CSV. However, that is not what customers want. They want to generate pdf reports that are end-user (e.g., management ready) that can be shared.

My suggestion is that you stop saying how difficult this is and figure out how to produce a reporting feature that end users can use and meets their requirements. At the moment, to be frank, its pretty half-baked. Pdf exports are cosmetically attractive until the reality strikes in that any tabular data only contains the first page of results. It is very limiting.

This ticket has been open since 2017 which is 7 years ago. I dont want to sound like a dog with a bone; maybe i'm wrong because I find it strange that there arent more people asking for this. How many customers use pdf export? Maybe that's the issue; maybe they dont. If they dont, maybe its because the functionality is incomplete.

[Heenawter]

cc @teresaalvarezsoler