[Reporting] Exporting raw data from table-based visualizations

[tsullivan]

Summary

This enhancement is a parallel effort of #17950 to provide better options for exporting raw data from table-based Kibana visualizations, in CSV form.

Numbers and timestamps will be formatted per user configuration (kibana.yml, advanced settings).

Use cases

• Get immediate CSV download of table data in the UI using a panel action
• Schedule CSV export of table data using an HTTP API

Walk through a user interaction

1. Create a saved object. TSVB Data Table visualization is the easiest example (maybe)
2. Add the visualization to a dashboard
3. Use a Panel Action to find ways to export the raw data:
   • Download CSV Data
     • Immediate download
     • Content is available for re-download in Management > Reporting
     • Similar to the raw data download options in "classical" visualizations, but this will be grounded in something that's accessible to integration with other applications
   • Copy POST URL
     • Carrying over this existing functionality will be possible
   • Schedule CSV Data Report
     • This is for much later, but this enhancement will lead into it.
     • Expensive queries are better suited to this, vs immediate download
     • When we have this, it will make sense to get rid of "Copy POST URL" because the URL is used for scheduling

Working on this feature leads into a solution for #18110

[tsullivan]

cc @joelgriffith @AlonaNadler

[AlonaNadler]

TSVB table is one example, mainly because it doesn't have the inspector, the main use case is with saved search within a dashboard.
Ideally, we can use the action panel for every table (TSVB table, saved search and aggregated table), the top priority from my perspective is saved search

[tsullivan]

Thanks for the input @AlonaNadler. We'll look at Saved Search next.

TSVB was a bit easier since all the querying is on the server side.

Aggregated table will be hard with its current implementation, because the saved object stored state has a relational nature with the index pattern object.

With Saved Search, we don't know how many rows of hits there will be, so we use the scan/scroll API to get it from Elasticsearch. I don't think that will be a barrier as we're going to use Server-side APIs to build the CSV - but that explains why there is no Inspector for it.

[AlonaNadler]

There are a few problems customers report:

• Not being able to export a saved search to a CSV from a dashboard
• PDF report doesn't export the full table when there are multiple pages
• Export the raw documents behind a visualization - not like in inspector but the actual original document the aggregation is based on.
• Export CSV of aggregated data behind visualization in TSVB - TSVB doesn't have the inspector, which means you can't export csv of the aggregated data when using TSVB visualization, in all the TSVB charts (line, metric, gauge, table) types (not just table).

When you say "Exporting raw data from table-based visualizations" is it the original document used in the aggregation or the aggregated data visualized in a chart (similar to inspector export to CSV)?

[Damientinker]

Hello there

I would just like to hear if this line

• Schedule CSV export of table data using an HTTP API

Is covering using watcher to generate the CSV file (similar to the one found in inspect on a data table visulalization (not the TSVB version))

Atm, the .PDF generation is not optimal, since it only shows the first page =)

[tsullivan]

TSVB table is one example, mainly because it doesn't have the inspector, the main use case is with saved search within a dashboard.

I'm pivoting this issue to saved search within a dashboard. It turns out that we have all the utilities we need on the server side to build the query based on the search saved object data.

Getting the aggregated table export for CSV will be hard to do in this design, because parsing aggregated table saved object data relies on a lot of services built in Angular.

Export CSV of aggregated data behind visualization in TSVB - TSVB doesn't have the inspector, which means you can't export csv of the aggregated data when using TSVB visualization, in all the TSVB charts (line, metric, gauge, table) types (not just table).

I think it will be best to hold off on the original idea to enable CSV export for TSVB tables, so we can focus on releasing the saved search export from Dashboard. But soon after this, we can work on CSV of aggregated data behind TSVB visualization. Assuming the data is 2-dimensional, we can get it into CSV.

When you say "Exporting raw data from table-based visualizations" is it the original document used in the aggregation or the aggregated data visualized in a chart (similar to inspector export to CSV)?

For exporting search, it is close to the original document - only the columns to show have to be set in the saved object data (or state has to be passed to the API as free-form data, which we'll allow to override the state that's in the saved object). For visualizations that aggregate data, "raw" means the bucketed aggregation metrics in the ES response.

[tsullivan]

• Schedule CSV export of table data using an HTTP API
• Is covering using watcher to generate the CSV file (similar to the one found in inspect on a data table visulalization (not the TSVB version))

My understanding of the Watcher integration of alerting is that it allows you to configure a "webhook action" that can POST a data payload to an HTTP host. (Please correct me if I'm wrong) This will be possible with the new API in the works, but it's also possible today with the "POST URL" you see shown in the Kibana UI for the Share > Reporting actions menu.

[bhavyarm]

cc @kunisen thanks!

[gkalangara]

This a very important feature for us.
We have a payed version and still not able to export data automatically.

[tsullivan]

could include: #36546

[tsullivan]

this is stalled as the current implementation of Data Table visualization depends on Angular modules (AggConfigs & tabifyAggResponse), that we can't access from the server-side for automation.

[tsullivan]

Related issue: Table data visualizations in PDFs

#17950

[Artiizan]

Would also love to see "Schedule CSV export of table data using an HTTP API" be implemented to allow generation of Kibana Visualizations as CSV using a watcher.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

Update: I have heard from @jimgoodwin in a sync meeting that the visualizations have separated the data modeling logic from the rendering logic. That should get us closer to this goal, but I'm not sure by how much.

cc @elastic/kibana-app-arch

[tsullivan]

Per #30982 (comment), this issue is blocked on App team making a source of data modeling logic that can be called from automation.

[fbaligand]

I'm very interested by this feature.
Numerous clients in my company have asked for this feature since years.
=> I would love this feature!

[lireanne]

I too would love this feature!

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[eskdera]

This feature would be very useful for us as well!

[carlopuri]

Please go on deploying this feature... very very loved feature

[Singh-Shivani]

Hi, is it deployed? how can I use it? @tsullivan @timroes

[dcolazin]

Moreover, it would be nice to inspect the TSVB-generated visualizations, in a similarly to what it is possible to do with other vizs.

[dyangel16]

Please go on deploying this feature, This feature would be very useful for us as well,want automate some reports using HTTP API!

[vani-hiver]

I've been following this feature from quite a while now. Would really like to see this deployed.

[juan-domenech]

FYI onlookers: The CSV export option is available under Inspect for Data Table Visualisations.

Visualize Library -> New visualization -> Data Table -> Inspect -> Download CSV

[dej611]

+1 https://discuss.elastic.co/t/exporting-data-from-a-visualization/320976/4

[drewdaemon]

+1 https://discuss.elastic.co/t/increase-the-lens-max-results/345601/

[teresaalvarezsoler]

@tsullivan isn't this possible already via Inspect? I wouldn't make the main download action from the panel menu more complex adding additional options.

[tsullivan]

Hi @teresaalvarezsoler this issue is to use the Reporting framework to run the export, which will involve server-side APIs. That is the only way that retrieving the CSV can be automated.

I wouldn't make the main download action from the panel menu more complex adding additional options.

We have a redesigned share modal that we can use for this: #189143