Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reporting] generate csv from new api for saved object ids #31188

Conversation

tsullivan
Copy link
Member

@tsullivan tsullivan commented Feb 14, 2019

Summary

This PR explores a way of generating CSV using a new API that uses simple params in the query string and meaninful params in a POST body.

Usage

  1. Create a TSVB Table
  2. Call the API manually:
    POST /api/reporting/v1/generate/<endpoint>/<saved object id>
    {
      "timerange": {
        "timezone": "UTC",
        "min": "1972-01-01T07:00:00.000Z",
        "max": "2015-01-01T06:59:59.999Z"
      },
      "state": {}
    }
    

New API endpoints:

  • /api/reporting/v1/generate/immediate/csv/saved-object/<id>
    • For on-demand download, e.g from panel action
    • CSV content and text/csv content-type header in the response
  • /api/reporting/v1/generate/csv/saved-object/<id>:
    • For scheduling, similar to many Reporting integrations today.

For now, this can probably only support TSVB tables. Future expansion is possible, starting with visualization types that query / model their data on the server side.

Screenshot

image

TODO:

Release notes: none. This is an internal-only API until it is able to support more types of saved objects.

@tsullivan tsullivan added enhancement New value added to drive a business result (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead labels Feb 14, 2019
@tsullivan
Copy link
Member Author

cc @joelgriffith

@tsullivan tsullivan force-pushed the feature/reporting/csv-export-panel-action branch 3 times, most recently from d00adb1 to 92e15fa Compare February 15, 2019 05:27
@tsullivan tsullivan force-pushed the feature/reporting/csv-export-panel-action branch from 92e15fa to 33781b0 Compare February 15, 2019 05:32
@joelgriffith
Copy link
Contributor

Sorry to let this fall by the wayside, I'll take a gander on Monday morning! Hope is to get this in, then #30895 so we can use the types you've got!

@tsullivan
Copy link
Member Author

@joelgriffith not a huge rush on this! I think
#30895 is probably more needed

@elastic elastic deleted a comment from elasticmachine Feb 19, 2019
@elastic elastic deleted a comment from elasticmachine Feb 19, 2019
@tsullivan
Copy link
Member Author

tsullivan commented Feb 20, 2019

Ok, I'm not so sure I want to take on a fix for sorting in this PR. Currently in master, TSVB is showing odd things for sort:

image
image
image

If TSVB table sorting is all happening on the client side, this API will have to be another layer of data modeling to sort the result data. Also, it will need to get the sorting options in the state object in POST params


const generateCsv = createGenerateCsv(logger);
return async function executeJob(job: JobDocPayload): Promise<JobDocOutputPseudo> {
const { basePath, objects, headers: serializedEncryptedHeaders, jobParams } = job; // FIXME how to remove payload.objects for cleanup?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joelgriffith
Copy link
Contributor

I'm going to be pulling/playing with this branch today, stay tuned

@elastic elastic deleted a comment from elasticmachine Feb 28, 2019
@elastic elastic deleted a comment from elasticmachine Feb 28, 2019
@tsullivan
Copy link
Member Author

I'm going to be pulling/playing with this branch today, stay tuned

Cool! I think this is in usable shape for saved search by ID

docvalue_fields: [],
query: {
bool: {
filter: [],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: use kibanaSavedObjectMeta.searchSourceJSON to get the filters

@elastic elastic deleted a comment from elasticmachine Mar 1, 2019

export const WHITELISTED_JOB_CONTENT_TYPES = [
'application/json',
'application/pdf',
'text/csv',
CONTENT_TYPE_CSV,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason it's uppercased here? Not sure if there's a semantic reason... my guess is not

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's also exported as a const and used elsewhere

@joelgriffith
Copy link
Contributor

This looks good, and seems to be working well in my panel work. LGTM

@elastic elastic deleted a comment from elasticmachine Mar 2, 2019
@elastic elastic deleted a comment from elasticmachine Mar 2, 2019
@tsullivan tsullivan force-pushed the feature/reporting/csv-export-panel-action branch from fe8c4fe to 97eb031 Compare March 2, 2019 01:46
@elastic elastic deleted a comment from elasticmachine Mar 2, 2019
@tsullivan tsullivan force-pushed the feature/reporting/csv-export-panel-action branch from 26077df to d08767a Compare March 2, 2019 02:12
@tsullivan
Copy link
Member Author

@joelgriffith sorry I missed your review. I rewinded other changes I made back to the point that you reviewed

@tsullivan tsullivan merged commit 3d82ec9 into elastic:feature/reporting/csv-export-panel-action Mar 2, 2019
@elasticmachine
Copy link
Contributor

💔 Build Failed

@tsullivan
Copy link
Member Author

This PR was missing a file that was referenced. It was actually from a file rename. I pushed the fix in this commit: cc2a451940

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead enhancement New value added to drive a business result review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants