[Reporting] Fix performance of CSV generation #120309
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tsullivan
added
8.0.0
v7.15.3
v7.16.1
v7.17.0
v8.1.0
(Deprecated) Feature:Reporting
|
Pinging @elastic/kibana-reporting-services (Team:Reporting Services) |
|
Pinging @elastic/kibana-app-services (Team:AppServicesUx) |
dokmic
reviewed
Dec 3, 2021
x-pack/plugins/reporting/server/export_types/csv_searchsource/generate_csv/generate_csv.ts
Outdated
Show resolved
Hide resolved
dokmic
approved these changes
Dec 3, 2021
jloleysens
approved these changes
Dec 3, 2021
There was a problem hiding this comment.
Great work @tsullivan 🍻
Approving to unblock, but I think we should consider adding the suggested perf code change too.
|
|
||
| const asyncGenerateRow = async (dataTableRow: Record<string, any>): Promise<string> => { | ||
| return new Promise((resolve) => { | ||
| setImmediate(() => { |
There was a problem hiding this comment.
Could we add a comment explaining why we are using setImmediate here? Perhaps a link to the to nodejs documentation too.
x-pack/plugins/reporting/server/export_types/csv_searchsource/generate_csv/generate_csv.ts
Outdated
Show resolved
Hide resolved
1 task
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
tsullivan
commented
Dec 3, 2021
| * 1. Partition the synchronous process with fewer partitions, by using | ||
| * the loop counter to call setImmediate only every N amount of rows. | ||
| * Testing is required to see what the best N value for most data will | ||
| * be. |
There was a problem hiding this comment.
I think this is a feasible option. Filed #120426
tsullivan
added a commit
to tsullivan/kibana
that referenced
this pull request
Dec 3, 2021
* [Reporting] Fix performance of CSV generation * use a for loop with 1 operation instead of 3 chained maps * do without the callback * update comment
tsullivan
added a commit
to tsullivan/kibana
that referenced
this pull request
Dec 3, 2021
* [Reporting] Fix performance of CSV generation * use a for loop with 1 operation instead of 3 chained maps * do without the callback * update comment
tsullivan
added a commit
that referenced
this pull request
Dec 4, 2021
tsullivan
added a commit
that referenced
this pull request
Dec 4, 2021
jportner
added
v8.0.0
auto-backport
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 4, 2021
* [Reporting] Fix performance of CSV generation * use a for loop with 1 operation instead of 3 chained maps * do without the callback * update comment
💔 Backport failed
Successful backport PRs will be merged automatically after passing CI. To backport manually run: |
kibanamachine
added a commit
that referenced
this pull request
Dec 4, 2021
* [Reporting] Fix performance of CSV generation * use a for loop with 1 operation instead of 3 chained maps * do without the callback * update comment Co-authored-by: Tim Sullivan <[email protected]>
TinLe
pushed a commit
to TinLe/kibana
that referenced
this pull request
Dec 22, 2021
* [Reporting] Fix performance of CSV generation * use a for loop with 1 operation instead of 3 chained maps * do without the callback * update comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Closes #120275
Problem: CSV Generation is a CPU-intensive process. It was generating rows in a way that would not allow NodeJS to give control to other pending events: it was blocking the event loop.
Solution: This PR changes the implementation to use the "partitioning" strategy to yield control back to the event loop after every CSV row is generated.
Reference: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/#complex-calculations-without-blocking-the-event-loop
Testing
metricbeat*data that contains 806k hitsCPU utilization and event loop delay gets very high when generating a large CSV export, but Kibana is still able to process other events.