Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Profiling tool: Scale to run large number of event logs. #3046

Closed
nartal1 opened this issue Jul 27, 2021 · 0 comments · Fixed by #3161
Closed

[FEA] Profiling tool: Scale to run large number of event logs. #3046

nartal1 opened this issue Jul 27, 2021 · 0 comments · Fixed by #3161
Assignees
@tgravescs
Labels
feature request New feature or request P1 Nice to have for release tools

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Jul 27, 2021

Is your feature request related to a problem? Please describe.
As a user, I want to be able to process 100 event logs after filtering.

Additional context
This needs to refactor the profiling tool. Such as no more Spark SQL Dataframe operations for aggregations.

Acceptance criteria: Able to process all 100+ NDS queries within 1 hour.
@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify P1 Nice to have for release labels Jul 27, 2021
@tgravescs tgravescs removed the ? - Needs Triage Need team to review and classify label Jul 27, 2021
@tgravescs tgravescs self-assigned this Jul 27, 2021
@tgravescs tgravescs mentioned this issue Aug 6, 2021
Merged
@sameerz sameerz added the tools label Aug 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgravescs tgravescs

Labels
feature request New feature or request P1 Nice to have for release tools
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Rework Profile tool to not require Spark to run and process files faster tgravescs/spark-rapids
3 participants
@tgravescs @sameerz @nartal1