Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT-#6398: Improved performance of list-like objects insertion into HDK DataFrames #6412

Merged
merged 6 commits into from
Jan 26, 2024
Merged

FEAT-#6398: Improved performance of list-like objects insertion into HDK DataFrames #6412

merged 6 commits into from
Jan 26, 2024

Conversation

AndreyPavlenko
Copy link
Collaborator

@AndreyPavlenko AndreyPavlenko commented Jul 24, 2023 •
edited
Loading

What do these changes do?

If the partition contains either pandas DataFrame or pyarrow Table, insert the object directly into the frame/table, otherwise create a single-column frame and join the frames by rowid.

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Improve performance of list-like objects insertion into DataFrames  #6398
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

@AndreyPavlenko AndreyPavlenko force-pushed the issue-6398 branch 6 times, most recently from 6dfdfc4 to 6e33d36 Compare July 27, 2023 22:26
@AndreyPavlenko AndreyPavlenko force-pushed the issue-6398 branch 6 times, most recently from 9d6575b to eb7ade3 Compare July 29, 2023 19:47
@AndreyPavlenko AndreyPavlenko mentioned this pull request Aug 9, 2023
Merged
7 tasks
@AndreyPavlenko AndreyPavlenko force-pushed the issue-6398 branch 3 times, most recently from 2dfdf73 to 48f2a81 Compare September 25, 2023 21:07
@AndreyPavlenko AndreyPavlenko marked this pull request as ready for review September 25, 2023 22:25
@AndreyPavlenko AndreyPavlenko requested review from a team as code owners September 25, 2023 22:25
@AndreyPavlenko AndreyPavlenko mentioned this pull request Sep 28, 2023
Closed
7 tasks
@AndreyPavlenko AndreyPavlenko changed the title FEAT-#6398: Improved performance of list-like objects insertion into DataFrames FEAT-#6398: Improved performance of list-like objects insertion into HDK DataFrames Jan 22, 2024
anmyachev
anmyachev previously approved these changes Jan 24, 2024
View reviewed changes
@anmyachev
Copy link
Collaborator

@dchigarev any other comments?

anmyachev
anmyachev previously approved these changes Jan 24, 2024
View reviewed changes
@YarShev
Copy link
Collaborator

YarShev commented Jan 26, 2024

@anmyachev, @dchigarev, can we merge this?

@anmyachev
Copy link
Collaborator

@YarShev let's merge after #6887 if @dchigarev agree with that changes

dchigarev
dchigarev previously approved these changes Jan 26, 2024
View reviewed changes
@anmyachev
Copy link
Collaborator

@AndreyPavlenko please rebase

AndreyPavlenko and others added 6 commits January 26, 2024 19:25
@AndreyPavlenko
FEAT-modin-project#6398: Improved performance of list-like objects in…
944c818 
…sertion into HDK DataFrames

If the partition contains either pandas DataFrame or pyarrow Table,
insert the object directly into the frame/table, otherwise create
a single-column frame and join the frames by rowid.

Signed-off-by: Andrey Pavlenko <[email protected]>
@AndreyPavlenko @dchigarev @anmyachev
Apply suggestions from code review
5ce1156 
Co-authored-by: Dmitry Chigarev <[email protected]>
Co-authored-by: Anatoly Myachev <[email protected]>
@AndreyPavlenko
Apply suggestions from code review
0d48cd8
@AndreyPavlenko @anmyachev
Update modin/experimental/core/execution/native/implementations/hdk_o…
ab03652 
…n_native/dataframe/dataframe.py

Co-authored-by: Anatoly Myachev <[email protected]>
@AndreyPavlenko
Apply suggestions from code review
372267a
@AndreyPavlenko
Apply suggestions from code review
2e73aea
@AndreyPavlenko AndreyPavlenko dismissed stale reviews from dchigarev and anmyachev via 2e73aea January 26, 2024 18:26
@anmyachev anmyachev merged commit fe19363 into modin-project:master Jan 26, 2024
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anmyachev anmyachev anmyachev approved these changes

@dchigarev dchigarev dchigarev left review comments

@aregm aregm Awaiting requested review from aregm aregm is a code owner

@gshimansky gshimansky Awaiting requested review from gshimansky gshimansky is a code owner

@ienkovich ienkovich Awaiting requested review from ienkovich ienkovich is a code owner

@Garra1980 Garra1980 Awaiting requested review from Garra1980 Garra1980 is a code owner

@YarShev YarShev Awaiting requested review from YarShev YarShev is a code owner

@vnlitvinov vnlitvinov Awaiting requested review from vnlitvinov vnlitvinov is a code owner

@devin-petersohn devin-petersohn Awaiting requested review from devin-petersohn devin-petersohn is a code owner

@mvashishtha mvashishtha Awaiting requested review from mvashishtha mvashishtha is a code owner

@RehanSD RehanSD Awaiting requested review from RehanSD RehanSD is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve performance of list-like objects insertion into DataFrames
4 participants
@AndreyPavlenko @anmyachev @YarShev @dchigarev