Skip to content
This repository has been archived by the owner on Jul 25, 2022. It is now read-only.

Add custom global allocator #30

Merged
merged 2 commits into from
Feb 25, 2022
Merged

Add custom global allocator #30

merged 2 commits into from
Feb 25, 2022

Conversation

matthewmturner
Copy link
Contributor

@matthewmturner matthewmturner commented Feb 23, 2022 •
edited
Loading

Closes #27

@matthewmturner
Copy link
Contributor Author

@Dandandan FYI

@Dandandan
Copy link
Contributor

Did you also try mimalloc (see https://github.com/pola-rs/polars/blob/master/py-polars/Cargo.toml#L22). Should give similar results, might be faster or slower on some queries. Still interested which one is the best to pick.

houqp
houqp reviewed Feb 23, 2022
View reviewed changes
Cargo.toml Outdated
@@ -35,6 +35,7 @@ datafusion = { version = "^7.0.0", features = ["pyarrow"] }
datafusion-expr = { version = "^7.0.0" }
datafusion-common = { version = "^7.0.0", features = ["pyarrow"] }
uuid = { version = "0.8", features = ["v4"] }
snmalloc-rs = {version = "0.2", features = ["cache-friendly"]}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the cache-friendly feature has been dropped from upstream in newer releases I believe.

@matthewmturner
Copy link
Contributor Author

mimalloc results below - faster on almost all queries except Q8 and Q10. This may be the better option.

q1: 0.0366870829999999
q2: 0.32823987499999996
q3: 1.1415230410000001
q4: 0.028246750000000098
q5: 1.1575570830000004
q6: 1.2152348750000006
q7: 1.1023409590000002
q8: 2.9869192910000004
q9: 0.5675542499999988
q10: 25.437209041

@matthewmturner matthewmturner changed the title Add sn-malloc as global allocator Add custom global allocator Feb 23, 2022
@matthewmturner
Copy link
Contributor Author

@Dandandan to confirm - I haven't updated PR with mimalloc yet. Did you approve for snmalloc or mimalloc?

@Dandandan
Copy link
Contributor

Ah I didn't catch that 😅. I think mimalloc would be a good choice to go forward with. I think it will be interesting in the future to keep experimenting with this (e.g. there is an upcoming smalloc 2 version).

@matthewmturner
Copy link
Contributor Author

Ok sounds good! Will update.

@jimexist jimexist merged commit 368b50e into datafusion-contrib:main Feb 25, 2022
@jonmmease jonmmease mentioned this pull request May 19, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@houqp houqp houqp left review comments

@jimexist jimexist jimexist approved these changes

@Dandandan Dandandan Dandandan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use custom allocator in Python build
4 participants
@matthewmturner @Dandandan @jimexist @houqp