Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IO operations "from/load" to the LazyFrame implementation #451

Merged
merged 3 commits into from
Dec 16, 2022

Conversation

philss
Copy link
Member

@philss philss commented Dec 16, 2022

This is related to #227

In the first PR for that issue, #202, we discussed a possible lazy: true option to control which backend to use. I end up using both :backend and :lazy to make it easier for clients to choose the backend. This adds the possibility for a backend to be implemented both using an eager (the default) and lazy version. WDYT?

I also choose to "delegate" the load of binaries to the eager implementation, and then converting it to lazy. This is mostly for convenience, because there is not such loaders in the Polars backend.

The idea is to avoid validations for those options in functions that
accept backend selection.
@cigrainger
Copy link
Member

I love the direction. Definitely like having both :backend and :lazy opts. Yeah I think that also makes sense regarding loading binaries. It's in memory already anyway so you don't lose much compared to loading from file. Definitely should document that clearly though.

@philss
Copy link
Member Author

philss commented Dec 16, 2022

Definitely should document that clearly though.

@cigrainger You mean about the :lazy option or the usage of the eager by the load_ functions?
Anyway, I'm going to add documentation in a future PR.

@philss philss merged commit dd575f0 into elixir-explorer:main Dec 16, 2022
@philss philss deleted the ps-lazy-io branch December 16, 2022 20:10
@philss philss mentioned this pull request Dec 17, 2022
44 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants