-
Notifications
You must be signed in to change notification settings - Fork 843
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhancement proposal: refactor elif statements for partitioners in auto.py #1716
Comments
@scanny Do you have any additional thoughts on what this could/should look like? |
I sketched some ideas out in a branch here. It's far from complete, and probably too much of a refactor but some ideas along the same line of thinking |
Oh nice. Is there an open issue that branch is being developed for? I admittedly didn't look for any open Issues related to this, but did just find this one: |
No, there wasn't a specific issue, mostly personal musings over a weekend. It's mildly related to #1521. e.g. reducing the amount of kwargs we're passing around, and long-term simplifying the user experience. |
Fixed by #3806. |
There's a lot of redundancy in auto.py when going through the
elif
statements to check thefiletype
and send the document to the correct partitioner. This proposes collecting all the options users can define into a single object that can be unpacked by each partitioner so partition calls can look something like this instead of having to explicitly pass all the arguments (but still explicitly pass all data source parameters):We could create a
PartitionerOptions(TypedDict)
type that specifies the names and types of all the keyword-value pairs that can appear as options. This could help prevent typos and such and provides general type-safety.The text was updated successfully, but these errors were encountered: