Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Lake][SWE] Cleanup tables. Query them directly via DataStore, or create registry. #772

Closed
2 tasks done
idiom-bytes opened this issue Mar 11, 2024 · 1 comment
Closed
2 tasks done

[Lake][SWE] Cleanup tables. Query them directly via DataStore, or create registry. #772

idiom-bytes opened this issue Mar 11, 2024 · 1 comment
Assignees
@kdetry
Labels
Type: Enhancement New feature or request

Comments

@idiom-bytes
Copy link
Member

idiom-bytes commented Mar 11, 2024 •
edited by kdetry
Loading

Motivation

GQLDataFactory + ETL should clean up Tables further.
Right now, they are instantiated and cached in a way that is not very clean and leads to poor SWE.

Towards a solution

  1. Remove Tables() from GQL + ETL such that they don't have to be cached in the object.
  2. Move Tables() to a registry or keep abstracting them further such that they can be queried/accessed from anywhere.
  3. Leverage CSVDataStore and PersistentDataStore as singletons to improve how this is handled

DoD

  • There is no more duplication of Tables inside of GQL + ETL
  • Cleanup etl.py do_sync_step, and other objects that are using Tables
@idiom-bytes idiom-bytes added the Type: Enhancement New feature or request label Mar 11, 2024
@idiom-bytes idiom-bytes mentioned this issue Mar 11, 2024
Closed
50 tasks
@kdetry kdetry self-assigned this Mar 12, 2024
@kdetry kdetry mentioned this issue Mar 12, 2024
Merged
idiom-bytes added a commit that referenced this issue Mar 12, 2024
@kdetry @idiom-bytes
Fix #772 - TableRegistry Implementation (#781)
9623300 
* issue772-TableRegistry

* issue772 - Test fixes

* issue772 - Test fixes

* Cleaned up table registry in etl.py. Also updated black, so am pushing latest formatting

* adjusted black to meet cicd requirements, as we're getting different formatting outputs

---------

Co-authored-by: idiom-bytes <[email protected]>
@idiom-bytes
Copy link
Member Author

Registry, and queries have been updated/address.
Ticket completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kdetry kdetry

Labels
Type: Enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kdetry @idiom-bytes