Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Load regex support #513

Merged
merged 23 commits into from
Dec 12, 2022
Merged

feat: Load regex support #513

merged 23 commits into from
Dec 12, 2022

Conversation

gaurav274
Copy link
Member

@gaurav274 gaurav274 commented Dec 4, 2022 •
edited
Loading

  1. Add regex support to load video commands. LOAD VIDEO "/path/to/files/*.mp4" INTO MyVidoes;
  2. Add support to persist the loaded file names in RDBMS (SQLite DB)
  3. Extensive error handling to prevent loading duplicate file paths, corrupt files, invalid file paths, etc.
  4. We roll back in case a load fails in between. This helps avoid an ugly table state.
  5. Current implementation fails on the first error. In the future, we can add an optional flag to let the user control what to do in such cases, skip is one option supported in other DBs.
  6. Upload followed by LOAD is no longer supported.
  7. Remove _row_id from the select *
  8. Fix Drop command - delay binding to executor so that we can gracefully handle the DROP IF EXISTS case.
  9. Fix Rename Command - disable rename on the structured table. Sqlaclhemy does not let easily rename table. Not an important query to support right now.

gaurav274 and others added 18 commits November 16, 2022 12:08
@gaurav274
Merge branch 'master' of github.com:georgia-tech-db/eva
a923620
@gaurav274
feat: image support init
316bca9
@gaurav274
feat: LOAD cmd clean up + storage Engine moved to factory pattern
9faf6a4
@gaurav274
feat: checkpoint
791d21b
@jiashenC
attempt to fix row_id
62f5c83
@jiashenC
reformat
d6cc244
@gaurav274
checkpoint
1ef34cf
@gaurav274
Merge branch 'populate-row-id' of github.com:georgia-tech-db/eva into…
2efaaae 
… img-support
@gaurav274
feat: regex support with rollback
1788c71
@gaurav274
Merge branch 'master' of github.com:georgia-tech-db/eva
e77d9b2
@gaurav274
feat: support regex and roll_back on failure
e84bbc9
@gaurav274
fix: undo import image code
c959c0b
@gaurav274
fix: code clean up
03a92ee
@gaurav274
bug: add missing data files
436b93b
@gaurav274
test: fix old test cases
49ad2ed
@gaurav274
feat: drop table clean up
e40dc06
@gaurav274
bug: remove rename support for sturtured data
3f5d131
@gaurav274
test: remove unnecessary test cases
613689f
jiashenC
jiashenC reviewed Dec 6, 2022
View reviewed changes
eva/executor/rename_executor.py Outdated
CatalogManager().rename_table(self.node.new_name, self.node.old_table.table)
try:
obj = self.node.old_table.table.table_obj
print(obj)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra print here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will remove it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will remove it

eva/parser/drop_statement.py
super().__init__(StatementType.DROP)
self._table_refs = table_refs
self._table_infos = table_infos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just trying to understand the difference here. Why is this changed to TableInfo from TableRef?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is that drop will always be on a table, so it is unnecessary to use the abstraction TableRef which can be a join, subquery etc

xzdandy
xzdandy approved these changes Dec 12, 2022
View reviewed changes
eva/storage/opencv_storage_engine.py
raise RuntimeError(str(e))
except Exception as e:
logger.exception(str(e))
raise RuntimeError(str(e))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate error handler?

@gaurav274 gaurav274 merged commit 6b46e69 into master Dec 12, 2022
@gaurav274 gaurav274 deleted the load_regex_support branch December 12, 2022 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiashenC jiashenC jiashenC left review comments

@xzdandy xzdandy xzdandy approved these changes

@jarulraj jarulraj Awaiting requested review from jarulraj

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gaurav274 @xzdandy @jiashenC