Files list end point refactor #1529
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
korgan00
requested changes
Nov 7, 2024
korgan00
reviewed
Nov 7, 2024
|
Thank you so much @korgan00 , as soon as I return from vacations I will take a look 🙏 |
korgan00
requested changes
Nov 20, 2024
|
Apart from the comment from enum I think I addressed all your comments, @korgan00 . Thank you so much for the review, definitely it got quite better the PR. |
korgan00
requested changes
Nov 20, 2024
|
I closed the conversation around the enum assuming is fine for you, @korgan00 👍 |
korgan00
approved these changes
Nov 21, 2024
Merged
Tansito
added a commit
that referenced
this pull request
Jan 13, 2025
* Create services storage script * Files list end point refactor (#1529) * First refactorization for files * Completed use cases for list end-point * Renamed storage to file_storage * Migrated files to file_storage * Migrated get files to the new service * Check if function exists first * Separated end-points to the new provider * Improved tests * Restore original fixtures * Solved the problem with the external if * Added Literal type in the FileStorage service * functions methods refactorization * use self.username instead of the variable * run path sanitization only one time * function title is mandatory for file storage * fix black * Updated 403 by 404 * Updated swagger for list * Makes use of enum with integers * use is instead of equals * unify path build * get_function refactor * Add support new files (#1546) * new list files for the gateway refactor * unify files url * remove enum and linter * remove provider parameter * Check provider * Download end point refactor (#1547) * refactor of download end-points * additional test to check non existing file * renamed files test to v1_files * added additional checks for the query * included the checks in the list end-point * include swagger documentation updated * remove unneeded try except * make use of regex instead of a manual parsing * check not all instead of None * rename file_extension_is_valid * Update gateway/api/utils.py Co-authored-by: Goyo <[email protected]> --------- Co-authored-by: Goyo <[email protected]> * add support to new download files endpoints (#1550) * Gateway/delete end point (#1554) * add support to new delete files endpoints * fix get parameter * fix tests * delete programDelete fixture * update file_storage/remove_file description * Upload end point (#1555) * Update upload end-point * removed unused imports * updated comment in upload end-point * remove file extension limitation from files (#1559) * Client - files upload refactor (#1557) * add support to new upload file endpoints * replace data with params * Client - files delete refactor (#1556) * add support to new delete files endpoints * fix get parameter * fix tests * adapt files delete to the refactor * replace data with params * remove not used fixture * Integration tests fix (#1561) * add support to new delete files endpoints * fix get parameter * fix tests * adapt files delete to the refactor * replace data with params * remove not used fixture * tests fixed * fix client integration * fix context manager in files when download * added a new test for provider end-points * fix black * migrated old tests * Update gateway/api/services/file_storage.py Co-authored-by: Goyo <[email protected]> * remove additional line --------- Co-authored-by: David <[email protected]> * Trace decorator (#1553) * create a trace decorator * fix decorator * change documentation --------- Co-authored-by: David <[email protected]> * Files refactor to include repositories logic (#1560) * remove unneeded provider methods * create access policies file * refactor get_functions repository method * refactor groups and repositories * repository refactor from files * fix some linter problems * fixed a bug when the user retrieves a function * fix lint * refactor of get_function method * remove artifact test file * remove programs access policies * refactor programs references to functions * group repository refactor * rename groups repository into user repository * simplified get_function methods * fix query * adapt get_functions methods * updated comments * create path if doesn't exist * remove some unused code * fix files client * fix typos * fixed the creation of the directory * added a test for the provider end-points * fix some typos from the provider end-points * fix black on tests * files documentation updated (#1565) * Refactor storage in scheduler (#1563) * refactor storage in scheduler * fix typo in ray template * remove unused node_image * use sub-paths for the cluster template * storage absolute path refactor * docstrings updated in the client --------- Co-authored-by: Goyo <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR is part of an effort to stabilize
filesAPI end-points in #1509This PR focuses on
files/andfiles/provider/end-points to be able to list the files in the working directories for the user and for the provider.Details and comments
files/end-point was separated in two:files/now contains the paths:username/andusername/provider/functionwith the purpose to separated files from functions from the user's root folderfiles/provider/to manage the pathprovider/functionso now each function will have its own folder in the provider directory.FileStoragewas created to manage the logic of access to this storage for providers and users with the purpose to simplify the view.