Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JobManager: store joblist on http url as Parquet #571

Closed
2 tasks done
VincentVerelst opened this issue Jun 11, 2024 · 3 comments
Closed
2 tasks done

JobManager: store joblist on http url as Parquet #571

VincentVerelst opened this issue Jun 11, 2024 · 3 comments
Assignees
@VincentVerelst
Labels
feature request

Comments

@VincentVerelst
Copy link
Collaborator

VincentVerelst commented Jun 11, 2024 •
edited
Loading

Open-EO/openeo-gfmap#107

  • Support Parquet
  • Support URL
@VincentVerelst VincentVerelst self-assigned this Jun 11, 2024
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 11, 2024
@VincentVerelst
Changed job tracker file format from CSV to Parquet Open-EO#571
8a04438
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 14, 2024
@VincentVerelst
MultiBackendJobManager no longer converts job_tracker geometry column…
be598e3 
… to shapely.BaseGeometry but keeps it as string Open-EO#571
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 14, 2024
@VincentVerelst
added support for Artifactory URL in job tracker, next to PosixPath O…
be88f07 
…pen-EO#571
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 17, 2024
@VincentVerelst
added unit tests for is_url and url_exists Open-EO#571
cc97ff0
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 18, 2024
@VincentVerelst
edited unit tests for test_url_exists Open-EO#571
3ba0de4
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 18, 2024
@VincentVerelst
When resuming job_tracker from existing URL, first download the parqu…
eb3bb87 
…et before reading it Open-EO#571
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 18, 2024
@VincentVerelst
Added unit tests for MultiBackEndJobManager._resume_df() Open-EO#571
6839daf
VincentVerelst added a commit to VincentVerelst/openeo-python-client that referenced this issue Jun 19, 2024
@VincentVerelst
added more unit tests Open-EO#571
743b6d6
VincentVerelst added a commit that referenced this issue Jun 24, 2024
@VincentVerelst
Created JobTrackerStorage class to handle the resuming and storing of…
585460f 
… the job tracker as CSV file. #571
VincentVerelst added a commit that referenced this issue Jun 24, 2024
@VincentVerelst
Modified JobTrackerStorage().normalize_df to not convert geometry col…
ff28b80 
…umn from string to shapely geometry #571
VincentVerelst added a commit that referenced this issue Jun 24, 2024
@VincentVerelst
added extra unit tests for JobTrackerStorage #571
305ac2a
@VincentVerelst VincentVerelst mentioned this issue Jun 24, 2024
Merged
VincentVerelst added a commit that referenced this issue Jul 3, 2024
@VincentVerelst
added output_file as an argument of JobTrackerStorage class #571
fd5f071
VincentVerelst added a commit that referenced this issue Jul 9, 2024
@VincentVerelst
Added support for Parquet file in JobDatabaseStorage #571
4f114f8
VincentVerelst added a commit that referenced this issue Jul 9, 2024
@VincentVerelst
added pyarrow to test dependencies #571
16630e9
VincentVerelst added a commit that referenced this issue Jul 9, 2024
@VincentVerelst
Provived read/write helper classes for CSV and Parquet in JobDatabase…
61bf103 
…Storage and added the option for a custom read/write helper #571
VincentVerelst added a commit that referenced this issue Jul 9, 2024
@VincentVerelst
Made JobDatabaseStorage private #571
0957b52
VincentVerelst added a commit that referenced this issue Jul 9, 2024
@VincentVerelst
fix unit tests #571
5da029b
VincentVerelst added a commit that referenced this issue Jul 10, 2024
@VincentVerelst
Implemented JobDataBaseInterface plus support for CSV and Parquet files
7b992b5 
#571
VincentVerelst added a commit that referenced this issue Jul 15, 2024
@VincentVerelst
removed unnecessary imports in test suite #571
c699f3c
VincentVerelst added a commit that referenced this issue Jul 15, 2024
@VincentVerelst
formatting #571
af35be9
VincentVerelst added a commit that referenced this issue Jul 16, 2024
@VincentVerelst
Added parsing of wkt strings in CsvJobDataBase #571
2da7721
soxofaan added a commit that referenced this issue Jul 17, 2024
@soxofaan
Merge pull request #580 from Open-EO/issue571-job-tracker-storage
ba19dda 
Issue #571 Support Parquet storage in MultiBackendJobManager
@soxofaan
Copy link
Member

just merged #580

@VincentVerelst
Copy link
Collaborator Author

@soxofaan , I've made a gist on how I'd implement URL support in the MultiBackendJobManager: https://gist.github.com/VincentVerelst/aef96f03b5b45a3506d8945730af8221
It would require a user to always provide a url_read_write_helper if they provide a URL. We could foresee maybe some default helpers for CSV and Parquet on Artifactory?

Could you already have a look and leave your comments, before I start the full implementation?

soxofaan added a commit that referenced this issue Jul 19, 2024
@soxofaan
Issue #571 start supporting custom job databases in MultiBackendJobMa…
4b19919 
…nager
@soxofaan soxofaan mentioned this issue Jul 19, 2024
Merged
@soxofaan
Copy link
Member

As a next step I was more thinking like PR #595

soxofaan added a commit that referenced this issue Jul 23, 2024
@soxofaan
Issue #571 start supporting custom job databases in MultiBackendJobMa…
cdc6661 
…nager
soxofaan added a commit that referenced this issue Jul 23, 2024
@soxofaan
Issue #571/#595 make JobDatabaseInterface an abstract base class
1ce55e5
soxofaan added a commit that referenced this issue Jul 23, 2024
@soxofaan
Issue #571/#595 basic docs for JobDatabaseInterface feature
37c5139
soxofaan added a commit that referenced this issue Jul 23, 2024
@soxofaan
fixup! Issue #571/#595 basic docs for JobDatabaseInterface feature
a906340
soxofaan added a commit that referenced this issue Jul 23, 2024
@soxofaan
Issue #571 improve reconstruction of geometries in CsvJobDatabase/Par…
4b8593c 
…quetJobDatabase
soxofaan added a commit that referenced this issue Jul 24, 2024
@soxofaan
Issue #599/#571 Make geopandas optional dependency again
f7ee552 
improve docs around that too
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VincentVerelst VincentVerelst

Labels
feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@soxofaan @VincentVerelst