Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
✨ Read GeoTIFF files from remote urls via object_store (#5)
* ➕ Add object_store A generic object store interface for uniformly interacting with AWS S3, Google Cloud Storage, Azure Blob Storage and local files! Also added the url crate. * ➕ Add tokio An event-driven, non-blocking I/O platform for writing asynchronous I/O backed applications! * 🧪 Add unit test to read GeoTIFF from remote http url Doing test driven development by trying to load a remote GeoTIFF file via http into a numpy array. * ✨ Implement reading of GeoTIFF from remote url via object_store Allow passing remote URLs to the read_geotiff_py function. URLs are handled via object_store's parse_url function. Data is read asynchronously into an in-memory buffer (within a tokio runtime), and passed into the synchronous io::geotiff::read_geotiff function. Also added a short example in the Python docstring on how this can be used. * 👔 Handle local filepaths by prepending with file:// Handle the RelativeUrlWithoutBase parsing error raised when local filepaths are given without `file://` in front. Also moved the parse url logic out of the async block. * 💚 Build wheels on manylinux_2_28 Resolve `warning: [email protected]: #error "ARM assembler must define __ARM_ARCH"` on aarch64. Xref PyO3/maturin-action#222 * 🔧 Try using ghcr.io/rust-cross/manylinux_2_28-cross docker image Getting an error like `/usr/bin/python3: No module named pip` when using the quay.io/pypa/manylinux_2_28_x86_64:latest docker image, so trying with ghcr.io/rust-cross/manylinux_2_28-cross instead. Note that the aarch64/armv7/s390x/ppc64le targets use this rust-cross image by default. * 🗑️ Remove x86 (i686) build target on linux Don't compile on i686 since manylinux_2_28 doesn't support it according to https://github.com/rust-cross/manylinux-cross/blob/1c46d3b27a55eff53d675bf515d6361d7cc1abde/README.md?plain=1#L25-L35 and https://github.com/pypa/manylinux/blob/dbe0aa5c374c5b69769253d2a77820b08fe80a7d/README.rst?plain=1#L46. Note that x86 is an alias of i686 (see https://github.com/PyO3/maturin-action/blob/60d11847b29f81ca5375519a8eb33cc336ba4bfa/src/index.ts#L119). * ♻️ Use Url::from_file_path to add file:// to filepath Use built-in function at https://docs.rs/url/2.5.0/url/struct.Url.html#method.from_file_path to add the file:// prefix instead of the DIY method at f9607f3. Hoping that this would handle Windows paths too. Also changed .expect to .map_err to raise PyValueError instead of panicking. * 📝 Document installation and usage in main README.md Add instructions on how to install cog3pio in Rust or Python (need to pull development version from git for now), and sample code to read a GeoTIFF file from a http url. Included some tips on pinning the specific commit hash, and mentioned that the current crate/library only supports reading single-band float32 GeoTIFF files. Also checked the box that reading from remote storage is implemented. * ♻️ Refactor to remove unwrap statements and better error handling Gracefully handle errors when retrieving the GeoTIFF data in the async block, using turbofish operator as suggested by https://rust-lang.github.io/async-book/07_workarounds/02_err_in_async_blocks.html. Fixed some errors missing the format! macro, and added unit tests to catch parsing and decoding errors. In the read_geotiff function, changed the function to return a TiffError instead of ShapeError, so that ? can be used in the decoding portions. * 📝 Copy example usage to crate-level docs as doctest Copy the Rust code from 15b0dbc to src/lib.rs as a doctest. Need to change `crate-type = ["rlib"]` in Cargo.toml so that `cargo test --doc` works, but will configure that in CI later.
- Loading branch information
