Async flatgeobuf reading via object-store

[kylebarron]

This is a draft for now because HTTPFgbReader is not generic over backends. See flatgeobuf/flatgeobuf#345

[Libbum]

I've been watching your progress on this here and in the flatgeobuf crate. This capability would be super helpful not just for geo-arrow, but for broader applications also.

1. Is there anything in particular I can help you with to finish off this feature?
2. Do you think it would be practical / possible to release this as a separate crate? If not: would it be possible to at least expose these methods for use elsewhere?

[kylebarron]

This is waiting on a new release of flatgeobuf. Hopefully this will just work once that's released.

If you're interested, you could test this branch with the latest main of flatgeobuf, and then verify that it actually works.

2. Do you think it would be practical / possible to release this as a separate crate? If not: would it be possible to at least expose these methods for use elsewhere?

Release what as a separate crate? The only non-geoarrow-specific part of this is object_store_reader.rs which is much too small to release as its own crate IMO. More likely would be to see if http_range_client is interested in a PR to integrate with object-store

[Libbum]

Thanks :) I thought there was still something to finish up due to the todo! in reader_async, but that was a total skim on my part—it's clearly just a return type to be plugged in.

see if http_range_client is interested in a PR to integrate with object-store

Valid point, that sound like a cleaner plan regardless.

Cool, I'll play around with flatgeobuf master and let you know what I find out 🚀

[kylebarron]

it looks like 4.1 was just published, so you should test with that

[Libbum]

range (at least in my testing) looks like bytes=0-12943 - so your split is insufficient.

I've done let split_range = range.split(&['=', '-']).collect::<Vec<_>>(); and then incremented the indexes on line 17, 18.

Still looking, but I'm getting a response with this change at least 🚀

[Libbum]

A refactor including errors:

        let bytes_range = range
            .split(&['=', '-'])
            .filter_map(|n| n.parse::<usize>().ok());
        if let Some((start_range, end_range)) = bytes_range.collect_tuple() {
            // Add one to the range because HTTP range strings are end-inclusive
            let bytes = self
                .reader
                .get_range(&self.location, start_range..end_range + 1)
                .await
                .map_err(|_| HttpError::HttpStatus(500))?;
            return Ok(bytes);
        }
        Err(HttpError::HttpStatus(500))

[Libbum]

use itertools:Itertools; //collect_tuple
use http_range_client::HttpError;

[Libbum]

With that small fix above, this seems to be doing exactly what I'd expect!

This is so great—I've been managing PSKs to get access to encrypted flatgeobufs on s3 up until now. Super clunky. Seamless with this wrapper. Thanks Kyle :)

[kylebarron]

🚀

[kylebarron]

Merging with follow ups tracked in #534. Not yet working for s3 files