Use `u64` range instead of `usize`, for better wasm32 support #6961

XiangpengHao · 2025-01-10T15:55:26Z

Which issue does this PR close?

Closes Support 32-bit Architectures / Replace usize with u64 #5351

Rationale for this change

Ranges should be u64 so that 32 bit platforms can read files larger than 4GB. More details can be found in #5351

What changes are included in this PR?

This is a rather simple change. No functionality change for 64 bit platforms. But is a breaking change for trait implementors.
Given that we already break one in #6619, it's seems like a good timing to also include this change.

This PR added and removed a few casting. I have checked the casting is ok, but please help me check again.

My principle of using usize vs u64:

Anything already in memory should use usize, e.g., slicing a memory range
O.w. use u64

Are there any user-facing changes?

alamb · 2025-01-10T19:03:53Z

The CI failure https://github.com/apache/arrow-rs/actions/runs/12713696201/job/35442156669?pr=6961 should be fixed with this PR:

Fix WASM CI for Rust 1.84 release #6963

alamb

Thanks @XiangpengHao ❤️

Can you help me understand what currently doesn't work with 32 bit builds now?

We have an existing test for WASM32 that seems to work fine: https://github.com/apache/arrow-rs/actions/runs/12680096825/job/35341184364

I also locally tried checking with an i686 target and that worked fine too 🤔

rustup target add i686-unknown-linux-gnu
cd object_store
cargo check --target=i686-unknown-linux-gnu
...

🤔

alamb · 2025-01-10T19:22:08Z

🤔 maybe it is related to wanting to use the http features?

But it still seems to compile just fine for me 🤔

sudo apt-get install gcc-multilib
rustup target add i686-unknown-linux-gnu
cd object_store
cargo check --target=i686-unknown-linux-gnu --features=http
...

alamb · 2025-01-11T18:01:09Z

Can you help me understand what currently doesn't work with 32 bit builds now?

For anyone following along, the answer is here: #5351 (comment)

I still think we should try and add some sort of test that would fail on wasm32 before this change and not after the change. If we don't do that I feel like:

We are less sure we fixed all the places that require change
We may inadvertently introduce regressions but no test would fail

I'll see if I can find some time over the next day or two to help, if no on ebeats me to it

XiangpengHao · 2025-01-13T21:54:47Z

add some sort of test that would fail on wasm32

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

Xuanwo · 2025-01-14T15:00:31Z

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

OpenDAL has an edge test for this; it's worth taking a look.

https://github.com/apache/opendal/blob/781682cc05b31d1f12f91fb7568a868f038c4d3b/.github/workflows/test_edge.yml#L62-L107

The test code could be found here: https://github.com/apache/opendal/tree/main/core/edge/s3_read_on_wasm

alamb

Thank you @XiangpengHao -- I reviewed this PR carefully and I think the API changes are good but there are a few places where going from u64 to usize should be checked (as it could truncate on WASM).

I may be mistaken (my casting knowledge is mostly left over from C/C++ 😅 )

I also have a few other suggestions related to comments, but otherwise this is good to go from my perspectivel

Thank you again for helping push this along

object_store/src/azure/client.rs

object_store/src/chunked.rs

object_store/src/lib.rs

object_store/src/local.rs

object_store/src/memory.rs

object_store/src/util.rs

alamb · 2025-01-14T15:32:57Z

object_store/src/util.rs

@@ -332,7 +333,9 @@ mod tests {
            &ranges,
            |range| {
                fetches.push(range.clone());
-                futures::future::ready(Ok(Bytes::from(src[range].to_vec())))
+                futures::future::ready(Ok(Bytes::from(
+                    src[range.start as usize..range.end as usize].to_vec(),


again, I think we need to check the conversion here rather than truncating

this function doesn't return error, I just unwrap here, as it's only a test

alamb · 2025-01-14T15:41:54Z

BTW I merged up from main. I also think the obejct_store emulator tests are unrelated to this PR. I have filed

Intermittent failure with object_store aws::tests::s3_test emulator #6981

To track

alamb · 2025-01-14T16:57:17Z

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

OpenDAL has an edge test for this; it's worth taking a look.

https://github.com/apache/opendal/blob/781682cc05b31d1f12f91fb7568a868f038c4d3b/.github/workflows/test_edge.yml#L62-L107

The test code could be found here: https://github.com/apache/opendal/tree/main/core/edge/s3_read_on_wasm

Thank you @Xuanwo -- I didn't see this response before posting my update

alamb · 2025-01-14T17:16:18Z

I restarted the emulator test to see if it passes:
Old: https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35594807418?pr=6961

alamb · 2025-01-14T19:27:55Z

Now the emulator test is passing after 2 retries: https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35607985446?pr=6961

🤔

Co-authored-by: Andrew Lamb <[email protected]>

alamb

Thanks @XiangpengHao -- this looks great to me

XiangpengHao added 3 commits January 9, 2025 16:46

u64 ranges

c1aab14

more u64

b114802

make clippy happy

450a268

github-actions bot added the object-store Object Store Interface label Jan 10, 2025

even more u64

b6670cd

alamb added the api-change Changes to the arrow API label Jan 10, 2025

alamb reviewed Jan 10, 2025

View reviewed changes

alamb mentioned this pull request Jan 10, 2025

Support 32-bit Architectures / Replace usize with u64 #5351

Open

XiangpengHao mentioned this pull request Jan 13, 2025

ObjectStore WASM32 Support #6818

Open

3 tasks

Merge remote-tracking branch 'apache/main' into u64_range

ebcb38e

alamb mentioned this pull request Jan 14, 2025

Run object_store tests (on CI) with WASM build/runtime #6980

Open

alamb changed the title ~~Switch to use u64 range instead of usize, for better wasm32 support~~ Use u64 range instead of usize, for better wasm32 support Jan 14, 2025

alamb reviewed Jan 14, 2025

View reviewed changes

alamb mentioned this pull request Jan 14, 2025

Intermittent failure with object_store aws::tests::s3_test emulator #6981

Open

XiangpengHao and others added 3 commits January 15, 2025 16:49

Update object_store/src/lib.rs

af30a4c

Co-authored-by: Andrew Lamb <[email protected]>

Update object_store/src/lib.rs

5608f01

Co-authored-by: Andrew Lamb <[email protected]>

address comments

077e99b

alamb approved these changes Jan 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `u64` range instead of `usize`, for better wasm32 support #6961

Use `u64` range instead of `usize`, for better wasm32 support #6961

XiangpengHao commented Jan 10, 2025 •

edited by alamb

Loading

alamb commented Jan 10, 2025

alamb left a comment

alamb commented Jan 10, 2025

alamb commented Jan 11, 2025 •

edited

Loading

XiangpengHao commented Jan 13, 2025

Xuanwo commented Jan 14, 2025 •

edited

Loading

alamb left a comment

alamb Jan 14, 2025

XiangpengHao Jan 16, 2025

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb left a comment

Use u64 range instead of usize, for better wasm32 support #6961

Are you sure you want to change the base?

Use u64 range instead of usize, for better wasm32 support #6961

Conversation

XiangpengHao commented Jan 10, 2025 • edited by alamb Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

alamb commented Jan 10, 2025

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jan 10, 2025

alamb commented Jan 11, 2025 • edited Loading

XiangpengHao commented Jan 13, 2025

Xuanwo commented Jan 14, 2025 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

alamb Jan 14, 2025

Choose a reason for hiding this comment

XiangpengHao Jan 16, 2025

Choose a reason for hiding this comment

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb commented Jan 14, 2025

alamb left a comment

Choose a reason for hiding this comment

Use `u64` range instead of `usize`, for better wasm32 support #6961

Use `u64` range instead of `usize`, for better wasm32 support #6961

XiangpengHao commented Jan 10, 2025 •

edited by alamb

Loading

alamb commented Jan 11, 2025 •

edited

Loading

Xuanwo commented Jan 14, 2025 •

edited

Loading