Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test zarrita compatibility #3

Merged
merged 52 commits into from
Jun 3, 2024
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
f6c5b8d
add zarrita python script
brokkoli71 May 14, 2024
a678fd4
add testReadFromZarrita
brokkoli71 May 14, 2024
8a8311e
renamed zarrita_write
brokkoli71 May 14, 2024
270e690
add testWriteToZarrita
brokkoli71 May 15, 2024
b23e392
Merge branch 'main' into test-zarrita-compatibility
brokkoli71 May 15, 2024
21ba578
parameterized codecs in testReadFromZarrita
brokkoli71 May 16, 2024
8b9828a
fixed camelcase in zarr.json
brokkoli71 May 16, 2024
9973117
add venv for executing zarrita in linux
brokkoli71 May 17, 2024
351b7f8
adapt test-setup for linux
brokkoli71 May 17, 2024
dd3e939
adapt test-setup for linux
brokkoli71 May 17, 2024
4125940
fix testReadFromZarrita for gzip
brokkoli71 May 21, 2024
c3f1510
parameterize codecs in testReadFromZarrita
brokkoli71 May 21, 2024
18693d7
remove unused imports
brokkoli71 May 21, 2024
ecd1dea
add testCodecsWriteRead
brokkoli71 May 21, 2024
adbcd43
reformat
brokkoli71 May 21, 2024
0f50859
remove unnecessary argument
brokkoli71 May 22, 2024
db122db
add testCodecTranspose
brokkoli71 May 22, 2024
842ad54
start remove constants "C" adn "F" from Transpose Codec's order
brokkoli71 May 22, 2024
cf0207d
add CoreArrayMetadata to codec object instead of passing as argument …
brokkoli71 May 24, 2024
363c37b
Merge remote-tracking branch 'origin/main' into fix-transpose-codec
brokkoli71 May 24, 2024
e502479
remove wrong dependency
brokkoli71 May 24, 2024
292ecc6
set fail-fast: false
brokkoli71 May 24, 2024
30bb41d
specify testSourceDirectory
brokkoli71 May 24, 2024
cf105f0
added property index_location to ShardingIndexedCodec
brokkoli71 May 25, 2024
e49f016
add indexLocation in ShardingCodec.
brokkoli71 May 27, 2024
770e1dd
change junit version for TestUtils
brokkoli71 May 27, 2024
fbd190a
remove creation of dir testdata
brokkoli71 May 27, 2024
9e3a240
update dependencies for JUnit 5
brokkoli71 May 27, 2024
4ca12b7
install zarrita in CI
brokkoli71 May 27, 2024
5b8ca3e
correct python version, maven no-transfer-progress
brokkoli71 May 27, 2024
db4f765
add venv for executing zarrita to CI
brokkoli71 May 27, 2024
6b793cd
add /venv_zarrita to .gitignore
brokkoli71 May 27, 2024
d553f17
remove deprecated zarrita venv
brokkoli71 May 27, 2024
cd84d08
test CI
brokkoli71 May 27, 2024
215f52d
fix tests for windows
brokkoli71 May 27, 2024
71c7548
update python path for windows in ci.yml
brokkoli71 May 27, 2024
bb4f50b
add Development Start-Guide to Run Tests Locally
brokkoli71 May 29, 2024
d62070c
correct Development Start-Guide to python3.11
brokkoli71 May 29, 2024
d2d3e04
Merge pull request #1 from scalableminds/fix-transpose-codec
brokkoli71 May 29, 2024
67f5cd9
add support of shardingCodec indexLocation=start
brokkoli71 May 29, 2024
4f9dc72
code cleanup
brokkoli71 May 29, 2024
e51ac9d
Merge pull request #2 from scalableminds/fix-sharding-codec
brokkoli71 May 29, 2024
dbcee48
add testZstdLibrary
brokkoli71 May 29, 2024
26964dc
fix Zstd compression and decompression
brokkoli71 May 30, 2024
2d8b7b2
cleanup code
brokkoli71 May 30, 2024
554715f
cleanup code
brokkoli71 May 30, 2024
29541d5
remove unused method
brokkoli71 May 30, 2024
9733e07
Merge pull request #4 from scalableminds/fix-zstd-codec
brokkoli71 May 30, 2024
8d4eae5
include crc32c codec into tests
brokkoli71 May 30, 2024
fbb2b67
incorporate feedback for java version and merge tests
brokkoli71 Jun 3, 2024
db47786
add more variation to codec-configuration in tests with zarrita
brokkoli71 Jun 3, 2024
39ffce5
fix BloscCodec with shuffle = "shuffle"
brokkoli71 Jun 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
include crc32c codec into tests
brokkoli71 committed May 30, 2024
commit 8d4eae50ee92014432f321a5a221710bdd1fa798
4 changes: 4 additions & 0 deletions src/main/java/dev/zarr/zarrjava/v3/codec/CodecBuilder.java
Original file line number Diff line number Diff line change
@@ -139,6 +139,10 @@ public CodecBuilder withSharding(int[] chunkShape,
return this;
}

public CodecBuilder withCrc32c() {
codecs.add(new Crc32cCodec());
return this;
}
private void autoInsertBytesCodec() {
if (codecs.stream().noneMatch(c -> c instanceof ArrayBytesCodec)) {
Codec[] arrayArrayCodecs = codecs.stream().filter(c -> c instanceof ArrayArrayCodec)
9 changes: 4 additions & 5 deletions src/test/java/dev/zarr/zarrjava/ZarrTest.java
Original file line number Diff line number Diff line change
@@ -127,9 +127,8 @@ public void testZstdLibrary(int clevel, boolean checksumFlag) throws IOException
assert exitCode == 0;
}

//TODO: add crc32c
@ParameterizedTest
@ValueSource(strings = {"blosc", "gzip", "zstd", "bytes", "transpose", "sharding_start", "sharding_end"})
@ValueSource(strings = {"blosc", "gzip", "zstd", "bytes", "transpose", "sharding_start", "sharding_end", "crc32c"})
public void testWriteToZarrita(String codec) throws IOException, ZarrException, InterruptedException {
StoreHandle storeHandle = new FilesystemStore(TESTOUTPUT).resolve("write_to_zarrita", codec);
ArrayMetadataBuilder builder = Array.metadataBuilder()
@@ -161,7 +160,7 @@ public void testWriteToZarrita(String codec) throws IOException, ZarrException,
builder = builder.withCodecs(c -> c.withSharding(new int[]{4, 4}, c1 -> c1.withBytes("LITTLE"), "end"));
break;
case "crc32c":
//missing
builder = builder.withCodecs(CodecBuilder::withCrc32c);
break;
default:
throw new IllegalArgumentException("Invalid Codec: " + codec);
@@ -195,7 +194,7 @@ public void testWriteToZarrita(String codec) throws IOException, ZarrException,


@ParameterizedTest
@ValueSource(strings = {"blosc", "gzip", "zstd", "bytes", "transpose", "sharding_start", "sharding_end"})
@ValueSource(strings = {"blosc", "gzip", "zstd", "bytes", "transpose", "sharding_start", "sharding_end", "crc32c"})
public void testCodecsWriteRead(String codec) throws IOException, ZarrException {
int[] testData = new int[16 * 16 * 16];
Arrays.setAll(testData, p -> p);
@@ -231,7 +230,7 @@ public void testCodecsWriteRead(String codec) throws IOException, ZarrException
builder = builder.withCodecs(c -> c.withSharding(new int[]{2, 2, 4}, c1 -> c1.withBytes("LITTLE"), "start"));
break;
case "crc32c":
//missing
builder = builder.withCodecs(CodecBuilder::withCrc32c);
break;
default:
throw new IllegalArgumentException("Invalid Codec: " + codec);