Code

Structured zarr dtype as bytes#176

Code

Open

brian-michell wants to merge 46 commits intogoogle:mastergoogle/tensorstore:masterfrom brian-michell:Structured_zarr_as_bytesbrian-michell/tensorstore:Structured_zarr_as_bytesCopy head branch name to clipboard

+261-55

Commits

Commits on Jun 20, 2024

• Implemented concept of "open as byte array" for any dtype

  brian-michell

  committed
  4aa8cae

  Copy full SHA for 4aa8cae

Commits on Jun 21, 2024

• Added open option for byte array

  brian-michell

  committed
  ac6ba5c

  Copy full SHA for ac6ba5c

• Removed open option for byte array

  brian-michell

  committed
  6b43911

  Copy full SHA for 6b43911

• Allow modification of ParseDTypeOptions

  brian-michell

  committed
  c4beb11

  Copy full SHA for c4beb11

• Update default behavior

  brian-michell

  committed
  43cc7c8

  Copy full SHA for 43cc7c8

• Emplace instead of replace

  brian-michell

  committed
  a76363a

  Copy full SHA for a76363a

• Update field selection to use emplaced field

  brian-michell

  committed
  6342417

  Copy full SHA for 6342417

Commits on Jun 24, 2024

• Stopped synthetic dimension from getting seralized

  brian-michell

  committed
  f684f17

  Copy full SHA for f684f17

• Pushback synthetic field rather than make it the only field

  brian-michell

  committed
  93e92ce

  Copy full SHA for 93e92ce

• Skip the synthetic field as part of the number of bytes

  brian-michell

  committed
  874d581

  Copy full SHA for 874d581

• Fix const qualifier

  brian-michell

  committed
  c7408d4

  Copy full SHA for c7408d4

• Properly modify const object

  brian-michell

  committed
  83acfa6

  Copy full SHA for 83acfa6

• Properly modify has_fields logic

  brian-michell

  committed
  7ac53a4

  Copy full SHA for 7ac53a4

• Pulled check outside loop and fixed logic error

  brian-michell

  committed
  2c0bdc3

  Copy full SHA for 2c0bdc3

• Explicit initilization of values. Ensured has_fields for structured data without field

  brian-michell

  committed
  442daa0

  Copy full SHA for 442daa0

• Implemented proper field_shape for synthetic field

  brian-michell

  committed
  92145fc

  Copy full SHA for 92145fc

• Added valid case for an empty named field

  brian-michell

  committed
  77ce1ca

  Copy full SHA for 77ce1ca

• Disabled setting has_fields to false

  brian-michell

  committed
  1113ec7

  Copy full SHA for 1113ec7

• Fix encoding

  brian-michell

  committed
  9d9a555

  Copy full SHA for 9d9a555

Commits on Jun 25, 2024

• Fixed chunkgrid to use the true size

  brian-michell

  committed
  1d597ad

  Copy full SHA for 1d597ad

• Temporary drop of synthetic field for metadata validation

  brian-michell

  committed
  ea1ae5c

  Copy full SHA for ea1ae5c

• Fix single field dtype without name incorrectly passing

  brian-michell

  committed
  5355201

  Copy full SHA for 5355201

• Fix multiple synthetic fields getting added

  brian-michell

  committed
  5c9bd16

  Copy full SHA for 5c9bd16

• Removed extra argument

  brian-michell

  committed
  c4f0178

  Copy full SHA for c4f0178

• Catch case where there was no field due to being a struct array of size 1

  brian-michell

  committed
  288b175

  Copy full SHA for 288b175

• Ignore the synthetic dimension when chunking

  brian-michell

  committed
  de830ef

  Copy full SHA for de830ef

• Remove the synthetic dimension when it is not needed

  brian-michell

  committed
  619a03a

  Copy full SHA for 619a03a

• Update tests to reflect DType without going through the spec driver process

  brian-michell

  committed
  bb68abf

  Copy full SHA for bb68abf

• Update test to reflect metadata containing synthetic dimension. (Handled as no field provided)

  brian-michell

  committed
  e40dfdd

  Copy full SHA for e40dfdd

• Remove check that is no-longer true

  brian-michell

  committed
  8fc4fa6

  Copy full SHA for 8fc4fa6

• Remove test that is no longer true

  brian-michell

  committed
  9912b32

  Copy full SHA for 9912b32

• Only use the actual fields for fill values

  brian-michell

  committed
  6015c31

  Copy full SHA for 6015c31

• Cleaning up convention

  brian-michell

  committed
  5ae3d36

  Copy full SHA for 5ae3d36

• Fixed incorrect assertion

  brian-michell

  committed
  8ca6c74

  Copy full SHA for 8ca6c74

Commits on Jun 26, 2024

• Add beginning of new test for field-free opening

  brian-michell

  committed
  211beab

  Copy full SHA for 211beab

• Fix num_bytes not getting set

  brian-michell

  committed
  a4d7e73

  Copy full SHA for a4d7e73

• Allow the metadata to be one size higher in the event of structured data without field

  brian-michell

  committed
  e51b3fe

  Copy full SHA for e51b3fe

• Properly specify the chunk grid for synthetic data field

  brian-michell

  committed
  f30948c

  Copy full SHA for f30948c

• Added fallback for zarr synthetic field

  brian-michell

  committed
  824d5bc

  Copy full SHA for 824d5bc

• Fix test to reflect num_bytes getting properly set

  brian-michell

  committed
  364a569

  Copy full SHA for 364a569

Commits on Jul 1, 2024

• Disabled ignoring 'true size' for chunk layout

  brian-michell

  committed
  620af12

  Copy full SHA for 620af12

• Fixed metadata caching causing extra dimension getting added

  brian-michell

  committed
  60060a7

  Copy full SHA for 60060a7

Commits on Jul 5, 2024

• Fix hardcoded case for opening existing zarr without field

  brian-michell

  committed
  5b56a98

  Copy full SHA for 5b56a98

• Implement naieve fix for fill_value assertion failure on no field struct data

  brian-michell

  committed
  08fbd48

  Copy full SHA for 08fbd48

• Remove incorrect decrement of struct data

  brian-michell

  committed
  c1c0ccd

  Copy full SHA for c1c0ccd

Commits on Jul 9, 2024

• Added test for opening existing store as struct array

  brian-michell

  committed
  96144b7

  Copy full SHA for 96144b7