-
Notifications
You must be signed in to change notification settings - Fork 184
Making Serialization Changes
This wiki article contains the steps needed to update the serialization schemas. This is needed only if you intend to change the serialization format.
Cap'n Proto support schema evolution in a way that allows compatibility and maintaining the message encoding. See evolving-your-protocol
There are two specifications that must be updated when making changes to the serialization format. The canonical specification is the Cap'n Proto Spec. The OpenAPI specification exists to provide non libtiledb clients the ability to use the json format for fetching array schemas. At some point in the future we will automate the openAPI specification from the Cap'n Proto specification.
First the Cap'n Proto spec file must be updated, it is located in the tiledb/sm/serialization folder.
Edit the tiledb-rest.capnp file as appropriate.
The important notes about making backward compatible changes are, new structure or fields can be added anywhere in the file. New fields must have id larger than any existing field of a struct (i.e. they must numerically be added to the end of the structures).
Take the following example specification:
struct Domain {
# Domain of array
cellOrder @0 :Text;
# Tile Order
dimensions @1 :List(Dimension);
# Array of dimensions
tileOrder @2 :Text;
# Tile Order
type @3 :Text;
# Datatype of domain
}
If you wanted to a a new field called name you would add it like:
struct Domain {
# Domain of array
cellOrder @0 :Text;
# Tile Order
dimensions @1 :List(Dimension);
# Array of dimensions
tileOrder @2 :Text;
# Tile Order
type @3 :Text;
# Datatype of domain
name @4 :Text;
# Example new field of name of domain
}
After the specification has been updated new c++ classes must be generated for posix and win32.
You must use the capnp compile command to generate the new classes.
If you have build the super build of tiledb you can find capnp located in $BUILD_DIR/externals/install/bin/capnp
To run from the super build with bash from the tiledb/sm/serialization directory:
export BUILD_DIR=/path/to/tiledb_build
PATH=${PATH}:$BUILD_DIR/externals/install/bin $BUILD_DIR/externals/install/bin/capnp compile -I $BUILD_DIR/externals/install/include -oc++ tiledb-rest.capnp
or after vcpkg :
export BUILD_DIR=/path/to/tiledb_build
PATH=${PATH}:$BUILD_DIR/vcpkg_installed/x64-osx/tools/capnproto $BUILD_DIR/vcpkg_installed/x64-osx/tools/capnproto/capnp compile -I $BUILD_DIR/vcpkg_installed/x64-osx/include -oc++ tiledb-rest.capnp
If you are using a Cap'n Proto that is globally installed on your system, you must use Cap'n Proto 0.6.1
The following command should be used for generating the tiledb-rest classes.
capnp compile -oc++ tiledb-rest.capnp
The commands above will produce two files tiledb-rest.capnp.c++ and tiledb-rest.capnp.h which you must copy to $TILDB_DIR/tiledb/sm/serialization/posix and $TILDB_DIR/tiledb/sm/serialization/win32 (respectively for the appropriate build environment).
Based on the changes to serialization it also important to update the relevant *_to_capnp or *_from_capnp sections of code located in $TILDB_DIR/tiledb/sm/serialization/array_schema.cc or $TILDB_DIR/tiledb/sm/serialization/query.cc.
More details to come..
The openAPI specification is needed for non libtiledb usage. Any PR changing the Cap'n Proto specification must also update the openAPI specification.
First step to updating the openapi spec is cloning the appropriate repository and creating a new branch to work on:
git clone [email protected]:TileDB-Inc/TileDB-Cloud-API-Spec.git
cd TileDB-Cloud-API-Spec
git checkout -b <branch_name>
Then edit the relevant model, just as you did with the capnproto spec file:
https://github.com/TileDB-Inc/TileDB-Cloud-API-Spec/blob/master/openapi-v1.yaml