-
Notifications
You must be signed in to change notification settings - Fork 2
Work with schemas
zoo is a sequence centric data structure, which influences the database schema design.
The minimum schema has 2 components or "fields":
- _id
- sequence
However, the typical (and more useful) sequence record will typically hold three more fields.:
- metadata
- relative
- derivative
Sequence information is at the center of zoo's functionalities. It is defined as a string of an arbitrary alphabet, typically RNA, DNA or Protein. As a consequence, each genome segment of a segmented virus such as Influenza A receives its own document, and is linked to the other segments of a given sample.
Metadata describe the way a sequence "came to be known". Where was it sampled from, who by, from which host, through which sample preparation and sequencing methods?
Relative information includes taxonomy, phylogeny and linked information. It addressed the question of how a given sequence string compares to others. Parts of phylogenetic trees or multiple sequence alignments are archived in this category.
Derived information summarizes or reexpresses the information contained in the sequence, including annotations, minhashes and alternative encodings. Derived information is usually heavily dependent on the original sequence. For example, the annotation open reading frame (ORF) derives from the sequence's start and a stop codon position. By definition, derived sequence information does not by itself make any sense without the underlying raw sequence information.
Note that all categories interact, e.g. we could use minhash signatures (derivative) to compare a sequence to other ones in the database, storing the top 5 closest sequence IDs (relative).
Since zoo is not restricted to any particular use case, some schemas are presented here to give an idea how data can be structured in zoo for different viruses and sets of sequences.
Note that schemas are composable: The Influenza A virus schema can incorporate the (intentionally generic) annotation schema, if the latter suits the intended purpose.
Note also that the base schema is "assumed" by some of the zoo package functions, so if you tinker with it (especially deleting keys) things might break. Which is ok, you can probably fix them.
Access to the schemas:
import json
from zoo import get_schema
with open(get_schema('influenza_a_virus.json')) as infile:
iav = json.load(infile)
with open(get_schema('annotation.json')) as infile:
anno = json.load(infile)
iav['derivative']['annotation'].append(anno)
To see how you can enforce a schema when importing data, refer to the corresponding wiki entry "Working with documents".
Note: This section is for reference only and not yet instructive at all.
be aware:
Not all constraints can be expressed. JSON Schema limits itself to describing the structure of JSON data, it cannot express functional constraints. If we take an NFS entry as an example, JSON Schema alone cannot check that the submitted NFS server's hostname, or IP address, is actually correct: this check is left to applications.
resources:
- JSON schema
- tutorial
- validation tool (Python)
Where do schemas come from? Ultimately, there will probably be some dedicated part of the registry, where there are curated schemas for different virus instances. However, for now they are shared with the data themselves.
Somebody curating a data cell does so with certain criteria. For example, the nomenclature for Influenza A virus is well defined, and we might want to enforce future entries to have a valid nomenclature entry. We can do this by defining a schema (or a set of schemas) that take this into account. Then, on initializing/ adding/ committing a data cell, the option --validate
allows schema enforcement.
For an example of how schema validation is implemented in zoo, see this example.
- representing "empty" in JSON, tl;dr: there is no standard (stackoverflow, 21120999)
- ...