Extending PR 106 #110
Extending PR 106 #110
Conversation
fixed a couple typos and filled-in the text since I was here, then I edited the whole thing for my own clarity fwiw in the third paragraph I'm not happy with: "Unfortunately this means one might accidentally install everything again and replicate their system.", since I think it's not clear what actions the user must take for this to occur and the word install doesn't give me an indication of what that means in Julia where you would add or load or use. It could also indicate if this is primarily a memory problem and if so which memory, primary, secondary, something else? 🍰
unfilled-in lines
|
|
||
| This information makes it possible to reconstruct the original environment necessary to deserialize the objects if the internal serialization format has introduced breaking changes. |
There was a problem hiding this comment.
I still don't love this sentence, but I want to make it clear that the metadata serves more as breadcrumbs for resolving serialization issues.
There was a problem hiding this comment.
Yes, I think it is not great.
Even if the serializer hasn't introduced breaking changes, if the structure of the object itself has changed that is also a problem, that need to be dealt with via loading the old object.
Maybe more like
| This information makes it possible to reconstruct the original environment necessary to deserialize the objects if the internal serialization format has introduced breaking changes. | |
| In general, serializers are not able to handle deserialized objects if the type defintions of the stored objects have changed. | |
| For example if a field has been added then there is no such field in the stored data so it can't be loaded using the new version of the package. | |
| JLSO stores the package version, thus allowing you to load the old version of the package from before that change and so be able to recover the object. | |
| Similar occurs if the serializer itself changes. |
Even though it is longer
Codecov Report
@@ Coverage Diff @@
## master #110 +/- ##
==========================================
+ Coverage 97.56% 97.67% +0.11%
==========================================
Files 6 6
Lines 205 215 +10
==========================================
+ Hits 200 210 +10
Misses 5 5
Continue to review full report at Codecov.
|
|
|
||
| This information makes it possible to reconstruct the original environment necessary to deserialize the objects if the internal serialization format has introduced breaking changes. | ||
| Since the metadata is formatted in language-agnostic BSON (no Julia extensions), all you need is a BSON reader. | ||
| Due to the size of this metadata, JLSO files are not ideal for storing many small files. |
There was a problem hiding this comment.
I wonder if some of this would be clearer in the context of an FAQ section like
Who is this intended for?
Why is this metadata useful?
Why is there a balancing act between performance, flexibility and reliability in serialization formats?
There was a problem hiding this comment.
Maybe. Can be a follow up PR.
Some of that wouldn't want to be in a readme?
| that employs a serializer, and a compressor, handles all the other concerns including metadata and saving. | ||
| Such that the serializer just needs to determine how to turn a julia object into a stream`Vector{UInt8}`, | ||
| and the compressor just needs to determine how to turn one stream of `UInt8`s into a smaller one (and the reverse). | ||
| Think of it less as a serialization format and more of a container for arbitrarily serialized objects and metadata. |
There was a problem hiding this comment.
The point of calling it a container was not merely that that it contains things.
It is specifically referring to a container format
https://en.wikipedia.org/wiki/Container_format_(computing)
such as the Real Media format https://en.wikipedia.org/wiki/RealMedia
which can in turn wrap multiple different codecs/formats (serializers in our case)
With the key untility of JLSO that the serializer and compressors just have to work with Vector{UInt8} and JLSO takes care of hooking them up and of metadata.
(which is very much what container formats do).
I think moving that information away from the start hides this.
Possibly can solve by just linking to wikipedia on container format though
|
|
||
| This information makes it possible to reconstruct the original environment necessary to deserialize the objects if the internal serialization format has introduced breaking changes. | ||
| Since the metadata is formatted in language-agnostic BSON (no Julia extensions), all you need is a BSON reader. |
There was a problem hiding this comment.
Maybe
| Since the metadata is formatted in language-agnostic BSON (no Julia extensions), all you need is a BSON reader. | |
| Since the metadata is formatted in language-agnostic BSON (no Julia extensions), in the worst case you can access these breadcumbs with any BSON reader. |
| - JLSO version number | ||
| - Julia version number | ||
| - Serialization format (e.g., "julia_serialize", "bson") | ||
| - Compression method like "gzip" | ||
| - A docker image URI | ||
| - Project.toml | ||
| - Manifest.toml (possibly compressed) |
There was a problem hiding this comment.
Do we need all this listed in the README?
Closes #106