[CT-91] Separate serialization for internal use and for artifacts with published schemas

[gshank]

Right now we use the same serialization for creating the partial_parse.msgpack file, and the manifest.json (and other artifacts). In some cases we need an attribute in internal processing but it's not necessary for the manifest.json, but implementing the attribute means we have to update the published json schema.

Investigate ways to have different serialization. One possibility: we use an 'omit_none' flag that gets passed along for serialization. Perhaps we could implement a similar flag for artifacts.

The goal would be to implement this for 1.1.0.

[kwigley]

Sorry for adding my 2 cents. I think these should be different data classes entirely. I don't think the same data class that is used for internal purposes should be used to write external artifacts. We can using mapping to go from internal representation <--> external representation and only the external representation needs to care about serialization. I think there is a path forward here that does not involve the (in)famous "hologram" package as well. Happy to provide examples when thinking about this more!

[jtcohen6]

Moving away from hologram, and toward a better dataclass -> JSONSchema serialization package, may also resolve some of the residual issues with the schemas we're producing today: #4657

[jtcohen6]

The original prompt/motivation for this issue:

• We made a tiny change to the manifest schema in [#2479] Allow unique_id to take a list #4618 (just a new default)
• By our current practice, we'll need to bump the manifest version to v5, creating work for everyone who parses metadata from the manifest

I'm interested in any small, tactical lifts we can make so that we're not bumping the manifest schema version in every single new minor version. At the same time, we're eager to provide other interfaces into manifest information, over which we have more control: #4807

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[leahwicz]

Closing this one because things have changed and we are looking to create artifacts off of the protobuf schemas