Add Namespace Definition/Imports to Operator Log

[esoterra]

As proposed in November 1st meeting, I think a good way to start prototyping federation is to introduce a basic namespace-to-origin mapping into the registrty state via two new Operator Log Entries.

1. ImportNamespace { name: String, registry: String } - This registry imports the namespace name from registry.
2. DefineNamespace { name: String } - This registry defines the namespace name in its own logs.

A registry's name is currently required to be a domain name and a registry's "location" is currently required to map to itself. In the future we can relax this constraint if we want to add indirection between registry name and registry location.

As a validation rule, it would be illegal for an operator log to contain two or more *Namespace entries using the same name value. This means that every namespace is mapped to exactly zero or one registries (either a remote or this one) and that that mapping does not change over time.

This feature when combined with the previously discussed Warg-Registry header will make it possible for clients to reason about the relationships between registries and talk to registries about each other. Most (or all?) registries will want to require that releases only reference namespaces that are bound and that requests with Warg-Registry headers only use header values which are registries in the operator log as this acts as a record of which registries they federate with.

Future Note 1: ImportNamespace can be extended in the future with an additional field that indicates what name from the specified registry is being imported as name and when no value is provided for that it is implicitly the same as name.

Future Note 2: The no duplicate definitions/imports validation rule can be relaxed in the future in order to introduce namespace re-mapping.

[calvinrp]

To clarify, registry: String and its use in the Warg-Registry header need to validate as a domain name or authority?

[lann]

We could add a header to federated (Warg-Registry) responses:
Warg-Registry-Location: <registry URL>

In the short-term that would always be e.g. https://{registry} but it might let us avoid needing to require that registry must be a domain name. In the future (with more design work) it might also be a good mechanism for updating an imported registry's origin URL.

[calvinrp]

To elaborate...

Warg-Registry header and registry values in the proposed operator log entry are arbitrary unique string IDs (but not necessarily globally unique across other registries).

To handle the use case where a client wants to check the actual source registry API in addition to what is provided by the proxying registry, a Warg-Registry-Location header is optionally provided the current URL to the source registry. That source registry URL may change over time but the identifier used in the Warg-Registry header wouldn't change.

How do we feel about that?

Do we still need future extension to remap a namespace under this implementation?

[esoterra]

For now, I would really like to stick to the names being Domain Names because the fact that they're globally unique means they mean the same thing regardless of who your home registry is (which simplifies user assumptions) and we can cache things from the same registry name together because they refer to the same state.

To zero in on something Calvin said, I think the Warg-Registry-Location proposal should be considered a candidate implementation for Namespace Remapping and not part of this feature because they serve very overlapping purposes.

I would like to move forward initially without the additional Warg-Registry-Location header. To carve out some room for options like this one, I'll update my initial post to talk about registry names as being "currently required to be a domain name" and "currently required to map to a location 1:1" leaving us the room to consider future features as relaxing these constraints.

[calvinrp]

On that note, do you think the client should globally share the cache of logs by registry domain, regardless if sourced directly or different proxy registries?

I could imagine proxy registry checkpoints being not necessarily up to date and in sync. That might be problematic.

I recall that we discussed separately caching previously. Thoughts?

Suppose could partially share things. For instance, separately store checkpoints and validators but fetch the logs and cache those globally.

[esoterra]

My intention is that clients can combine together information about a given registry from different sources (e.g. the registry itself, its home registry) as long as every new checkpoint it receives is checked against the registry it's about (either by getting it from its own origin or monitoring against the origin) and all data about the registry is checked against verified checkpoints.

There is some consistency stuff we need to be careful about, but I think this is the goal and is pretty powerful.

[calvinrp]

As a clarifying point, namespace importing does not imply proxying support. And a registry may support proxying other registries that aren't mentioned in the operator log.

So the operator log's ImportNamespace mappings define how published package dependencies are resolved but doesn't imply proxying support for those registries.

By default, the CLI tries to proxy through the default registry. If that isn't supported, then the client tries to directly request from the source registry.

How does that sound?

[calvinrp]

As a clarifying point, namespace importing does not imply proxying support. And a registry may support proxying other registries that aren't mentioned in the operator log.

So the operator log's ImportNamespace mappings define how published package dependencies are resolved but doesn't imply proxying support for those registries.

By default, the CLI tries to proxy through the default registry. If that isn't supported, then the client tries to directly request from the source registry.

How does that sound?

This might be problematic in the case where the registry domain needs to be remapped, but we don't have that ability.

[esoterra]

I'd like us to find a better term for "serving a response on behalf of another party" than proxy, but I think letting registries choose to not serve the content of all registries they import is fine and we can write the client to fall back to fetching from the imported registry itself.

I do think for certain common registry use cases (e.g. internal corporate registries) they'll serve all of it but we don't have to impose that as a requirement.

[calvinrp]

Think we are ready for a PR?

[esoterra]

I think so, it should be relatively straightforward to add the entry types and implement the validation logic.

Since the client already validates the operator log it'll get propagated to the client without any extra work.

[calvinrp]

Merged the server-side #225