refactor(flipt/validate): build entire snapshot on validate

[GeorgeMac]

Fixes #2086
Fixes: FLI-568

This is a draft while we consider the proposed change.
The outstanding change needed to deliver this is a decision around referential integrity error enforcement.

This PR changes a few things, however, primarily it builds an entire store snapshot on flipt validate.
Doing this means we can get full confidence that a snapshot can be built from a target set of configuration on validate.
This give full confidence during e.g. a CI step.

To achieve this however, I have actually added the CUE validate step to the snapshot builder itself.
Meaning now, failures to validate via the CUE schema will fail snapshot building at runtime.
(This never happened before and the snapshot builder could be more lenient than the CUE schema).

When I did this, I found a couple constraints that I laxed in the existing CUE schema:

1. Variant name is not required
2. Percentage on a threshold can be represented as either an int or float in config

So, merging this, we might be introducing more stringent validation to the FS backends at runtime.
That said, the parity between the snapshot and building should mean that flipt validate is all the confidence you need.

Beyond this, this also update flipt validate so that it can take zero or more paths.
Zero paths used to mean, validate nothing.
Now it means, perform the same process as the FS backends. Meaning, attempt to parse the flipt index file and find all the files that match it (defaulting to the features.yaml files we default to now).

flipt validate can still be called with existing files, and only those files will be validated.

Outstanding Decision

Currently, the fs backend snapshot builder will just fail silently when references cant be resolved.
For example:

flipt/internal/storage/fs/snapshot.go

Lines 364 to 367 in a4d2662

	variant, found := findByKey(d.VariantKey, flag.Variants...)
	if !found {
	continue
	}

We need to decide how to proceed here, in order to surface these as errors at validation time.
Depending on the decision, this may be considered a breaking change.

1. We could just return an error here

This would be the breaking change for anyone currently just relying on the silent dropping of unreferenceable resources.

2. We could warn instead, and optionally fail based on config

This idea is to add a warning log to these sites.
Then we could add a flag to both Flipts backend and flipt validate that could cause Flipt to fail fast in these situations.
For example, --strict.

This is obviously something that might be easy to miss, so has to be considered.
Maybe it is safe right now to just cut over and break this behaviour.

Update:

We chose to move forward with the breaking change.
So this PR introduces an error when a distributions variant cannot be resolved (as it does not exist).

[codecov]

Codecov Report

Merging #2093 (93d6296) into main (29d3f9d) will decrease coverage by 0.06%.
The diff coverage is 51.75%.

@@            Coverage Diff             @@
##             main    #2093      +/-   ##
==========================================
- Coverage   70.81%   70.76%   -0.06%     
==========================================
  Files          73       73              
  Lines        7011     7042      +31     
==========================================
+ Hits         4965     4983      +18     
- Misses       1761     1774      +13     
  Partials      285      285              

Files Changed	Coverage Δ
internal/cue/validate.go	62.06% <32.00%> (-23.98%)	⬇️
internal/storage/fs/snapshot.go	69.33% <47.14%> (+1.24%)	⬆️
internal/storage/fs/sync.go	86.02% <94.11%> (ø)
internal/storage/fs/store.go	79.31% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[yquansah]

@GeorgeMac I think a warning log for what we fail silently for makes sense for now so its not a breaking change. Maybe we can have a deprecation path forward that would make strict mode the default in the future.

[markphelps]

im good with making this a somewhat breaking change tbh