Implement derive(ValidateSchema) macro for generating cel validation on CRDs

[Danil-Grigorev]

Motivation

Related to #1367

CRDs allow to declare server-side validation rules using CEL. This functionality is supported via #[schemars(schema_with = "<schemagen-wrapper>")], but requires defining a method with handling logic, which may be error-prone.

Since kube owns CRD generation code, the idea is to simplify this process for added validation rules and achieve more declarative approach, similar to the kubebuilder library. This approach will be compatible with kopium generation based on existing CRD structures, already using CEL expressions.

Solution

Allow for a more native handling of CEL validation rules on the CRDs via a field macro.

#[derive(…, ValidateSchema)]
#[kube(
  rule = Rule::new("self.metadata.name == 'singleton'"),
…
)]
pub struct FooSpec {
    // Field with CEL validation
    #[serde(default)]
    #[cel_validate(
         rule = Rule::new("self != 'illegal'").message("string cannot be illegal").reason(Reason::FieldValueForbidden),
         rule = Rule::new("self != 'not legal'").reason(Reason::FieldValueInvalid),
     )]
    cel_validated: Option<String>,

This PR is a followup on #1621 which addresses some of the concerns.

1. Implemented by generating a JsonSchema, which allows further additions to the schema later, struct level validation rules and is not affected by schemars version.
2. Based on Implement CEL validation proc macro for generated CRDs #1621 (comment) - using methods from the kube::core and invoking from kube::derive.
3. Generally simplified the code to make it easier to maintain (Implement CEL validation proc macro for generated CRDs #1621 (comment))

Other things tried (TLDR)

Visitor trait:

It is possible to generate a new Visitor implementation per each validation rule. But the problem with this approach is that the generation happens for Validator derive on the structure, while the CustomResource derive is responsible for populating additional visitors for crd(). There is no one specific method which can collect all visitors under one chain, invoked from schemars. This likely requires every individual field in each struct to implement the Validated trait, involving creation of a shemars/serde type of logic.

Then the schemars Schema has no indicators for the source structure in the schema within Visitor, so there is no way (without generating schemars(title = “FooSpec”) as a metadata) to match the added visitor on the processed object param to make modifications. It is possible to add preserve_order feature to schemars and “search” for the property of the structure, as long as the source struct name is mapped to the Schema content.

With generating JsonSchema visitor extensions for more complex scenarios are possible to explore in the future.

Generating schemars attributes

While it is a viable option, such thing is not possible with derive macro, and has to use proc_macro instead, This approach is additionally hiding the updates of derive attributes under the hood, which feels unintuitive, as it performs updates to the macro markers, meant to generate code. Explored in #1621

[codecov]

Codecov Report

Attention: Patch coverage is 20.00000% with 36 lines in your changes missing coverage. Please review.

Project coverage is 75.2%. Comparing base (3ee4ae5) to head (ee96ec4).

Files with missing lines	Patch %	Lines
kube-derive/src/custom_resource.rs	28.2%	23 Missing ⚠️
kube-core/src/validation.rs	0.0%	11 Missing ⚠️
kube-derive/src/lib.rs	0.0%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1649     +/-   ##
=======================================
- Coverage   75.6%   75.2%   -0.3%     
=======================================
  Files         82      83      +1     
  Lines       7405    7450     +45     
=======================================
+ Hits        5591    5600      +9     
- Misses      1814    1850     +36     

Files with missing lines	Coverage Δ
kube/src/lib.rs	88.5% <ø> (ø)
kube-derive/src/lib.rs	0.0% <0.0%> (ø)
kube-core/src/validation.rs	0.0% <0.0%> (ø)
kube-derive/src/custom_resource.rs	75.2% <28.2%> (-7.3%)	⬇️

---- 🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[clux]

some comments and questions. i think this is a pretty cool approach.

[clux]

Interestingly, I see these ones in the generated docs under https://github.com/kube-rs/k8s-pb/blob/ce4261fb52266f05cd7a06dbb8f4c0fcaa41c06a/k8s-pb/src/apiextensions_apiserver/pkg/apis/apiextensions/v1/mod.rs#L736 but because of bad go enum usage it's just a doc comment :(

[Danil-Grigorev]

Would not pretend I saw it being generated, but yeah, missing enum is better to have :). Maybe worth adding From/Into conversion for ensuring compatibility.

[clux]

are these doc strings taken from anywhere?

[clux]

naming wise, possibly this introduces an confusion possiblities with garde::Validate - which we do advocate for

[clux]

pretty cool that you can do a full struct here like this. does it support shorthands as well if we were to create builders on Rule? could lead to:

#[validated(rule = Rule::rule("self != ''").message("failure message")]

possibly

[Danil-Grigorev]

it is possible, including something simple such as #[validated(rule = “self != ‘’”.into())].

[clux]

note to self; this may perhaps be churned through a cel crate against an object to perform validation client side. should investigate this later.

[clux]

Is the idea here now is to replace JsonSchema with ValidateSchema whenever we need to use rule annotations? I see it's not a universal replacement, the sub-structs still use JsonSchema.

[Danil-Grigorev]

It is a wrapper on top of the JsonSchema implementing it. Crate path for the JsonSchema can be overridden, same as in the CustomResource trait. Rule is one of the keywords which can be used here, there maybe more in the future, since the implementation is extensible and will not be affected by schemars versions. Sub-schemas don’t need ValidateSchema, if a rule is not used there. Implementing this would require to blanket implement the ValidateSchema (trait, a new one) for all standard types, and since we only modify “this current schema I see now” I tried to keep it simple, and it also was the only path forward.

[Danil-Grigorev]

I moved towards implementing option 3 from comment, so current changes are direct descendants of this (and the fact that I realized how to work around unordered properties in the schema)