Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update config model docs #15608

Merged
merged 3 commits into from
Aug 17, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 36 additions & 31 deletions docs/developer/meta/config-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

-----

All integrations use [pydantic](https://github.com/samuelcolvin/pydantic) models as the primary way to validate and interface with configuration.
All integrations use [pydantic](https://github.com/pydantic/pydantic) models as the primary way to validate and interface with configuration.

As config spec data types are based on OpenAPI 3, we [automatically generate](https://github.com/koxudaxi/datamodel-code-generator) the necessary code.
As [config spec](config-specs.md) data types are based on OpenAPI 3, we [automatically generate](https://github.com/koxudaxi/datamodel-code-generator) the necessary code.

The models reside in a package named `config_models` located at the root of a check's namespaced package. For example, a new integration named `foo`:

Expand All @@ -26,25 +26,22 @@ foo

There are 2 possible models:

- `SharedConfig` (ID: `shared`) that corresponds to the `init_config` section
- `InstanceConfig` (ID: `instance`) that corresponds to a check's entry in the `instances` section
- `SharedConfig` (ID: `shared`) that corresponds to the `init_config` section that is shared by all instances

All models are defined in `<ID>.py` and are available for import directly under `config_models`.

## Default values

The default values for optional settings are populated in `defaults.py` and are derived from the
[value](../meta/config-specs.md#values) property of config spec options.

The precedence is:

1. the `default` key
2. the `example` key, if it appears to represent a real value rather than an illustrative example and the `type` is a primitive
3. the default value of the `type` e.g. `string` -> `str()`, `object` -> `dict()`, etc.
[value](config-specs.md#values) property of config spec options. The precedence is the `default` key
followed by the `example` key (if it appears to represent a real value rather than an illustrative example
and the `type` is a primitive). In all other cases the default will be `None` and there will be no default
ofek marked this conversation as resolved.
Show resolved Hide resolved
getter function.

## Validation

The validation of fields for every model occurs in 6 stages.
The validation of fields for every model occurs in 3 high-level stages.
ofek marked this conversation as resolved.
Show resolved Hide resolved

### Initial

Expand All @@ -56,46 +53,55 @@ def initialize_<ID>(values: dict[str, Any], **kwargs) -> dict[str, Any]:
If such a validator exists in `validators.py`, then it is called once with the raw config that was supplied by the user.
The returned mapping is used as the input config for the subsequent stages.

### Default value population
### Field

If a field was not supplied by the user nor during the initialization stage, then its default value is
taken from `defaults.py`. This stage is skipped for required fields.
The value of each field goes through the following steps.

### Default field validators
#### Default value population

At this point `pydantic` will parse the values and perform validation of types, etc.
If a field was not supplied by the user nor during the [initialization stage](#initial), then its default value is
taken from `defaults.py`. This stage is skipped for required fields.

### Custom field validators
#### Custom field validators

The contents of `validators.py` are entirely custom and contain functions to perform extra validation if necessary.

```python
def <ID>_<OPTION_NAME>(value: Any, *, field: pydantic.fields.ModelField, **kwargs) -> Any:
def <ID>_<OPTION_NAME>(value: Any, *, field: pydantic.fields.FieldInfo, **kwargs) -> Any:
...
```

Such validators are called for the appropriate field of the proper model if the option was supplied by the user.
Such validators are called for the appropriate field of the proper model. The returned value is used as the
new value of the option for the subsequent stages.

The returned value is used as the new value of the option for the subsequent stages.
!!! note
rtrieu marked this conversation as resolved.
Show resolved Hide resolved
This only occurs if the option was supplied by the user.

### Pre-defined field validators
#### Pre-defined field validators

A new `validators` key under the [value](https://datadoghq.dev/integrations-core/meta/config-specs/#values) property of config
A `validators` key under the [value](https://datadoghq.dev/integrations-core/meta/config-specs/#values) property of config
spec options is considered. Every entry will refer to a relative import path to a [field validator](#custom-field-validators)
ofek marked this conversation as resolved.
Show resolved Hide resolved
under `datadog_checks.base.utils.models.validation` and is executed in the defined order.

The last returned value is used as the new value of the option for the [final](#final) stage.
!!! note
rtrieu marked this conversation as resolved.
Show resolved Hide resolved
This only occurs if the option was supplied by the user.
ofek marked this conversation as resolved.
Show resolved Hide resolved

#### Conversion to immutable types

Every `list` is converted to `tuple` and every `dict` is converted to [types.MappingProxyType](https://docs.python.org/3/library/types.html#types.MappingProxyType).

!!! note
rtrieu marked this conversation as resolved.
Show resolved Hide resolved
A field or nested field would only be a `dict` when it is defined as a mapping with arbitrary keys. Otherwise, it would be a model with its own properties as usual.

### Final

```python
def finalize_<ID>(values: dict[str, Any], **kwargs) -> dict[str, Any]:
def check_<ID>(model: pydantic.BaseModel) -> pydantic.BaseModel:
...
```

If such a validator exists in `validators.py`, then it is called with the cumulative result of all fields.

The returned mapping is used to instantiate the model.
If such a validator exists in `validators.py`, then it is called with the final constructed model. At this point it cannot
ofek marked this conversation as resolved.
Show resolved Hide resolved
be mutated so you may only raise errors.
ofek marked this conversation as resolved.
Show resolved Hide resolved

## Loading

Expand All @@ -116,17 +122,16 @@ class Check(AgentCheck, ConfigMixin):
...
```

It exposes the instantiated `InstanceConfig` model at `self.config` and `SharedConfig` model at `self.shared_config`.
It exposes the instantiated `InstanceConfig` model as `self.config` and `SharedConfig` model as `self.shared_config`.

## Immutability

All generated models are [configured as immutable](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability).
Additionally, every `list` is converted to `tuple` and every `dict` is converted to [immutables.Map](https://github.com/MagicStack/immutables).
In addition to each field being [converted to an immutable type](#conversion-to-immutable-types), all generated models are [configured as immutable](https://docs.pydantic.dev/2.0/usage/models/#faux-immutability).

## Deprecation

Every option marked as deprecated in the config spec will log a warning with information about when it will be removed and what to do.

## Enforcement

A validation command `ddev validate models` runs in our CI. To locally generate the proper files, run `ddev validate models [CHECK] --sync`.
A validation command [`validate models`](../ddev/cli.md#ddev-validate-models) runs in our CI. To locally generate the proper files, run `ddev validate models [INTEGRATION] --sync`.