-
Notifications
You must be signed in to change notification settings - Fork 2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Now it is possible to get access to the annotations of an Ingress resource from the template for Ingress resources. This allows users to implement custom annotations by modifying the template to insert NGINX configuration based on the presence on an annotation or its value. Additionally, the Ingress name and namespace is also avaiable in the template.
- Loading branch information
Showing
8 changed files
with
314 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Custom Annotations | ||
|
|
||
| Custom annotations enable you to quickly extend the Ingress Controller to support many advanced features of NGINX, such as rate limiting, caching, etc. | ||
|
|
||
| ## What are Custom Annotations | ||
|
|
||
| NGINX Ingress Controller supports a number of annotations that fine tune NGINX configuration (for example, connection timeouts) or enable additional features (for example, JWT validation). The complete list of annotations is available [here](../examples/customization). | ||
|
|
||
| The annotations are provided only for the most common features and use cases, meaning that not every NGINX feature or a customization option is available through the annotations. Additionally, even if an annotation is available, it might not give you the satisfactory level of control of a particular NGINX feature. | ||
|
|
||
| Custom annotations allow you to add an annotation for an NGINX feature that is not available as a regular annotation. In contrast with regular annotations, to add a custom annotation, you don't need to modify the Ingress Controller source code -- just modify the template. Additionally, with a custom annotation, you get full control of how the feature is implemented in NGINX configuration. | ||
|
|
||
| ## Usage | ||
|
|
||
| The Ingress Controller generates NGINX configuration for Ingress resources by executing a configuration template. See [NGINX template](../internal/nginx/templates/nginx.ingress.tmpl) or [NGINX Plus template](../internal/nginx/templates/nginx-plus.ingress.tmpl). | ||
|
|
||
| To support custom annotations, the template has access to the information about the Ingress resource - its *name*, *namespace* and *annotations*. It is possible to check if a particular annotation present in the Ingress resource and conditionally insert NGINX configuration directives at multiple NGINX contexts - `http`, `server`, `location` or `upstream`. Additionally, you can get the value that is set to the annotation. | ||
|
|
||
| Consider the following excerpt from the template, which was extended to support two custom annotations: | ||
|
|
||
| ``` | ||
| # This is the configuration for {{$.Ingress.Name}}/{{$.Ingress.Namespace}} | ||
| {{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}} | ||
| # Insert config for feature A if the annotation is set | ||
| {{end}} | ||
| {{with $value := index $.Ingress.Annotations "custom.nginx.org/feature-b"}} | ||
| # Insert config for feature B if the annotation is set | ||
| # Print the value assigned to the annotation: {{$value}} | ||
| {{end}} | ||
| ``` | ||
|
|
||
| Consider the following Ingress resource and note how we set two annotations: | ||
| ```yaml | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Ingress | ||
| metadata: | ||
| name: example-ingress | ||
| namespace: production | ||
| annotations: | ||
| custom.nginx.org/feature-a: "on" | ||
| custom.nginx.org/feature-b: "512" | ||
| spec: | ||
| rules: | ||
| - host: example.com | ||
| . . . | ||
| ``` | ||
| Assuming that the Ingress Controller is using that customized template, it will generate a config for the Ingress resource that will include the following part, generated by our template excerpt: | ||
| ``` | ||
| # This is the configuration for cafe-ingress/default | ||
|
|
||
| # Insert config for feature A if the annotation is set | ||
|
|
||
|
|
||
|
|
||
| # Insert config for feature B if the annotation is set | ||
| # Print the value assigned to the annotation: 512 | ||
| ``` | ||
|
|
||
| **Notes**: | ||
| * You can customize the template to insert you custom annotations via [custom templates](../examples/custom-templates). | ||
| * The Ingress Controller uses go templates to generate NGINX config. You can read more information about go templates [here](https://golang.org/pkg/text/template/). | ||
|
|
||
| See the examples in the next section that use custom annotations to configure NGINX features. | ||
|
|
||
| ### Custom Annotations with Mergeable Ingress Resources | ||
|
|
||
| A Mergeable Ingress resource consists of multiple Ingress resources - one master and one or several minions. Read more about Mergeable Ingress resources [here](../examples/mergeable-ingress-types). | ||
|
|
||
| If you'd like to use custom annotations with Mergeable Ingress resources, please keep the following in mind: | ||
| * Custom annotations can be used in the Master and in Minions. For Minions, you can access them in the template only when processing locations. | ||
|
|
||
| If you access `$.Ingress` anywhere in the Ingress template, you will get the master Ingress resource. To access a Minion Ingress resource, use `$location.MinionIngress`. However, it is only available when processing locations: | ||
| ``` | ||
| {{range $location := $server.Locations}} | ||
| location {{$location.Path}} { | ||
| {{with $location.MinionIngress}} | ||
| # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}} | ||
| {{end}} | ||
| . . . | ||
| } {{end}} | ||
| ``` | ||
| **Note**: `$location.MinionIngress` is a pointer. When a regular Ingress resource is processed in the template, the value of the pointer is `nil`. Thus, it is important that you check that `$location.MinionIngress` is not `nil` as in the example above using the `with` action. | ||
| * Minions do not inherent custom annotations of the master. | ||
| ## Example | ||
| See the [custom annotations example](../examples/custom-annotations). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # Custom Annotations | ||
|
|
||
| Custom annotations enable you to quickly extend the Ingress Controller to support many advanced features of NGINX, such as rate limiting, caching, etc. | ||
|
|
||
| Let's create a set of custom annotations to support [rate-limiting](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html): | ||
| * `custom.nginx.org/rate-limiting` - enables rate-limiting. | ||
| * `custom.nginx.org/rate-limiting-rate` - configures the rate of rate-limiting, with the default of `1r/s`. | ||
| * `custom.nginx.org/rate-limiting-burst` - configures the maximum bursts size of requests with the default of `3`. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * Read the [custom annotations doc](../../docs/custom-annotations.md) before going through this example first. | ||
| * Read about [custom templates](../custom-templates). | ||
|
|
||
| ## Step 1 - Customize the Template | ||
|
|
||
| Customize the template for Ingress resources to include the logic to handle and apply the annotations. | ||
|
|
||
| 1. Create a ConfigMap file with the customized template (`nginx-config.yaml`): | ||
| ```yaml | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: nginx-config | ||
| namespace: nginx-ingress | ||
| data: | ||
| ingress-template: | | ||
| . . . | ||
| # handling custom.nginx.org/rate-limiting` and custom.nginx.org/rate-limiting-rate | ||
| {{if index $.Ingress.Annotations "custom.nginx.org/rate-limiting"}} | ||
| {{$rate := index $.Ingress.Annotations "custom.nginx.org/rate-limiting-rate"}} | ||
| limit_req_zone $binary_remote_addr zone={{$.Ingress.Namespace}}-{{$.Ingress.Name}}:10m rate={{if $rate}}{{$rate}}{{else}}1r/s{{end}}; | ||
| {{end}} | ||
| . . . | ||
| {{range $server := .Servers}} | ||
| server { | ||
| . . . | ||
| {{range $location := $server.Locations}} | ||
| location {{$location.Path}} { | ||
| . . . | ||
| # handling custom.nginx.org/rate-limiting and custom.nginx.org/rate-limiting-burst | ||
| {{if index $.Ingress.Annotations "custom.nginx.org/rate-limiting"}} | ||
| {{$burst := index $.Ingress.Annotations "custom.nginx.org/rate-limiting-burst"}} | ||
| limit_req zone={{$.Ingress.Namespace}}-{{$.Ingress.Name}} burst={{if $burst}}{{$burst}}{{else}}3{{end}} nodelay; | ||
| {{end}} | ||
| . . . | ||
| ``` | ||
| The customization above consists of two parts: | ||
| * handling the `custom.nginx.org/rate-limiting` and `custom.nginx.org/rate-limiting-rate` annotations in the `http` context. | ||
| * handling the `custom.nginx.org/rate-limiting` and `custom.nginx.org/rate-limiting-burst` annotation in the `location` context. | ||
|
|
||
| **Note**: for the brevity, the unimportant for the example parts of the template are replaced with `. . .`. | ||
|
|
||
| 1. Apply the customized template: | ||
| ``` | ||
| $ kubectl apply -f nginx-config.yaml | ||
| ``` | ||
|
|
||
| 1. If the Ingress Controller fails to parse the customized template, it will attach an error event with the corresponding ConfigMap resource. You can see the events by running: | ||
| ``` | ||
| $ kubectl describe configmap nginx-config -n nginx-ingress | ||
| . . . | ||
| Events: | ||
| Type Reason Age From Message | ||
| ---- ------ ---- ---- ------- | ||
| Normal Updated 12s (x2 over 25s) nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated | ||
| ``` | ||
| In this case, we got the `Updated` event meaning that the template was parsed successfully. | ||
|
|
||
| ### Step 2 - Use Custom Annotations in an Ingress Resource | ||
|
|
||
| 1. Create a file with the following Ingress resource (`cafe-ingress.yaml`) and use the custom annotations to enable rate-limiting: | ||
| ```yaml | ||
| apiVersion: extensions/v1beta1 | ||
| kind: Ingress | ||
| metadata: | ||
| name: cafe-ingress | ||
| annotations: | ||
| kubernetes.io/ingress.class: "nginx" | ||
| custom.nginx.org/rate-limiting: "on" | ||
| custom.nginx.org/rate-limiting-rate: "5r/s" | ||
| custom.nginx.org/rate-limiting-burst: "1" | ||
| spec: | ||
| rules: | ||
| - host: "cafe.example.com" | ||
| http: | ||
| paths: | ||
| - path: /tea | ||
| backend: | ||
| serviceName: tea-svc | ||
| servicePort: 80 | ||
| - path: /coffee | ||
| backend: | ||
| serviceName: coffee-svc | ||
| servicePort: 80 | ||
| ``` | ||
|
|
||
| 1. Apply the Ingress resource: | ||
| ``` | ||
| $ kubectl apply -f cafe-ingress.yaml | ||
| ``` | ||
|
|
||
| 1. Since it is possible that the value we put in `custom.nginx.org/rate-limiting-rate` or `custom.nginx.org/rate-limiting-burst` annotation might be considered invalid by NGINX, make sure to run the following command to check if the configuration for the Ingress resource was successfully applied. As with the ConfigMap resource, in case of an error, the Ingress Controller will attach an error event to the Ingress resource: | ||
| ``` | ||
| $ kubectl describe ingress cafe-ingress | ||
| Events: | ||
| Type Reason Age From Message | ||
| ---- ------ ---- ---- ------- | ||
| Normal AddedOrUpdated 2m nginx-ingress-controller Configuration for default/cafe-ingress was added or updated | ||
| ``` | ||
| In this case, the config was successfully applied. | ||
|
|
||
| ### Step 3 -- Take a Look at the Generated NGINX Config | ||
|
|
||
| Take a look at the generated config for the cafe-ingress Ingress resource to see how the rate-limiting feature is enabled: | ||
| ``` | ||
| $ kubectl exec <nginx-ingress-pod> -n nginx-ingress -- cat /etc/nginx/conf.d/default-cafe-ingress.conf | ||
| ``` | ||
|
|
||
| ```nginx | ||
| # configuration for default/cafe-ingress | ||
| . . . | ||
| limit_req_zone $binary_remote_addr zone=default-cafe-ingress:10m rate=5r/s; | ||
| server { | ||
| listen 80; | ||
| . . . | ||
| location /tea { | ||
| limit_req zone=default-cafe-ingress burst=1 nodelay; | ||
| . . . | ||
| } | ||
| location /coffee { | ||
| limit_req zone=default-cafe-ingress burst=1 nodelay; | ||
| . . . | ||
| } | ||
| . . . | ||
| } | ||
| ``` | ||
| **Note**: the unimportant parts are replaced with `. . .`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.