From 27e574b06af9b4654c9126572f45c9b73cb6273a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jason=20Williams=20-=20NGI=D0=98X?= Date: Tue, 4 Jul 2023 07:18:49 -0700 Subject: [PATCH] Small Update to troubleshooting document (#3346) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Updatng troubleshooting document * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updating troubleshooting section guides * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestions from code review Co-authored-by: Luca Comellini Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Luca Comellini Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Luca Comellini Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Luca Comellini Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-configmap-policy.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-configmap-policy.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-virtualserver-virtualserverroute.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-configmap-policy.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-virtualserver-virtualserverroute.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress-controller.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Update docs/content/troubleshooting/troubleshoot-ingress.md Co-authored-by: Alan Dooley Signed-off-by: Jason Williams - NGIИX * Apply suggestions from code review Signed-off-by: Alan Dooley * Restructure, rename and rewrite various troubleshooting documents This commit adds the changes I have made thus far to the troubleshooting section of NGINX Ingress Controller's documentation: there is more to be done yet, but I am pushing these changes for others to see. The product naming is inconsistent, as is the layout of the pages - often, appropriate markdown is not used where it would make sense to, and formatting is otherwise missing. I will continue to update the documentation to fix these instances. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change instances of names, improve grammar and phrasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove test content from draft TransportServer document * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Jason Williams - NGIИX Signed-off-by: Alan Dooley Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Luca Comellini Co-authored-by: Alan Dooley Co-authored-by: Alan Dooley --- .../troubleshooting/troubleshoot-common.md | 200 ++++++++++++++++++ .../troubleshoot-configmap-policy.md | 43 ++++ .../troubleshoot-ingress-controller.md | 148 ------------- .../troubleshooting/troubleshoot-ingress.md | 23 ++ .../troubleshoot-transportserver.md | 9 + .../troubleshoot-virtualserver.md | 33 +++ ....md => troubleshooting-app-protect-dos.md} | 67 +++--- ....md => troubleshooting-app-protect-waf.md} | 64 +++--- 8 files changed, 377 insertions(+), 210 deletions(-) create mode 100644 docs/content/troubleshooting/troubleshoot-common.md create mode 100644 docs/content/troubleshooting/troubleshoot-configmap-policy.md delete mode 100644 docs/content/troubleshooting/troubleshoot-ingress-controller.md create mode 100644 docs/content/troubleshooting/troubleshoot-ingress.md create mode 100644 docs/content/troubleshooting/troubleshoot-transportserver.md create mode 100644 docs/content/troubleshooting/troubleshoot-virtualserver.md rename docs/content/troubleshooting/{troubleshooting-with-app-protect-dos.md => troubleshooting-app-protect-dos.md} (62%) rename docs/content/troubleshooting/{troubleshooting-with-app-protect.md => troubleshooting-app-protect-waf.md} (64%) diff --git a/docs/content/troubleshooting/troubleshoot-common.md b/docs/content/troubleshooting/troubleshoot-common.md new file mode 100644 index 0000000000..a192b9fa71 --- /dev/null +++ b/docs/content/troubleshooting/troubleshoot-common.md @@ -0,0 +1,200 @@ +--- +title: "Troubleshooting Common Issues" +date: 2021-07-13T21:01:29-06:00 +description: "This page describes how to troubleshoot common problems with NGINX Ingress Controller." +weight: 100 +draft: false +toc: true +tags: [ "docs" ] +doctypes: ["troubleshooting"] +aliases: + - /content/troubleshooting/troubleshoot-ingress-controller +--- + +## Common Problems + +The table below shows common problems with NGINX Ingress Controller you may encounter and how to address them. The following section explains how to gather additional information, and there is instruction available on fixing specific issues within the troubleshooting section of documentation. + +{{% table %}} +| Problem Area | Symptom | Troubleshooting Method | Common Cause | +|-----|-----|-----|-----| +| Startup | NGINX Ingress Controller fails to start. | Check the logs. | Misconfigured RBAC, a missing default server TLS Secret.| +| Ingress Resource and Annotations | The configuration is not applied | Check the events of the Ingress resource, check the logs, check the generated config. | Invalid values of annotations. | +| VirtualServer and VirtualServerRoute Resources | The configuration is not applied. | Check the events of the VirtualServer and VirtualServerRoutes, check the logs, check the generated config. | VirtualServer or VirtualServerRoute is invalid. | +| Policy Resource | The configuration is not applied. | Check the events of the Policy resource as well as the events of the VirtualServers that reference that policy, check the logs, check the generated config. | Policy is invalid. | +| ConfigMap Keys | The configuration is not applied. | Check the events of the ConfigMap, check the logs, check the generated config. | Invalid values of ConfigMap keys. | +| NGINX | NGINX responds with unexpected responses. | Check the logs, check the generated config, check the live activity dashboard (NGINX Plus only), run NGINX in the debug mode. | Unhealthy backend pods, a misconfigured backend service. | +{{% /table %}} + +## Troubleshooting Methods + +The commands in the next sections make the following assumptions: +* That NGINX Ingress Controller is deployed in the namespace `nginx-ingress`. +* `` is the name of one of the NGINX Ingress Controller pods. + +### Checking NGINX Ingress Controller Logs + +To check NGINX Ingress Controller logs, which include both information from NGINX Ingress Controller and NGINX's access and error logs, run the following command: + +```shell +kubectl logs -n nginx-ingress +``` + +### Checking the Generated Config +For each Ingress/VirtualServer resource, NGINX Ingress Controller generates a corresponding NGINX configuration file in the `/etc/nginx/conf.d folder`. + + Additionally, NGINX Ingress Controller generates the main configuration file `/etc/nginx/nginx.conf`, which includes all the configurations files from `/etc/nginx/conf.d`. The configuration for a VirtualServerRoute resource is located in the configuration file of the VirtualServer that references the resource. + +You can view the content of the main configuration file by running: + +```shell +kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf +``` + +Similarly, you can view the content of any generated configuration file in the `/etc/nginx/conf.d` folder. + +You can also print all NGINX configuration files together: + +```shell +kubectl exec -n nginx-ingress -- nginx -T +``` + +However, this command will fail if any of the configuration files is not valid. + +### Checking the Live Activity Monitoring Dashboard + +The live activity monitoring dashboard shows the real-time information about NGINX Plus and the applications it is load balancing, which is helpful for troubleshooting. To access the dashboard, follow the steps from [here](/nginx-ingress-controller/logging-and-monitoring/status-page). + +### Enabling debugging for NGINX Ingress Controller + +For additional NGINX Ingress Controller debugging, you can enable debug settings to get more verbose logging. + +Increasing the debug log levels for NGINX Ingress Controller will also apply to NGINX itself. + +There are two settings that need to be set to enable more verbose logging for NGINX Ingress Controller: + +1. Command Line Arguments +2. Configmap Settings + +**Command Line Arguments** + +When using `manifest` for deployment, use the command line argument `-nginx-debug` in your deployment or daemonset. + +If you want to increase the verbosity of the NGINX Ingress Controller process, you can also add the `-v` parameter. + +Here is a small snippet of setting these command line arguments in the `args` section of a deployment: + +```yaml +args: + - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config + - -enable-cert-manager + - -nginx-debug + - -v=3 +``` + +**ConfigMap Settings** +You can configure `error-log-level` in the NGINX Ingress Controller `configMap`: + +```yaml +kind: ConfigMap +apiVersion: v1 +metadata: + name: nginx-config + namespace: nginx-ingress +data: + error-log-level: "debug" + ``` + +**Using Helm** + +If you are using `helm`, you can adjust these two settings: + +``` +controller.nginxDebug = true or false +controller.loglevel = 1 to 3 value +``` + +For example, if using a `values.yaml` file: + +```yaml + ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in ConfigMap via `controller.config.entries`. + nginxDebug: true + + ## The log level of the Ingress Controller. + logLevel: 3 +``` + +This is a more complete `values.yaml` file when using `helm`: + +```yaml +controller: + kind: Deployment + nginxDebug: true + logLevel: 3 + annotations: + nginx: ingress-prod + pod: + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9113" + prometheus.io/scheme: http + extraLabels: + env: prod-weset + nginxplus: plus + image: + repository: nginx/nginx-ingress + tag: 2.4.1 + # NGINX Configmap + config: + entries: + error-log-level: "debug" + proxy_connet_timeout: "5s" + http-snippets: | + underscores_in_headers on; + ingressClass: nginx +``` + +By enabling the `nginx-debug` CLI argument and changing the `error-log-level` to `debug`, you can capture more output to use for debugging. + +**NOTE**: It is recommended to only enable `nginx-debug` CLI and the `error-log-level` for debugging purposes. + +#### Example debug NGINX Ingress Controller Output +These logs show some of the additional entries when debugging is enabled for NGINX Ingress Controller. + +```shell +I1026 15:39:03.269092 1 manager.go:301] Reloading nginx with configVersion: 1 +I1026 15:39:03.269115 1 utils.go:17] executing /usr/sbin/nginx-debug -s reload -e stderr +2022/10/26 15:39:03 [notice] 19#19: signal 1 (SIGHUP) received from 42, reconfiguring +2022/10/26 15:39:03 [debug] 19#19: wake up, sigio 0 +2022/10/26 15:39:03 [notice] 19#19: reconfiguring +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF0A420:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: add cleanup: 000056362AF0C318 +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF48230:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF00DE0:4096 +2022/10/26 15:39:03 [debug] 19#19: read: 46, 000056362AF00DE0, 3090, 0 +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF58670:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF12440:4280 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF13500:4280 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF145C0:4280 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5C680:4280 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5D740:4280 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5E800:4280 +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF5F8C0:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF41500:4096 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF638D0:8192 +2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types +2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF658E0:4096 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF668F0:5349 +2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E0, 4096, 0 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF67DE0:4096 +2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E1, 1253, 4096 +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF68DF0:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF6CE00:16384 @16 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF70E10:524288 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AFF0E20:524288 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B070E30:524288 +2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B0F0E40:400280 +``` + +Once you have completed your debugging process, you can change the values back to the original values. diff --git a/docs/content/troubleshooting/troubleshoot-configmap-policy.md b/docs/content/troubleshooting/troubleshoot-configmap-policy.md new file mode 100644 index 0000000000..315a3fc37c --- /dev/null +++ b/docs/content/troubleshooting/troubleshoot-configmap-policy.md @@ -0,0 +1,43 @@ +--- +title: Troubleshooting Policy Resources +description: "This page describes how to troubleshoot NGINX Ingress Controller Policy Resources." +weight: 200 +doctypes: [""] +toc: true +--- + +## Policy Resources + +After you create or update a Policy resource, you can use `kubectl describe` to check whether or not NGINX Ingress Controller accepted the policy: + +```shell +kubectl describe pol webapp-policy + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated +``` + +The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. + +However, the fact that a policy was accepted doesn’t guarantee that the NGINX configuration was successfully applied. + +To verify the configuration applied, check the events of the [VirtualServer and VirtualServerRoute resources](/nginx-ingress-controller/troubleshooting/troubleshoot-virtualserver) that reference the policy. + +## ConfigMap Resources +After you update the ConfigMap resource, you can immediately check if the configuration was successfully applied by NGINX: + +```shell +kubectl describe configmap nginx-config -n nginx-ingress +Name: nginx-config +Namespace: nginx-ingress +Labels: + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Updated 11s (x2 over 26m) nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated +``` + +Similar to *Policies*, the events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. diff --git a/docs/content/troubleshooting/troubleshoot-ingress-controller.md b/docs/content/troubleshooting/troubleshoot-ingress-controller.md deleted file mode 100644 index 7f425859ae..0000000000 --- a/docs/content/troubleshooting/troubleshoot-ingress-controller.md +++ /dev/null @@ -1,148 +0,0 @@ ---- -title: "Troubleshooting Common Issues" -date: 2021-07-13T21:01:29-06:00 -draft: true -description: "This document describes how to troubleshoot problems with the Ingress Controller." -# Assign weights in increments of 100 -weight: 100 -draft: false -toc: true -tags: [ "docs" ] -# Taxonomies -# These are pre-populated with all available terms for your convenience. -# Remove all terms that do not apply. -doctypes: ["troubleshooting"] -docs: "DOCS-619" ---- - -## Overview - -This document describes how to troubleshoot problems with the Ingress Controller. - -## Potential Problems - -The table below categorizes some potential problems with the Ingress Controller you may encounter and suggests how to troubleshoot those problems using one or more methods from the next section. - -{{% table %}} -| Problem Area | Symptom | Troubleshooting Method | Common Cause | -|-----|-----|-----|-----| -| Start | The Ingress Controller fails to start. | Check the logs. | Misconfigured RBAC, a missing default server TLS Secret.| -| Ingress Resource and Annotations | The configuration is not applied | Check the events of the Ingress resource, check the logs, check the generated config. | Invalid values of annotations. | -| VirtualServer and VirtualServerRoute Resources | The configuration is not applied. | Check the events of the VirtualServer and VirtualServerRoutes, check the logs, check the generated config. | VirtualServer or VirtualServerRoute is invalid. | -| Policy Resource | The configuration is not applied. | Check the events of the Policy resource as well as the events of the VirtualServers that reference that policy, check the logs, check the generated config. | Policy is invalid. | -| ConfigMap Keys | The configuration is not applied. | Check the events of the ConfigMap, check the logs, check the generated config. | Invalid values of ConfigMap keys. | -| NGINX | NGINX responds with unexpected responses. | Check the logs, check the generated config, check the live activity dashboard (NGINX Plus only), run NGINX in the debug mode. | Unhealthy backend pods, a misconfigured backend service. | -{{% /table %}} - -## Troubleshooting Methods - -Note that the commands in the next sections make the following assumptions: -* The Ingress Controller is deployed in the namespace `nginx-ingress`. -* `` is the name of one of the Ingress Controller pods. - -### Checking the Ingress Controller Logs - -To check the Ingress Controller logs -- both of the Ingress Controller software and the NGINX access and error logs -- run: -``` -$ kubectl logs -n nginx-ingress -``` - -Controlling the verbosity and format: -* To control the verbosity of the Ingress Controller software logs (from 1 to 4), use the `-v` [command-line argument](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments). For example, with `-v=3` you will get more information and the content of any new or updated configuration file will be printed in the logs. -* To control the verbosity and the format of the NGINX logs, configure the corresponding [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource). - -### Checking the Events of an Ingress Resource - -After you create or update an Ingress resource, you can immediately check if the NGINX configuration for that Ingress resource was successfully applied by NGINX: -``` -$ kubectl describe ing cafe-ingress -Name: cafe-ingress -Namespace: default -. . . -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Normal AddedOrUpdated 12s nginx-ingress-controller Configuration for default/cafe-ingress was added or updated -``` -Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. - -### Checking the Events of a VirtualServer and VirtualServerRoute Resources - -After you create or update a VirtualServer resource, you can immediately check if the NGINX configuration for that resource was successfully applied by NGINX: -``` -$ kubectl describe vs cafe -. . . -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated -``` -Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. - -Checking the events of a VirtualServerRoute is similar: -``` -$ kubectl describe vsr coffee -. . . -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Normal AddedOrUpdated 1m nginx-ingress-controller Configuration for default/coffee was added or updated -``` - -### Checking the Events of a Policy Resource - -After you create or update a Policy resource, you can use `kubectl describe` to check whether or not the Ingress Controller accepted the Policy: -``` -$ kubectl describe pol webapp-policy -. . . -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated -``` -Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the policy was successfully accepted. - -However, the fact that a policy was accepted doesn't guarantee that the NGINX configuration was successfully applied. To confirm that, check the events of the VirtualServer and VirtualServerRoute resources that reference that policy. - -### Checking the Events of the ConfigMap Resource - -After you update the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) resource, you can immediately check if the configuration was successfully applied by NGINX: -``` -$ kubectl describe configmap nginx-config -n nginx-ingress -Name: nginx-config -Namespace: nginx-ingress -Labels: -. . . -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Normal Updated 11s (x2 over 26m) nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated -``` -Note that in the events section, we have a `Normal` event with the `Updated` reason, which informs us that the configuration was successfully applied. - -### Checking the Generated Config - -For each Ingress/VirtualServer resource, the Ingress Controller generates a corresponding NGINX configuration file in the `/etc/nginx/conf.d` folder. Additionally, the Ingress Controller generates the main configuration file `/etc/nginx/nginx.conf`, which includes all the configurations files from `/etc/nginx/conf.d`. The config of a VirtualServerRoute resource is located in the configuration file of the VirtualServer that references the resource. - -You can view the content of the main configuration file by running: -``` -$ kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf -``` - -Similarly, you can view the content of any generated configuration file in the `/etc/nginx/conf.d` folder. - -You can also print all NGINX configuration files together: -``` -$ kubectl exec -n nginx-ingress -- nginx -T -``` -However, this command will fail if any of the configuration files is not valid. - -### Checking the Live Activity Monitoring Dashboard - -The live activity monitoring dashboard shows the real-time information about NGINX Plus and the applications it is load balancing, which is helpful for troubleshooting. To access the dashboard, follow the steps from [here](/nginx-ingress-controller/logging-and-monitoring/status-page). - -### Running NGINX in the Debug Mode - -Running NGINX in the [debug mode](https://docs.nginx.com/nginx/admin-guide/monitoring/debugging/) allows us to enable its debug logs, which can help to troubleshoot problems in NGINX. Note that it is highly unlikely that a problem you encounter with the Ingress Controller is caused by a bug in the NGINX code, but it is rather caused by NGINX misconfiguration. Thus, this method is rarely needed. - -To enable the debug mode, set the `error-log-level` to `debug` in the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and use the `-nginx-debug` [command-line argument](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) when running the Ingress Controller. diff --git a/docs/content/troubleshooting/troubleshoot-ingress.md b/docs/content/troubleshooting/troubleshoot-ingress.md new file mode 100644 index 0000000000..c31b473027 --- /dev/null +++ b/docs/content/troubleshooting/troubleshoot-ingress.md @@ -0,0 +1,23 @@ +--- +title: Troubleshooting Ingress Resources +description: "This page describes how to troubleshoot NGINX Ingress Controller Policy Resources." +weight: 300 +doctypes: [""] +toc: true +--- + +## Ingress Resources +After you create or update an Ingress resource, you can immediately check if the NGINX configuration for that Ingress resource was successfully applied by NGINX: + +```shell +kubectl describe ing cafe-ingress +Name: cafe-ingress +Namespace: default + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 12s nginx-ingress-controller Configuration for default/cafe-ingress was added or updated +``` + +The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. diff --git a/docs/content/troubleshooting/troubleshoot-transportserver.md b/docs/content/troubleshooting/troubleshoot-transportserver.md new file mode 100644 index 0000000000..1ed7e07b1d --- /dev/null +++ b/docs/content/troubleshooting/troubleshoot-transportserver.md @@ -0,0 +1,9 @@ +--- +title: Troubleshooting TransportServer Resources +description: "." +weight: 400 +doctypes: [""] +draft: true +--- + +# Troubleshooting TransportServer Resources diff --git a/docs/content/troubleshooting/troubleshoot-virtualserver.md b/docs/content/troubleshooting/troubleshoot-virtualserver.md new file mode 100644 index 0000000000..c74a258f49 --- /dev/null +++ b/docs/content/troubleshooting/troubleshoot-virtualserver.md @@ -0,0 +1,33 @@ +--- +title: Troubleshooting VirtualServer Resources +description: "This page describes how to troubleshoot VirtualServer and VirtualServer Resource Events." +weight: 500 +doctypes: [""] +toc: true +aliases: + - /content/troubleshooting/virtualserver-virtualserverroute +--- +## Inspecting VirtualServer and VirtualServerRoute Resource Events +After creating or updating a VirtualServer resource, you can immediately check if the NGINX configuration for that resource was successfully by using `kubectl describe vs `: + +```shell +kubectl describe vs cafe + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated +``` + +In the above example, we have a `Normal` event with the `AddedOrUpdate` reason, which informs us that the configuration was successfully applied. + +Checking the events of a VirtualServerRoute is similar: + +```shell +kubectl describe vsr coffee + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal AddedOrUpdated 1m nginx-ingress-controller Configuration for default/coffee was added or updated +``` diff --git a/docs/content/troubleshooting/troubleshooting-with-app-protect-dos.md b/docs/content/troubleshooting/troubleshooting-app-protect-dos.md similarity index 62% rename from docs/content/troubleshooting/troubleshooting-with-app-protect-dos.md rename to docs/content/troubleshooting/troubleshooting-app-protect-dos.md index 6ea65ee3d4..ab8bd9a278 100644 --- a/docs/content/troubleshooting/troubleshooting-with-app-protect-dos.md +++ b/docs/content/troubleshooting/troubleshooting-app-protect-dos.md @@ -1,80 +1,83 @@ --- -title: Troubleshooting with NGINX App Protect Dos -description: "This document describes how to troubleshoot problems with the Ingress Controller with the App Protect Dos module enabled." -weight: 2000 +title: Troubleshooting with NGINX App Protect DoS +description: "This document describes how to troubleshoot problems when using NGINX Ingress Controller and the App Protect DoS module." +weight: 600 doctypes: [""] -aliases: -- /app-protect/troubleshooting/ toc: true -docs: "DOCS-620" +aliases: + - /content/troubleshooting/troubleshooting-with-app-protect-dos --- -This document describes how to troubleshoot problems with the Ingress Controller with the App Protect DoS module enabled. - -For general troubleshooting of the Ingress Controller, check the general [troubleshooting](/nginx-ingress-controller/troubleshooting/) documentation. +To troubleshoot other parts of NGINX Ingress Controller, check the [troubleshooting]({{< relref "troubleshooting/troubleshoot-common" >}}) section of the documentation. ## Potential Problems -The table below categorizes some potential problems with the Ingress Controller when App Protect DoS module is enabled. It suggests how to troubleshoot those problems, using one or more methods from the next section. +The table below categorizes some potential problems with NGINX Ingress Controller when the App Protect DoS module is enabled. It suggests how to troubleshoot those problems, using one or more methods from the next section. {{% table %}} |Problem area | Symptom | Troubleshooting method | Common cause | | ---| ---| ---| --- | -|Start | The Ingress Controller fails to start. | Check the Ingress Controller logs. | Misconfigured DosProtectedResource, APDosLogConf or APDosPolicy. | +|Start | NGINX Ingress Controller fails to start. | Check the NGINX Ingress Controller logs. | Misconfigured DosProtectedResource, APDosLogConf or APDosPolicy. | |DosProtectedResource, APDosLogConf, APDosPolicy or Ingress Resource. | The configuration is not applied. | Check the events of the DosProtectedResource, APDosLogConf, APDosPolicy and Ingress Resource, check the Ingress Controller logs. | DosProtectedResource, APDosLogConf or APDosPolicy is invalid. | {{% /table %}} ## Troubleshooting Methods -### Check the Ingress Controller and App Protect DoS logs +### Checking NGINX Ingress Controller and App Protect DoS logs -App Protect DoS logs are part of the Ingress Controller logs when the module is enabled. To check the Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs](/nginx-ingress-controller/troubleshooting/#checking-the-ingress-controller-logs) of the Troubleshooting guide. +App Protect DoS logs are part of the NGINX Ingress Controller logs when the module is enabled. To check the Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs]({{< relref "troubleshooting/troubleshoot-common#checking-nginx-ingress-controller-logs" >}}s) of the Troubleshooting guide. -For App Protect DoS specific logs, look for messages starting with `APP_PROTECT_DOS`, for example: -``` +For App Protect DoS specific logs, look for messages starting with `APP_PROTECT_DOS`, such as: + +```shell 2021/06/14 08:17:50 [notice] 242#242: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 242, "mode": "operational", "mode_changed": true } ``` -### Check events of an Ingress Resource +### Checking Ingress Resource Events -Follow the steps of [Checking the Events of an Ingress Resource]({{< relref "troubleshooting/troubleshoot-ingress-controller.md#checking-the-events-of-an-ingress-resource" >}}). +Follow the steps of [Troubleshooting Ingress Resources]({{< relref "troubleshooting/troubleshoot-ingress" >}}). -### Check events of a VirtualServer Resource +### Checking VirtualServer Resource Events -Follow the steps of [Checking the Events of a VirtualServer Resource]({{< relref "troubleshooting/troubleshoot-ingress-controller.md#checking-the-events-of-a-virtualserver-and-virtualserverroute-resources" >}}). +Follow the steps of [Troubleshooting VirtualServer Resources]({{< relref "troubleshooting/troubleshoot-virtualserver" >}}). -### Check events of DosProtectedResource +### Checking for DoSProtectedResource Events After you create or update an DosProtectedResource, you can immediately check if the NGINX configuration was successfully applied by NGINX: -``` -$ kubectl describe dosprotectedresource dos-protected + +```shell +kubectl describe dosprotectedresource dos-protected Name: dos-protected Namespace: default -. . . + Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 2s nginx-ingress-controller Configuration for default/dos-protected was added or updated ``` + Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. If the DosProtectedResource refers to a missing resource, you should see a message like the following: -``` + +```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 8s nginx-ingress-controller dos protected refers (default/dospolicy) to an invalid DosPolicy: DosPolicy default/dospolicy not found ``` + This can be fixed by adding the missing resource. -### Check events of APDosLogConf +### Checking for APDosLogConf Events After you create or update an APDosLogConf, you can immediately check if the NGINX configuration was successfully applied by NGINX: -``` -$ kubectl describe apdoslogconf logconf + +```shell +kubectl describe apdoslogconf logconf Name: logconf Namespace: default -. . . + Events: Type Reason Age From Message ---- ------ ---- ---- ------- @@ -85,8 +88,9 @@ Note that in the events section, we have a `Normal` event with the `AddedOrUpdat ### Check events of APDosPolicy After you create or update an APDosPolicy, you can immediately check if the NGINX configuration was successfully applied by NGINX: -``` -$ kubectl describe apdospolicy dospolicy + +```shell +kubectl describe apdospolicy dospolicy Name: dospolicy Namespace: default . . . @@ -95,7 +99,8 @@ Events: ---- ------ ---- ---- ------- Normal AddedOrUpdated 2m25s nginx-ingress-controller AppProtectDosPolicy default/dospolicy was added or updated ``` -Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. + +The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. ## Run App Protect DoS in Debug log Mode diff --git a/docs/content/troubleshooting/troubleshooting-with-app-protect.md b/docs/content/troubleshooting/troubleshooting-app-protect-waf.md similarity index 64% rename from docs/content/troubleshooting/troubleshooting-with-app-protect.md rename to docs/content/troubleshooting/troubleshooting-app-protect-waf.md index 2bf9fec20f..5cec4f6480 100644 --- a/docs/content/troubleshooting/troubleshooting-with-app-protect.md +++ b/docs/content/troubleshooting/troubleshooting-app-protect-waf.md @@ -1,19 +1,18 @@ --- -title: Troubleshooting with NGINX App Protect -description: "This document describes how to troubleshoot problems when using NGINX Ingress Controller with the NGINX App Protect WAF module enabled." -weight: 2000 +title: Troubleshooting with NGINX App Protect WAF +description: "This document describes how to troubleshoot problems when using NGINX Ingress Controller and the NGINX App Protect WAF module." +weight: 700 doctypes: [""] -aliases: - - /app-protect/troubleshooting/ toc: true -docs: "DOCS-621" +aliases: + - /content/troubleshooting/troubleshooting-with-app-protect --- -This document describes how to troubleshoot problems with the Ingress Controller with the [App Protect](/nginx-app-protect/) module enabled. +This document describes how to troubleshoot problems with NGINX Ingress Controller with the [App Protect](/nginx-app-protect/) module enabled. -For general troubleshooting of the Ingress Controller, check the general [troubleshooting]({{< relref "troubleshooting/troubleshoot-ingress-controller.md" >}}) documentation. +For general troubleshooting of NGINX Ingress Controller, check the general [troubleshooting]({{< relref "troubleshooting/troubleshoot-common" >}}) documentation. -{{< see-also >}}You can find more troubleshooting tips in the NGINX App Protect WAF [troubleshooting guide](/nginx-app-protect/troubleshooting/) {{< /see-also >}}. +{{< see-also >}} You can find more troubleshooting tips in the NGINX App Protect WAF [troubleshooting guide](/nginx-app-protect/troubleshooting/) {{< /see-also >}}. ## Potential Problems @@ -29,9 +28,9 @@ The table below categorizes some potential problems with the Ingress Controller ## Troubleshooting Methods -### Check the Ingress Controller and App Protect logs +### Check NGINX Ingress Controller and App Protect logs -App Protect logs are part of the Ingress Controller logs when the module is enabled. To check the Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs]({{< relref "troubleshooting/troubleshoot-ingress-controller.md#checking-the-ingress-controller-logs" >}}) of the Troubleshooting guide. +App Protect logs are part of NGINX Ingress Controller logs when the module is enabled. To check NGINX Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs]({{< relref "troubleshooting/troubleshoot-common#checking-the-ingress-controller-logs" >}}) of the Troubleshooting guide. For App Protect specific logs, look for messages starting with `APP_PROTECT`, for example: ``` @@ -40,16 +39,17 @@ For App Protect specific logs, look for messages starting with `APP_PROTECT`, fo ### Check events of an Ingress Resource -Follow the steps of [Checking the Events of an Ingress Resource]({{< relref "troubleshooting/troubleshoot-ingress-controller.md#checking-the-events-of-an-ingress-resource" >}}). +Follow the steps of [Checking the Events of an Ingress Resource]({{< relref "troubleshooting/troubleshoot-ingress" >}}). ### Check events of APLogConf After you create or update an APLogConf, you can immediately check if the NGINX configuration was successfully applied by NGINX: -``` -$ kubectl describe aplogconf logconf + +```shell +kubectl describe aplogconf logconf Name: logconf Namespace: default -. . . + Events: Type Reason Age From Message ---- ------ ---- ---- ------- @@ -60,25 +60,26 @@ Note that in the events section, we have a `Normal` event with the `AddedOrUpdat ### Check events of APPolicy After you create or update an APPolicy, you can immediately check if the NGINX configuration was successfully applied by NGINX: -``` -$ kubectl describe appolicy dataguard-alarm + +```shell +kubectl describe appolicy dataguard-alarm Name: dataguard-alarm Namespace: default -. . . + Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 2m25s nginx-ingress-controller AppProtectPolicy default/dataguard-alarm was added or updated ``` -Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. +The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. -### Replace the policy +### Replace the Policy NOTE: This method only applies if using [external references](/nginx-app-protect/configuration/#external-references) -If items on the external reference change but the spec of the APPolicy remains unchanged (even when re-applying the policy), kubernetes will not detect the update. +If items on the external reference change but the spec of the APPolicy remains unchanged (even when re-applying the policy), Kubernetes will not detect the update. In this case you can force-replace the resource. This will remove the resource and add it again, triggering a reload. For example: -``` +```shell kubectl replace appolicy -f your-policy-manifest.yaml --force ``` @@ -87,24 +88,25 @@ kubectl replace appolicy -f your-policy-manifest.yaml --force NOTE: This method only applies if you're using [external references](/nginx-app-protect/configuration/#external-references) in NGINX App Protect policies. To check what servers host the external references of a policy: -``` -kubectl get appolicy mypolicy -o jsonpath='{.items[*].spec.policy.*.link}' | tr ' ' '\n' +```shell +kubectl get appolicy mypolicy -o jsonpath='{.items[*].spec.policy.*.link}' | tr ' ' '\n' http://192.168.100.100/resources/headersettings.txt ``` You can check the total time a http request takes, in multiple ways eg. using curl: -``` + +```shell curl -w '%{time_total}' http://192.168.100.100/resources/headersettings.txt ``` ## Run App Protect in Debug Mode -When you set the Ingress Controller to use debug mode, the setting also applies to the App Protect WAF module. See [Running NGINX in the Debug Mode]({{< relref "troubleshooting/troubleshoot-ingress-controller.md#running-nginx-in-the-debug-mode" >}}) for instructions. +When you set NGINX Ingress Controller to use debug mode, the setting also applies to the App Protect WAF module. See [Running NGINX in the Debug Mode]({{< relref "troubleshooting/troubleshoot-common.md#running-nginx-in-the-debug-mode" >}}) for instructions. ## Known Issues -When using the Ingress Controller with the App Protect WAF module, the following issues have been reported. The occurrence of these issues is commonly related to a higher number of Ingress Resources with App Protect being enabled in a cluster. +When using NGINX Ingress Controller with the App Protect WAF module, the following issues have been reported. The occurrence of these issues is commonly related to a higher number of Ingress Resources with App Protect being enabled in a cluster. When you make a change that requires NGINX to apply a new configuration, the Ingress Controller reloads NGINX automatically. Without the App Protect WAF module enabled, usual reload times are around 150ms. If App Protect WAF module is enabled and is being used by any number of Ingress Resources, these reloads might take a few seconds instead. @@ -116,15 +118,15 @@ In order to reduce these inconsistencies, we advise that you do not apply change ### NGINX Fails to Start or Reload -The first time the Ingress Controller starts, or whenever there is a change that requires reloading NGINX, the Ingress Controller will verify if the reload was successful. The timeout for this verification is normally 4 seconds. When App Protect is enabled, this timeout is 20 seconds. +The first time NGINX Ingress Controller starts, or whenever there is a change that requires reloading NGINX, NGINX Ingress Controller will verify if the reload was successful. The timeout for this verification is normally 4 seconds. When App Protect is enabled, this timeout increases to 20 seconds. -This timeout should be more than enough to verify configurations. However, when numerous Ingress Resources with App Protect enabled are handled by the Ingress Controller at the same time, you may find that you need to extend the timeout further. Examples of when this might be necessary include: +This timeout should be more than enough to verify configurations. However, when numerous Ingress resources with App Protect enabled are handled by NGINX Ingress Controller at the same time, you may find that you need to extend the timeout further. Examples of when this might be necessary include: - You need to apply a large amount of Ingress Resources at once. -- You are running the Ingress Controller for the first time in a cluster where the Ingress Resources with App Protect enabled are already present. +- You are running NGINX Ingress Controller for the first time in a cluster where the Ingress resources with App Protect enabled are already present. You can increase this timeout by setting the `nginx-reload-timeout` [cli-argument]({{< relref "configuration/global-configuration/command-line-arguments.md#cmdoption-nginx-reload-timeout" >}}). When using the User Defined Signature feature, an update to an `APUserSig` requires more reload time from NGINX Plus compared with the other AppProtect resources. As a consequence, we recommend increasing the `nginx-reload-timeout` to 30 seconds if you're planning to use this feature. -If you are using external references in your Nginx App Protect policies, verify if the servers hosting the referenced resources are available and that their response time is as short as possible (see the Check the Availability of APPolicy External References section). If the references are not available during the Ingress Controller startup, the pod will fail to start. In case the resources are not available during a reload, the reload will fail, and NGINX Plus will use the previous correct configuration. +If you are using external references in your NGINX App Protect policies, verify if the servers hosting the referenced resources are available and that their response time is as short as possible (see the Check the Availability of APPolicy External References section). If the references are not available during NGINX Ingress Controller startup, the pod will fail to start. In case the resources are not available during a reload, the reload will fail, and NGINX Plus will use the previous correct configuration.