Problem creating ingress for Server

[mwoodpatrick]

I'm using k3d with an nginx ingress controller and trying to setup an ingress for argo workflows.

I applied the ingress below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    cert-manager.io/cluster-issuer: "selfsigned-issuer"
  name: argo-workflows-ingress
  namespace: argo
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - argo.westie.dev.to
    secretName: argo-ingress-tls
  rules:
    - host: argo.westie.dev.to
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: argo-server
              port:
                number: 2746

When I describe the created ingress I get:

k -n argo describe ing argo-workflows-ingress
Name:             argo-workflows-ingress
Labels:           <none>
Namespace:        argo
Address:          172.19.0.3,172.19.0.4,172.19.0.5,172.19.0.6
Ingress Class:    nginx
Default backend:  <default>
TLS:
  argo-ingress-tls terminates argo.westie.dev.to
Rules:
  Host                Path  Backends
  ----                ----  --------
  argo.westie.dev.to
                      /   argo-server:2746 (10.42.1.235:2746)
Annotations:          cert-manager.io/cluster-issuer: selfsigned-issuer
                      ingress.kubernetes.io/ssl-redirect: false
Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  CreateCertificate  5m9s                 cert-manager-ingress-shim  Successfully created Certificate "argo-ingress-tls"
  Normal  Sync               5m1s (x2 over 5m9s)  nginx-ingress-controller   Scheduled for sync

However when I access the location:

https://argo.westie.dev.to/

I get a 502 bad gateway error.

I have several other apps deployed on this cluster with very similar ingress specifications which work perfectly well.

I can access using a port-forward

Any suggestions as to what I'm doing wrong or best way to debug would be most appreciated.

[cyberslot]

I confronted with the same problem (on GKE) and after some extensive research realized that a lot of people struggled with that topic. So I decided to give a try to Gateway API instead (NGINX Gateway Fabric) and after countless of attempts/permutations I still have no working solution on the table, in contrast to ArgoCD, where both Ingress and Gateway API implementations are achievable with ease.

[vaboston]

I noticed some differences on the Server Deployment server file:

• environment variable BASE_HREF: /
• container args different

I haven't tested to make sure because it works with helm

[agilgur5]

• environment variable BASE_HREF: /

That's the default, so that has no semantic difference

• container args different

Because they're configurable in Helm, so that is expected too

[terrytangyuan]

Hi @ognyan-lazarov-cloudoffice,

I'd like to apologize on behalf of the Argo project for the accusation made against you in this thread. I do not agree with the accusation or the way it has been handled.

I will hide the accusation on this thread now to keep it on topic.

[ognyan-lazarov-cloudoffice]

Hi @ognyan-lazarov-cloudoffice,

I'd like to apologize on behalf of the Argo project for the accusation made against you in this thread. I do not agree with the accusation or the way it has been handled.

I will hide the accusation on this thread now to keep it on topic.

Thanks for that, but you have nothing to apologize for!
Argo project is a great initiative and you all guys are doing really fantastic job!
Keep going in that way.

[air3ijai]

Just deployed Argo Workflows in Kubernetes using argo-workflows helm chart and used regular settings for OAuth2-Proxy

  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
      nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
    tls:
    - hosts:
      - ${hostname}
      secretName: ${hostname}-tls
    hosts:
    - ${hostname}
    paths: ["/"]
    pathType: Prefix

And got 502 Bad Gateway after OAuth2-Proxy authentications passed.

1. Checked Ingress logs and observer that it is communicating with the Argo server via HTTPS.

2. Looking at the docs - https://argo-workflows.readthedocs.io/en/latest/argo-server

   Since v3.0 the Argo Server listens for HTTPS requests, rather than HTTP.

3. Checking the Argo server

   kubectl get pod argo-workflows-server-785699d947-gtqxm -n argo -o=jsonpath='{.spec.containers[].args}'
   

   ["server","--configmap=argo-workflows-controller-configmap","--auth-mode=server","--secure=false","--loglevel","info","--gloglevel","0","--log-format","text"]%                                                       ~ $
   

   "--secure=false"

4. Updated Ingress annotations to use HTTP - Backend Protocol

   ingress:
     annotations:
       nginx.ingress.kubernetes.io/backend-protocol: "HTTP"

   All started to work as expected.

5. Updated charts values

   server:
     secure: true
     ingress:
       annotations:
       nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Now, all works as expected using HTTPS for backend communication.

---

I would assume that the whole story is started because we expect that Argo Server is listening on HTTPS, but chart default value is set to false

  # -- Run the argo server in "secure" mode. Configure this value instead of `--secure` in extraArgs.
  ## See the following documentation for more details on secure mode:
  ## https://argo-workflows.readthedocs.io/en/stable/tls/
  secure: false

At the same time, Ingress documentation point us to use HTTPS, which is default in v3

If TLS is enabled (default in v3.0 and after), the ingress controller must be told that the backend uses HTTPS.