Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP error 500 #60

Closed
cowegner opened this issue Jan 20, 2025 · 28 comments
Closed

HTTP error 500 #60

cowegner opened this issue Jan 20, 2025 · 28 comments
Labels
question Further information is requested

Comments

@cowegner
Copy link

I am very excited about your project. Valuable addition to Traefik plugins.

I tried to configure it and only got that far. I'm testing with Traefik's dashboard and using a standalone Traefik instance (no docker). I get only "HTTP GET error" displayed on access of the dashboard URL.

Firefox web developer tools give me:

GET https://traefik.<RedactedDomain>/dashboard/

Status 500
Version HTTP/2
Übertragen  181 B (15 B Größe)
Anfrage-Priorität Highest
DNS-Auflösung System

I don't see any signs, that the OIDC plugin is invoked as a middleware at all.
To make sure everything else is OK I also tested using basic-auth: this shows up in the Traefik logs and the dashboard is displayed.
Testing my Keycloak instance (directly, without Traefik) using OpenID Connect Debugger also succeeds without errors.

Middleware configuration (snippet) - taken almost completely from your project:

    middlewares-keycloak-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          Provider:
            Url: "https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealmName>/protocol/openid-connect/auth"
            ClientId: "traefik-dashboard"
            ClientSecret: "<RedactedSecret>"
            # UsePkce: true # Use PKCE if your Provider supports this
          Scopes: ["openid", "profile", "email"]
          Headers:
            - Name: "X-Oidc-Username"
              Value: "{{`{{ .claims.preferred_username }}`}}"
            - Name: "X-Oidc-Subject"
              Value: "sub"
            - Name: "Authorization"
              Value: "{{`Bearer {{ .accessToken }}`}}"
          AuthorizationHeader:
            Name: "CustomAuth"
          AuthorizationCookie:
            Name: "CustomAuth"

Router configuration (snippet):

  routers:
    dashboard:
      entryPoints:
        - websecure
      rule: Host(`traefik.<RedacteDomain>`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      priority: 1
      service: api@internal
      middlewares:
        # Basic auth for tests
        # - middlewares-basic-auth@file
        # Keycloak auth
        - middlewares-keycloak-auth
      # Enforce TLS and apply my "default" TLS options (defined in default-tls@file) 
      tls: {}
@cdanis
Copy link
Contributor

cdanis commented Jan 20, 2025 via email

@sevensolutions
Copy link
Owner

Just one one thing i can see: The Provider URL for Keycloak should only contain everything up to the realm name.
So instead of:

Url: "https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealmName>/protocol/openid-connect/auth"

it should be:

Url: "https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealmName>"

The middleware will then use the OIDC discovery endpoint to detect all the URLs.

@cowegner
Copy link
Author

cowegner commented Jan 20, 2025

Still no joy :-(

Outcome is the same. The URL line now reads

Url: "https://keycloak.DOMAIN.net:8443/realms/REALM"

The callback URL is one thing I haven't understood yet. I know these callback URLS from IdPs, but I am not clear about your docs regarding this line and the Trafik dashboard which has the URL

https://traefik.<RedactedDomain>/dashboard/
and has no logic implemented on /oidc/callback and will not do anything with what comes along the way this URL path.

The middleware must also wrap the /oidc/callback path

What also worries me is that I don't see anything related to your OIDC plugin in the Traefik log. Shouldn't I see anything?

If it helps: I have no problems sending you my whole config without "redacted" changes over a private channel.

@cdanis
Copy link
Contributor

cdanis commented Jan 20, 2025 via email

@cowegner
Copy link
Author

Could you put the path into my config as an example, please.

@cdanis
Copy link
Contributor

cdanis commented Jan 20, 2025

routers:
    callback:
      entryPoints:
        - websecure
      rule: Host(`traefik.<RedacteDomain>`) && PathPrefix(`/oidc`)
      priority: 1
      service: noop@internal
      middlewares:
        # Keycloak auth
        - middlewares-keycloak-auth
      # Enforce TLS and apply my "default" TLS options (defined in default-tls@file) 
      tls: {}

    dashboard:
      entryPoints:
        - websecure
      rule: Host(`traefik.<RedacteDomain>`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      priority: 1
      service: api@internal
      middlewares:
        # Basic auth for tests
        # - middlewares-basic-auth@file
        # Keycloak auth
        - middlewares-keycloak-auth
      # Enforce TLS and apply my "default" TLS options (defined in default-tls@file) 
      tls: {}

Apologies if that's not quite right, I'm on my phone

@cowegner
Copy link
Author

cowegner commented Jan 20, 2025

Nothing to apologize. I'm grateful.

We are moving. With your code line I now get a 404 back instead of HTTP 500. Still clueless :-(

No, sorry. After closing and opening Firefox again it is still 500. But maybe I have to put the callback also in the middleware section?!

I am pretty sure that now it's only a small thing, but I don't get it.

From the pug docs I understand, that the CallbackUri should be part of middleware section, not router. What I believe how it has to be:

    middlewares-keycloak-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          CallbackUri: "https://traefik.<RedactedDomain>/oidc/callback"
          Provider:
            Url: "https://keycloak.<RedacteDomain>:8443/realms/<RedactedRealm>"
            ClientId: "traefik-dashboard"
            ClientSecret: "<RedactedSecret>"
          Scopes: ["openid", "profile", "email"]

http:
  routers:
    dashboard:
      entryPoints:
        - websecure
      rule: Host(`traefik.<RedactedDomain>`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      priority: 1
      service: api@internal
      middlewares:
        - middlewares-keycloak-auth@file
      tls: {}

    callback:
      entryPoints:
        - websecure
      rule: Host(`traefik.<RedactedDomain>`) && PathPrefix(`/oidc`)
      priority: 1
      service: noop@internal
      middlewares:
       - middlewares-keycloak-auth
      tls: {}

In the client in Keycloak "/oidc/callback" is specified as a valid redirect URI as well.

Still the same outcome of HTTP 500, even I additionally specify a "callback router", as @cdanis mentioned.

@astappiev
Copy link
Contributor

What do Traefik logs tell you? There might be some more info about the cause of the error.
It also helped me to check logs of Kanidm (Keycloak in your case), there were hints about which part of the request was wrong.

@cowegner
Copy link
Author

cowegner commented Jan 20, 2025 via email

@sevensolutions
Copy link
Owner

sevensolutions commented Jan 21, 2025

@cowegner I've tried the same setup to secure the traefik dashboards.
Here is my working config:

http:
  middlewares:
    oidc-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          Provider:
            UrlEnv: "PROVIDER_URL"
            ClientIdEnv: "CLIENT_ID"
            ClientSecretEnv: "CLIENT_SECRET"
            UsePkce: false
            ValidateAudience: false
          SessionCookie:
            Secure: false

  routers:
    traefik-dashboard:
      entryPoints: ["web"]
      rule: "Host(`dashboard.127.0.0.1.sslip.io`)"
      service: dashboard@internal
      middlewares: ["oidc-auth@file"]
    traefik-dashboard-api:
      entryPoints: ["web"]
      rule: "Host(`dashboard.127.0.0.1.sslip.io`) && PathPrefix(`/api`)"
      service: api@internal
      middlewares: ["oidc-auth@file"]
    traefik-dashboard-auth: # Not really needed, because the first router would also include that
      entryPoints: ["web"]
      rule: "Host(`dashboard.127.0.0.1.sslip.io`) && PathPrefix(`/oidc/callback`)"
      service: noop@internal
      middlewares: ["oidc-auth@file"]

A few things to mention here:

  1. Make sure you're using HTTPS or set SessionCookie.Secure: false, as i've done in this sample. Otherwise the cookie will not be set correctly. You'll see that in the Browser debugger. Another sign for this is if you get stuck in an endless redirect loop.
  2. For Keycloak it may be necessary to turn of audience validation as shown in the sample.
  3. The last router is not really needed in this example, because the first one would already include this route.
    The important thing is that there must be a router, which routes the /oidc/callback (or whatever you specify for CallbackUri) to the same middleware.

If the middleware is loaded correctly, you should see a similar log output like this:

traefik-1  | 2025-01-21 17:23:28 [INFO] [traefik-oidc-auth] Loading Configuration...
traefik-1  | 2025-01-21 17:23:28 [INFO] [traefik-oidc-auth] Provider Url: https://login.REDACTED.com/realms/REDACTED
traefik-1  | 2025-01-21 17:23:28 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
traefik-1  | 2025-01-21 17:23:28 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
traefik-1  | 2025-01-21 17:23:28 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
traefik-1  | 2025-01-21 17:23:28 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  false true default 0}
traefik-1  | 2025-01-21 17:23:28 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...

@cowegner
Copy link
Author

cowegner commented Jan 21, 2025

This is really weird. I completely replaced my Traefik config with a test one, based on your config. I omitted TLS at all and am now accessing the dashboard URL using http. I run Firefox as a test browser, so I deleted everything: cookies, etc. In the following I have replaced my domain name using X instead of .

Still I get the same message calling http://traefik.X/dashboard/: HTTP GET error in Browser window, Firefox dev shows HTTP error 500.

Here is my complete config.

My static Traefik config now - traefik.yaml:

# Global
global:
  checkNewVersion: true
  sendAnonymousUsage: false

serversTransport:
  # Set this to false for TESTs to skip SSL certificate verification
  insecureSkipVerify: false

# EntryPoints into Traefik
entryPoints:
  web:
    address: :80

# API and dashboard service
# On/Off must be defined in static config
api:
  dashboard: true
  debug: true

# Providers
providers:
  file:
    directory: "/etc/traefik/dynamic"
    watch: true
    # If debugging is needed
    # Also see logging section
    debugLogGeneratedTemplate: true
  providersThrottleDuration: "10"

# Logging
log:
  level: TRACE
  filePath: "/var/log/traefik/traefik.log"
# Access Log
accessLog:
  filePath: "/var/log/traefik/traefik-access.log"

# 3rd party plugins
# see https://plugins.traefik.io/plugins
experimental:
  plugins:
    traefik-oidc-auth:
      moduleName: "github.com/sevensolutions/traefik-oidc-auth"
      version: "v0.5.0"

My only dynamic config now - router-dashboard.yaml:

http:
  middlewares:
    oidc-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          Provider:
            Url: "https://keycloak.X:8443/realms/<REALM>"
            ClientId: "traefik-dashboard"
            ClientSecret: "<SECRET>"
            UsePkce: false
            ValidateAudience: false
          SessionCookie:
            Secure: false

  routers:
    dashboard:
      entryPoints:
        - web
      rule: "Host(`traefik.X`) && PathPrefix(`/dashboard`)"
      service: dashboard@internal
      middlewares:
        - oidc-auth@file
    api:
      entryPoints:
        - web
      rule: "Host(`traefik.X`) && PathPrefix(`/api`)"
      service: api@internal
      middlewares:
        - oidc-auth@file
    auth:
      entryPoints:
        - web
      rule: "Host(`traefik.X`) && PathPrefix(`/oidc/callback`)"
      service: noop@internal
      middlewares:
        - oidc-auth@file
        

In /var/log/traefik/traefik.log I see among others:

Traefik version 3.3.2 built on 2025-01-14T15:52:24Z version=3.3.2
<...>

2025-01-21T19:15:41+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:241 > Loading plugins... plugins=["traefik-oidc-auth"]
2025-01-21T19:15:41+01:00 DBG github.com/traefik/traefik/v3/pkg/plugins/plugins.go:30 > Loading of plugin: traefik-oidc-auth: github.com/sevensolutions/[email protected]
2025-01-21T19:15:41+01:00 DBG github.com/hashicorp/[email protected]/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/download/github.com/sevensolutions/traefik-oidc-auth/v0.5.0
2025-01-21T19:15:41+01:00 DBG github.com/hashicorp/[email protected]/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/validate/github.com/sevensolutions/traefik-oidc-auth/v0.5.0
2025-01-21T19:15:41+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:251 > Plugins loaded. plugins=["traefik-oidc-auth"]
2

<...>

2025-01-21T19:15:42+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=web middlewareName=oidc-auth@file routerName=auth@file
2025-01-21T19:15:42+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=web middlewareName=oidc-auth@file routerName=dashboard@file
2025-01-21T19:15:42+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=web middlewareName=oidc-auth@file routerName=api@file
2025-01-21T19:15:42+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery

I don't see the other lines from your plugin! In particular I don't see "Configuration loaded successfully, starting OIDC Auth middleware"!

@cdanis
Copy link
Contributor

cdanis commented Jan 21, 2025

Do you have access to the stdout of the Traefik process? It seems like the plugin might be logging there, instead of to the log file.

@cdanis
Copy link
Contributor

cdanis commented Jan 21, 2025

Ah, as it turns out this is a limitation of Traefik: traefik/traefik#8204

@cowegner
Copy link
Author

Mmmh, this limitation makes it no fun at all. Thanks for disclosing this @cdanis

@cowegner
Copy link
Author

cowegner commented Jan 22, 2025

Here is the outcome of stdout from Traefik. It's complete up to the first call of the dashboard URL. My assumption, that the request never reached Keyclaok was obviously wrong. It seems to be a certificate error. Keycloak uses a LetsEncrypt (production, not staging) certificate.

2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealm>
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  false true default}
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealm>
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  false true default}
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealm>
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-22 09:20:40 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  false true default}
2025-01-22 09:20:40 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-22 09:20:53 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-22 09:20:53 [ERROR] [traefik-oidc-auth] http-get discovery endpoints - Err: Get "https://keycloak.<RedactedDomain>:8443/realms/<RedactedRealm>/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority
2025-01-22 09:20:53 [ERROR] [traefik-oidc-auth] Error while retrieving discovery document: HTTP GET error
2025-01-22 09:20:53 [ERROR] [traefik-oidc-auth] Error getting oidc discovery: HTTP GET error

Aren't the trusted OS certificates not honored? I can successfully do "curl https://helloworld.letsencrypt.org" from bash.

Nice thing: I'am starting to learn things about Traefik, Keycloak...even the Go language (just kiddin';-). Even from my first contact with Go I could modify the idc.go to make http request - thanks to the comments already in the file:

	// create a http client with configurable options
	// needed to skip certificate verification
	tr := &http.Transport{
	// 	MaxIdleConns:    10,
	//	IdleConnTimeout: 30 * time.Second,
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr}

	// Make HTTP GET request to the OpenID provider's discovery endpoint
	// resp, err := http.Get(wellKnownUrl.String())
	resp, err := client.Get(wellKnownUrl.String())


	if err != nil {
		log(logLevel, LogLevelError, "http-get discovery endpoints - Err: %s", err.Error())
		//return nil, errors.New("HTTP GET error - OIDC well known discovery endpoint; Err: %s", err.Error())
		return nil, errors.New("traefik-oidc-auth middleware - error accessing OIDC well known discovery endpoint:\n" + err.Error())
	}

Also the import of "crypto/tls" has been nessecary.

Now a browser reload shows a page from Keycloak: Invalid parameter: redirect_uri

@sevensolutions
Copy link
Owner

Ah, as it turns out this is a limitation of Traefik: traefik/traefik#8204

Thx @cdanis, i forgot about that. Iam always using docker to run my traefik instances...

@cowegner the TLS issue seems to be related to #54.
I will take a look on this on the weekend.

@cdanis
Copy link
Contributor

cdanis commented Jan 26, 2025

@cowegner the TLS issue should be very resolve-able for you with v0.6.0.

If you are getting that error from Keycloak still, you should double- and triple-check the redirect URI as you configured it in your Keycloak realm.

@cowegner
Copy link
Author

cowegner commented Jan 28, 2025

Alas, even with v0.6.0 there is still the (an?) error. IMHO, we are not reaching the redirect URL at this stage.

I did two runs:

  • 1st: httpS in Keycloak for everything enforced
  • 2nd: plain HTTP in Keycloak

1st (HTTPS) give me:

2025-01-28 08:14:52 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-28 08:14:52 [ERROR] [traefik-oidc-auth] http-get discovery endpoints - Err: Get "https://keycloak.DOMAIN.net:8443/realms/REALM/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority
2025-01-28 08:14:52 [ERROR] [traefik-oidc-auth] Error while retrieving discovery document: HTTP GET error
2025-01-28 08:14:52 [ERROR] [traefik-oidc-auth] Error getting oidc discovery: HTTP GET error

2nd (HTTP) gives me:

2025-01-28 08:18:37 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-28 08:18:37 [ERROR] [traefik-oidc-auth] http-get discovery endpoints - Err: Get "http://keycloak.DOMAIN.net:8443/realms/REALM/.well-known/openid-configuration": EOF
2025-01-28 08:18:37 [ERROR] [traefik-oidc-auth] Error while retrieving discovery document: HTTP GET error
2025-01-28 08:18:37 [ERROR] [traefik-oidc-auth] Error getting oidc discovery: HTTP GET error

Calling the URL "https://keycloak.DOMAIN.net:8443/realms/REALM/.well-known/openid-configuration" in Firefox (calling the URL with http only, the response is still redirected to https) gives me correct OIDC discovery JSON output.

@sevensolutions
Copy link
Owner

@cowegner hmm... can you please show me the log output from the beginning of the plugin?
Does it say Failed to append CA bundle. Using system certificates only. maybe?

@cowegner
Copy link
Author

Interesting: I don't see such a line nor any similar.

Here's the console output, as well as traefik.log, which includes my whole config.

root@traefik:~# /usr/bin/traefik --configFile=/etc/traefik/traefik.yaml --log true --log.level=TRACE
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.REDACTED.net:8443/realms/REDACTED
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  true true default 0}
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.REDACTED.net:8443/realms/REDACTED
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  true true default 0}
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.REDACTED.net:8443/realms/REDACTED
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-28 11:29:07 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  true true default 0}
2025-01-28 11:29:07 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-28 11:29:20 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-28 11:29:20 [ERROR] [traefik-oidc-auth] http-get discovery endpoints - Err: Get "https://keycloak.REDACTED.net:8443/realms/REDACTED/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority
2025-01-28 11:29:20 [ERROR] [traefik-oidc-auth] Error while retrieving discovery document: HTTP GET error
2025-01-28 11:29:20 [ERROR] [traefik-oidc-auth] Error getting oidc discovery: HTTP GET error
^Croot@traefik:~#
2025-01-28T11:29:06+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:107 > Traefik version 3.3.2 built on 2025-01-14T15:52:24Z version=3.3.2
2025-01-28T11:29:06+01:00 DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:114 > Static configuration loaded [json] staticConfiguration={"accessLog":{"addInternals":true,"fields":{"defaultMode":"keep","headers":{"defaultMode":"drop"},"names":{"StartUTC":"drop"}},"filePath":"/var/log/traefik/traefik-access.log","filters":{},"format":"common"},"api":{"basePath":"/","dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"letsencrypt":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"REDACTED","resolvers":["ns1.REDACTED.de:53","ns2.REDACTED.de:53"]},"email":"[email protected]","keyType":"RSA4096","storage":"/etc/traefik/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"observability":{"accessLogs":true,"metrics":true,"tracing":true},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https","to":"websecure"}}},"http2":{"maxConcurrentStreams":250},"observability":{"accessLogs":true,"metrics":true,"tracing":true},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"observability":{"accessLogs":true,"metrics":true,"tracing":true},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"experimental":{"plugins":{"traefik-oidc-auth":{"moduleName":"github.com/sevensolutions/traefik-oidc-auth","settings":{},"version":"v0.6.0"}}},"global":{"checkNewVersion":true},"log":{"filePath":"/var/log/traefik/traefik.log","format":"common","level":"TRACE"},"providers":{"file":{"debugLogGeneratedTemplate":true,"directory":"/etc/traefik/dynamic","watch":true},"providersThrottleDuration":"10s"},"serversTransport":{"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2025-01-28T11:29:06+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:633 > 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/

2025-01-28T11:29:06+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:241 > Loading plugins... plugins=["traefik-oidc-auth"]
2025-01-28T11:29:06+01:00 DBG github.com/traefik/traefik/v3/pkg/plugins/plugins.go:30 > Loading of plugin: traefik-oidc-auth: github.com/sevensolutions/[email protected]
2025-01-28T11:29:06+01:00 DBG github.com/hashicorp/[email protected]/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/download/github.com/sevensolutions/traefik-oidc-auth/v0.6.0
2025-01-28T11:29:07+01:00 DBG github.com/hashicorp/[email protected]/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/validate/github.com/sevensolutions/traefik-oidc-auth/v0.6.0
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:251 > Plugins loaded. plugins=["traefik-oidc-auth"]
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"debugLogGeneratedTemplate":true,"directory":"/etc/traefik/dynamic","watch":true}
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic/default-middlewares.yaml
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic/default-tls.yaml
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic/router-dashboard.yaml
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic/router-default-noop.yaml
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:572 > Template content: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-middlewares@file
#
# The middlewares SHOULD be tested by Mozilla 
# Default middlewares definition for HTTP router
# - can be referenced in router configurations
# - can be overwritten or supplemented by definitions in router configurations
http:
  middlewares:
    # Traefik local auth not as default, only for tests of selected services such as dashboard
    # Basic authentication using username and password - should be replaced by e.g. oAuth/OIDC
    middlewares-basic-auth:
      basicAuth:
        users:
          - "admin:REDACTED"
        headerField: X-WebAuth-User
    
    # Keycloak auth as default
    middlewares-keycloak-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          #CallbackUri: "https://traefik.REDACTED.net/oidc/callback"
          Provider:
            Url: "https://keycloak.REDACTED.net:8443/realms/REDACTED"
            # Test - http, no TLS
            #Url: "http://keycloak.REDACTED.net:8443/realms/REDACTED"
            ClientId: "traefik-dashboard"
            ClientSecret: "REDACTED"
            UsePkce: false
            ValidateAudience: false
            # SessionCookie:
            #   Secure: false
          # Scopes: ["openid", "profile", "email"]
          # Headers:
          #   - Name: "X-Oidc-Username"
          #     Value: "{{`{{ .claims.preferred_username }}`}}"
          #   - Name: "X-Oidc-Subject"
          #     Value: "sub"
          #   - Name: "Authorization"
          #     Value: "{{`Bearer {{ .accessToken }}`}}"
          # AuthorizationHeader:
          #   Name: "CustomAuth"
          # AuthorizationCookie:
          #   Name: "CustomAuth"

    # Rate Limit
    middlewares-rate-limit:
      rateLimit:
        average: 100
        period: 1m
        burst: 100

    # IP Allowlist
    middlewares-ipallowlist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.0.0/16"

    # Default chain - should be applied to any router
    middlewares-REDACTED-default-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist

    # Default basic auth chain - should be applied to test sensitive routers like Traefik dashboard as long as there are no more advanced auth mechanisms
    middlewares-REDACTED-default-basicauth-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist
          - middlewares-basic-auth

    # Default Auth chain using Keycloak - should be applied to test sensitive routers like Traefik dashboard as long as there are no more advanced auth mechanisms
    middlewares-REDACTED-default-auth-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist
          - middlewares-keycloak-auth
 filename=default-middlewares.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:573 > Rendering results: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-middlewares@file
#
# The middlewares SHOULD be tested by Mozilla 
# Default middlewares definition for HTTP router
# - can be referenced in router configurations
# - can be overwritten or supplemented by definitions in router configurations
http:
  middlewares:
    # Traefik local auth not as default, only for tests of selected services such as dashboard
    # Basic authentication using username and password - should be replaced by e.g. oAuth/OIDC
    middlewares-basic-auth:
      basicAuth:
        users:
          - "admin:REDACTED"
        headerField: X-WebAuth-User
    
    # Keycloak auth as default
    middlewares-keycloak-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          #CallbackUri: "https://traefik.REDACTED.net/oidc/callback"
          Provider:
            Url: "https://keycloak.REDACTED.net:8443/realms/REDACTED"
            # Test - http, no TLS
            #Url: "http://keycloak.REDACTED.net:8443/realms/REDACTED"
            ClientId: "traefik-dashboard"
            ClientSecret: "REDACTED"
            UsePkce: false
            ValidateAudience: false
            # SessionCookie:
            #   Secure: false
          # Scopes: ["openid", "profile", "email"]
          # Headers:
          #   - Name: "X-Oidc-Username"
          #     Value: "{{ .claims.preferred_username }}"
          #   - Name: "X-Oidc-Subject"
          #     Value: "sub"
          #   - Name: "Authorization"
          #     Value: "Bearer {{ .accessToken }}"
          # AuthorizationHeader:
          #   Name: "CustomAuth"
          # AuthorizationCookie:
          #   Name: "CustomAuth"

    # Rate Limit
    middlewares-rate-limit:
      rateLimit:
        average: 100
        period: 1m
        burst: 100

    # IP Allowlist
    middlewares-ipallowlist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.0.0/16"

    # Default chain - should be applied to any router
    middlewares-REDACTED-default-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist

    # Default basic auth chain - should be applied to test sensitive routers like Traefik dashboard as long as there are no more advanced auth mechanisms
    middlewares-REDACTED-default-basicauth-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist
          - middlewares-basic-auth

    # Default Auth chain using Keycloak - should be applied to test sensitive routers like Traefik dashboard as long as there are no more advanced auth mechanisms
    middlewares-REDACTED-default-auth-chain:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-ipallowlist
          - middlewares-keycloak-auth
 filename=default-middlewares.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:572 > Template content: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-tls@file
#
# Traefik dynamic config: default-tls@file
#
# Default TLS options for HTTP router
# - can be referenced in router configurations
# - can be overwritten or supplemented by definitions in router configurations
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: letsencrypt
        domain:
          main: "REDACTED.net"
          sans:
            - "*.REDACTED.net"
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256   # TLS 1.2 PFS, BSI 2030+, Traefik invalid
        # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384     # BSI 2025  
        # - TLS_ECDHE_ECDSA_WITH_AES_128_CCM          # BSI 2030+, Traefik invalid
        #  - TLS_ECDHE_ECDSA_WITH_AES_256_CCM         # BSI 2030+, Traefik invalid
        # - TLS_DHE_RSA_WITH_AES_128_CBC_SHA256       # BSI 2025
        # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256     # BSI 2025
        # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384     # BSI 2025
        # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305      # TLS 1.2 - not BSI
        - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_AES_128_GCM_SHA256                      # TLS 1.3, BSI 2030+
        - TLS_AES_256_GCM_SHA384                      # TLS 1.3, BSI 2030+
        # - TLS_CHACHA20_POLY1305_SHA256              # TLS 1.3 - not BSI
      curvePreferences:
        - CurveP384                                   # IANA 24 - BSI 2030+
        - CurveP521                                   # IANA 25 - BSI 2030+
      sniStrict: true
 filename=default-tls.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:573 > Rendering results: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-tls@file
#
# Traefik dynamic config: default-tls@file
#
# Default TLS options for HTTP router
# - can be referenced in router configurations
# - can be overwritten or supplemented by definitions in router configurations
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: letsencrypt
        domain:
          main: "REDACTED.net"
          sans:
            - "*.REDACTED.net"
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256   # TLS 1.2 PFS, BSI 2030+, Traefik invalid
        # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384     # BSI 2025  
        # - TLS_ECDHE_ECDSA_WITH_AES_128_CCM          # BSI 2030+, Traefik invalid
        #  - TLS_ECDHE_ECDSA_WITH_AES_256_CCM         # BSI 2030+, Traefik invalid
        # - TLS_DHE_RSA_WITH_AES_128_CBC_SHA256       # BSI 2025
        # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256     # BSI 2025
        # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384     # BSI 2025
        # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305      # TLS 1.2 - not BSI
        - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_AES_128_GCM_SHA256                      # TLS 1.3, BSI 2030+
        - TLS_AES_256_GCM_SHA384                      # TLS 1.3, BSI 2030+
        # - TLS_CHACHA20_POLY1305_SHA256              # TLS 1.3 - not BSI
      curvePreferences:
        - CurveP384                                   # IANA 24 - BSI 2030+
        - CurveP521                                   # IANA 25 - BSI 2030+
      sniStrict: true
 filename=default-tls.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:572 > Template content: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: router-dashboard@file
#
# Default HTTP Router
# - if no service endpoint is given by client
# - uses the Traefik noop service to blackhole the request instead of returning any disclosing error to the client
http:
  routers:
    dashboard:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/dashboard`)"
      service: api@internal
      middlewares:
        # Default auth: Keycloak
        - middlewares-REDACTED-default-auth-chain@file
        # auth settings for TESTS
        # - middlewares-REDACTED-default-basicauth-chain@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-dynamic-config@file) 
      tls: {}

    api:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/api`)"
      service: api@internal
      middlewares:
        # Default auth: Keycloak
        - middlewares-REDACTED-default-auth-chain@file
        # auth settings for TESTS
        # - middlewares-REDACTED-default-basicauth-chain@file
        # - middlewares-keycloak-auth@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-dynamic-config@file) 
      tls: {}

    callback:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/oidc/callback`)"
      priority: 1
      service: noop@internal
      middlewares:
        # Keycloak auth
       - middlewares-keycloak-auth@file
      # Enforce TLS and apply my "default" TLS options (defined in default-tls@file) 
      tls: {} filename=router-dashboard.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:573 > Rendering results: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: router-dashboard@file
#
# Default HTTP Router
# - if no service endpoint is given by client
# - uses the Traefik noop service to blackhole the request instead of returning any disclosing error to the client
http:
  routers:
    dashboard:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/dashboard`)"
      service: api@internal
      middlewares:
        # Default auth: Keycloak
        - middlewares-REDACTED-default-auth-chain@file
        # auth settings for TESTS
        # - middlewares-REDACTED-default-basicauth-chain@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-dynamic-config@file) 
      tls: {}

    api:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/api`)"
      service: api@internal
      middlewares:
        # Default auth: Keycloak
        - middlewares-REDACTED-default-auth-chain@file
        # auth settings for TESTS
        # - middlewares-REDACTED-default-basicauth-chain@file
        # - middlewares-keycloak-auth@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-dynamic-config@file) 
      tls: {}

    callback:
      entryPoints:
        - websecure
      rule: "Host(`traefik.REDACTED.net`) && PathPrefix(`/oidc/callback`)"
      priority: 1
      service: noop@internal
      middlewares:
        # Keycloak auth
       - middlewares-keycloak-auth@file
      # Enforce TLS and apply my "default" TLS options (defined in default-tls@file) 
      tls: {} filename=router-dashboard.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:572 > Template content: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-router@file
#
# Default HTTP Router
# - if no service endpoint is given by client
# - uses the Traefik noop service to blackhole the request instead of returning any disclosing error to the client
http:
  routers:
    noService:
      entryPoints:
        - websecure
      rule: Host(`as3.REDACTED.net`)
      priority: 1
      service: noop@internal
      middlewares:
        - middlewares-REDACTED-default-chain@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-tls@file) 
      tls: {}
 filename=router-default-noop.yaml providerName=file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:573 > Rendering results: # yaml-language-server: $schema=https://json.schemastore.org/traefik-v3.json
## Alternative: $schema=https://json.schemastore.org/traefik-v3-file-provider.json
#
# Traefik dynamic config: default-router@file
#
# Default HTTP Router
# - if no service endpoint is given by client
# - uses the Traefik noop service to blackhole the request instead of returning any disclosing error to the client
http:
  routers:
    noService:
      entryPoints:
        - websecure
      rule: Host(`as3.REDACTED.net`)
      priority: 1
      service: noop@internal
      middlewares:
        - middlewares-REDACTED-default-chain@file
      # Specifying a tls block should
      # - enforce TLS
      # - apply the "default" TLS options (defined in default-tls@file) 
      tls: {}
 filename=router-default-noop.yaml providerName=file
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}},"redirect-web-to-websecure":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}}},"models":{"traefik":{"observability":{"accessLogs":true,"metrics":true,"tracing":true}},"web":{"observability":{"accessLogs":true,"metrics":true,"tracing":true}},"websecure":{"observability":{"accessLogs":true,"metrics":true,"tracing":true}}},"routers":{"api":{"entryPoints":["traefik"],"priority":9223372036854775806,"rule":"PathPrefix(`/api`)","ruleSyntax":"v3","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":9223372036854775805,"rule":"PathPrefix(`/`)","ruleSyntax":"v3","service":"dashboard@internal"},"debug":{"entryPoints":["traefik"],"priority":9223372036854775806,"rule":"PathPrefix(`/debug`)","ruleSyntax":"v3","service":"api@internal"},"web-to-websecure":{"entryPoints":["web"],"middlewares":["redirect-web-to-websecure"],"priority":9223372036854775806,"rule":"HostRegexp(`^.+$`)","ruleSyntax":"v3","service":"noop@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"middlewares-basic-auth":{"basicAuth":{"headerField":"X-WebAuth-User","users":["admin:REDACTED"]}},"middlewares-ipallowlist":{"ipAllowList":{"sourceRange":["127.0.0.1/32","192.168.0.0/16"]}},"middlewares-keycloak-auth":{"plugin":{"traefik-oidc-auth":{"LogLevel":"DEBUG","Provider":{"ClientId":"traefik-dashboard","ClientSecret":"REDACTED","Url":"https://keycloak.REDACTED.net:8443/realms/REDACTED","UsePkce":"false","ValidateAudience":"false"}}}},"middlewares-rate-limit":{"rateLimit":{"average":100,"burst":100,"period":"1m0s"}},"middlewares-REDACTED-default-auth-chain":{"chain":{"middlewares":["middlewares-rate-limit","middlewares-ipallowlist","middlewares-keycloak-auth"]}},"middlewares-REDACTED-default-basicauth-chain":{"chain":{"middlewares":["middlewares-rate-limit","middlewares-ipallowlist","middlewares-basic-auth"]}},"middlewares-REDACTED-default-chain":{"chain":{"middlewares":["middlewares-rate-limit","middlewares-ipallowlist"]}}},"routers":{"api":{"entryPoints":["websecure"],"middlewares":["middlewares-REDACTED-default-auth-chain@file"],"rule":"Host(`traefik.REDACTED.net`) && PathPrefix(`/api`)","service":"api@internal","tls":{}},"callback":{"entryPoints":["websecure"],"middlewares":["middlewares-keycloak-auth@file"],"priority":1,"rule":"Host(`traefik.REDACTED.net`) && PathPrefix(`/oidc/callback`)","service":"noop@internal","tls":{}},"dashboard":{"entryPoints":["websecure"],"middlewares":["middlewares-REDACTED-default-auth-chain@file"],"rule":"Host(`traefik.REDACTED.net`) && PathPrefix(`/dashboard`)","service":"api@internal","tls":{}},"noService":{"entryPoints":["websecure"],"middlewares":["middlewares-REDACTED-default-chain@file"],"priority":1,"rule":"Host(`as3.REDACTED.net`)","service":"noop@internal","tls":{}}}},"tcp":{},"tls":{"options":{"default":{"alpnProtocols":["h2","http/1.1","acme-tls/1"],"cipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_AES_128_GCM_SHA256","TLS_AES_256_GCM_SHA384"],"clientAuth":{},"curvePreferences":["CurveP384","CurveP521"],"minVersion":"VersionTLS12","sniStrict":true}},"stores":{"default":{"defaultGeneratedCert":{"domain":{"main":"REDACTED.net","sans":["*.REDACTED.net"]},"resolver":"letsencrypt"}}}},"udp":{}} providerName=file
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"letsencrypt","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"REDACTED","resolvers":["ns1.REDACTED.de:53","ns2.REDACTED.de:53"]},"email":"[email protected]","keyType":"RSA4096","storage":"/etc/traefik/letsencrypt/acme.json","store":{}}
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:232 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2025-01-28T11:29:07+01:00 INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:884 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=letsencrypt.acme
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.REDACTED.net,REDACTED.net
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/chain/chain.go:22 > Creating middleware entryPointName=websecure middlewareName=middlewares-REDACTED-default-auth-chain@file middlewareType=Chain routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-keycloak-auth@file routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:33 > Creating middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:57 > Setting up IPAllowLister with sourceRange: [127.0.0.1/32 192.168.0.0/16] entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ratelimiter/rate_limiter.go:49 > Creating middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/extractor.go:45 > Using IPStrategy entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file routerName=dashboard@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/chain/chain.go:22 > Creating middleware entryPointName=websecure middlewareName=middlewares-REDACTED-default-chain@file middlewareType=Chain routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:33 > Creating middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:57 > Setting up IPAllowLister with sourceRange: [127.0.0.1/32 192.168.0.0/16] entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ratelimiter/rate_limiter.go:49 > Creating middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/extractor.go:45 > Using IPStrategy entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file routerName=noService@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/chain/chain.go:22 > Creating middleware entryPointName=websecure middlewareName=middlewares-REDACTED-default-auth-chain@file middlewareType=Chain routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-keycloak-auth@file routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:33 > Creating middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:57 > Setting up IPAllowLister with sourceRange: [127.0.0.1/32 192.168.0.0/16] entryPointName=websecure middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-ipallowlist@file routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ratelimiter/rate_limiter.go:49 > Creating middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/extractor.go:45 > Using IPStrategy entryPointName=websecure middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-rate-limit@file routerName=api@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=middlewares-keycloak-auth@file routerName=callback@file
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for traefik.REDACTED.net with TLS options default entryPointName=websecure
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for as3.REDACTED.net with TLS options default entryPointName=websecure
2025-01-28T11:29:07+01:00 DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:605 > Default ACME certificate generation is not required. ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme tlsStoreName=default
2025-01-28T11:29:20+01:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:84 > Accepting IP 192.168.11.101 middlewareName=middlewares-ipallowlist@file middlewareType=IPAllowLister
2025-01-28T11:29:30+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:51 > I have to go...
2025-01-28T11:29:30+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:52 > Stopping server gracefully
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=web
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=websecure
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=traefik
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:80: use of closed network connection" entryPointName=web
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=web
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
2025-01-28T11:29:30+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=web
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=traefik
2025-01-28T11:29:30+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=websecure
2025-01-28T11:29:30+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:76 > Server stopped
2025-01-28T11:29:30+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:169 > Shutting down

And additionally the traefik-access.log:

192.168.11.101 - - [28/Jan/2025:11:29:20 +0100] "GET /dashboard/ HTTP/2.0" 500 15 "-" "-" 1 "dashboard@file" "-" 12ms

@sevensolutions
Copy link
Owner

@cowegner thank you for the logs.
It looks like you didn't specify your CABundle file.

You're using a self-signed certificate for your keycloak right?
So you need to specify the certificate like this:

middlewares-keycloak-auth:
      plugin:
        traefik-oidc-auth:
          LogLevel: DEBUG
          Provider:
            Url: "https://keycloak.REDACTED.net:8443/realms/REDACTED"
            CABundleFile: "/etc/ssl/yourSelfSignedCertificatePublicKey.pem" # Or whereever it is
            ClientId: "traefik-dashboard"
            ClientSecret: "REDACTED"

Please have a look at the CABundleFile option which is new in v0.6.0.
For a quick test without the certificate you can also set InsecureSkipVerify: true instead of the CABundleFile but i would suggest using the CABundleFile

@cowegner
Copy link
Author

cowegner commented Jan 28, 2025

I didn't specify a CA bundle for I assumed that in this case the System bundle from the underlying OS is used. I run Traefik in a LXC not a docker container and therefore do see the System's CertificatePool.

I do not use a self-signed certificate, but one from LetsEncrypt (production, not staging).

@cowegner
Copy link
Author

OK, tested with InsecureSkipVerify: true

Console:

2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] Loading Configuration...
2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] Provider Url: https://keycloak.REDACTED.net:8443/realms/REDACTED
2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] I will use this URL for callbacks from the IDP: /oidc/callback
2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] Callback URL is relative, will overlay any wrapped host
2025-01-28 17:16:12 [DEBUG] [traefik-oidc-auth] Scopes: openid, profile, email
2025-01-28 17:16:12 [DEBUG] [traefik-oidc-auth] SessionCookie: &{Authorization /  true true default 0}
2025-01-28 17:16:12 [INFO] [traefik-oidc-auth] Configuration loaded successfully, starting OIDC Auth middleware...
2025-01-28 17:16:17 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-28 17:16:17 [INFO] [traefik-oidc-auth] OIDC Discovery successful. AuthEndPoint: https://keycloak.REDACTED.net:8443/realms/REDACTED/protocol/openid-connect/auth
2025-01-28 17:16:17 [WARN] [traefik-oidc-auth] Verifying token: unable to read session cookie: named cookie not present
2025-01-28 17:16:17 [INFO] [traefik-oidc-auth] Redirecting to OIDC provider...
2025-01-28 17:16:17 [DEBUG] [traefik-oidc-auth] AuthorizationEndPoint: https://keycloak.REDACTED.net:8443/realms/REDACTED/protocol/openid-connect/auth
2025-01-28 17:16:20 [INFO] [traefik-oidc-auth] Getting OIDC discovery document...
2025-01-28 17:16:20 [INFO] [traefik-oidc-auth] OIDC Discovery successful. AuthEndPoint: https://keycloak.REDACTED.net:8443/realms/REDACTED/protocol/openid-connect/auth
2025-01-28 17:16:20 [ERROR] [traefik-oidc-auth] exchangeAuthCode: received bad HTTP response from Provider: {"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}

Despite the error, my test user gets a valid token. I can prove by calling https://keycloak.REDACTED.net:8443/realms/REDACTED/account in a second tab in Firefox. Without the previous call to Traefik, I get the Keycloak login page. After calling Traefik and getting the above error in console output, I get shown my Keycloak account info.

Here is the relevant section from traefik.log:

2025-01-28T17:16:20+01:00 ERR fmt/print.go:305 > plugins-storage/sources/gop-795072553/src/github.com/sevensolutions/traefik-oidc-auth/main.go:203:17: panic: github.com/sevensolutions/traefik-oidc-auth.handleCallback(...) module=github.com/sevensolutions/traefik-oidc-auth plugin=plugin-traefik-oidc-auth runtime=
2025-01-28T17:16:20+01:00 ERR fmt/print.go:305 > plugins-storage/sources/gop-795072553/src/github.com/sevensolutions/traefik-oidc-auth/main.go:97:9: panic: github.com/sevensolutions/traefik-oidc-auth.ServeHTTP(...) module=github.com/sevensolutions/traefik-oidc-auth plugin=plugin-traefik-oidc-auth runtime=
2025-01-28T17:16:20+01:00 ERR github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:49 > Recovered from panic in HTTP handler [192.168.11.101:49562 - https://traefik.REDACTED.net/oidc/callback?state=eyJhY3Rpb24iOiJMb2dpbiIsInJlZGlyZWN0X3VybCI6Imh0dHBzOi8vdHJhZWZpay50dXB0aS5uZXQvZGFzaGJvYXJkLyJ9&session_state=9fc0a15a-696a-44f6-b8c3-17c7de94ab4c&iss=https%3A%2F%2Fkeycloak.REDACTED.net%3A8443%2Frealms%2FREDACTED&code=469fdcd6-4462-488d-9332-e7b81d5b4896.9fc0a15a-696a-44f6-b8c3-17c7de94ab4c.9634a8ed-7787-4983-bd59-34db8af0233b]: reflect: call of reflect.Value.Field on zero Value middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T17:16:20+01:00 ERR github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:53 > Stack: goroutine 75 [running]:
github.com/traefik/traefik/v3/pkg/middlewares/recovery.recoverFunc({0x70a4090, 0xc001aaf970}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:52 +0x239
panic({0x536a500?, 0xc002a04b58?})
	runtime/panic.go:785 +0x132
github.com/traefik/yaegi/interp.runCfg.func1()
	github.com/traefik/[email protected]/interp/run.go:226 +0x1ae
panic({0x536a500?, 0xc002a04b58?})
	runtime/panic.go:785 +0x132
github.com/traefik/yaegi/interp.runCfg.func1()
	github.com/traefik/[email protected]/interp/run.go:226 +0x1ae
panic({0x536a500?, 0xc002a04b58?})
	runtime/panic.go:785 +0x132
reflect.Value.Field({0x0?, 0x0?, 0x5084a60?}, 0x0?)
	reflect/value.go:1269 +0xcf
reflect.Value.FieldByIndex({0x0?, 0x0?, 0xf31089?}, {0xc001257ca8?, 0xc0016ac5f0?, 0xb15c520?})
	reflect/value.go:1302 +0x4e
github.com/traefik/yaegi/interp.getPtrIndexSeq.func2(0xc00163e790?)
	github.com/traefik/[email protected]/interp/run.go:2110 +0xca
github.com/traefik/yaegi/interp.runCfg(0xc002a77e00, 0xc00163e790, 0xc002386e90?, 0x616db60?)
	github.com/traefik/[email protected]/interp/run.go:234 +0x265
github.com/traefik/yaegi/interp.genFunctionWrapper.func1.1({0xc00221abd0, 0x2, 0x4?})
	github.com/traefik/[email protected]/interp/run.go:1042 +0x7a5
reflect.Value.call({0x51c2540?, 0xc00221ab70?, 0x0?}, {0x631c0f3, 0x4}, {0xc00221aba0, 0x2, 0x7548015ce108?})
	reflect/value.go:581 +0xca6
reflect.Value.Call({0x51c2540?, 0xc00221ab70?, 0x47bd29?}, {0xc00221aba0?, 0x6296920?, 0xc0013bd301?})
	reflect/value.go:365 +0xb9
github.com/traefik/yaegi/interp.call.func9.2({0xc00221aba0?, 0xc00221ab70?, 0x0?})
	github.com/traefik/[email protected]/interp/run.go:1310 +0x36
github.com/traefik/yaegi/interp.call.func9(0xc003703d90)
	github.com/traefik/[email protected]/interp/run.go:1330 +0xcc5
github.com/traefik/yaegi/interp.runCfg(0xc002a457c0, 0xc003703d90, 0xc002326d10?, 0x616db60?)
	github.com/traefik/[email protected]/interp/run.go:234 +0x265
github.com/traefik/yaegi/interp.genFunctionWrapper.func1.1({0xc00221a090, 0x2, 0x4?})
	github.com/traefik/[email protected]/interp/run.go:1042 +0x7a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
	github.com/traefik/[email protected]/stdlib/go1_22_net_http.go:296
github.com/traefik/traefik/v3/pkg/server/middleware.(*traceablePlugin).ServeHTTP(0x70a2400?, {0x708c4a0?, 0xc001fd75e0?}, 0x8?)
	github.com/traefik/traefik/v3/pkg/server/middleware/plugins.go:55 +0x26
github.com/traefik/traefik/v3/pkg/middlewares/observability.(*middlewareTracing).ServeHTTP(0xc001f80f40, {0x708c4a0, 0xc001fd75e0}, 0xc00291a3c0)
	github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:69 +0x41a
github.com/traefik/traefik/v3/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc001f80f80, {0x708c4a0, 0xc001fd75e0}, 0xc00291a3c0)
	github.com/traefik/traefik/v3/pkg/middlewares/accesslog/field_middleware.go:41 +0xf7
github.com/traefik/traefik/v3/pkg/middlewares/accesslog.InitServiceFields({0x708c4a0, 0xc001fd75e0}, 0xc00291a3c0, {0x7029620, 0xc001f80f80}, 0xc0023ea7c0)
	github.com/traefik/traefik/v3/pkg/middlewares/accesslog/field_middleware.go:77 +0x132
github.com/traefik/traefik/v3/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc001f80fc0, {0x708c4a0, 0xc001fd75e0}, 0xc00291a3c0)
	github.com/traefik/traefik/v3/pkg/middlewares/accesslog/field_middleware.go:39 +0xd9
github.com/traefik/traefik/v3/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0007b0060, {0x708c4a0, 0xc001fd75e0}, 0xc00291a140, {0x7029620, 0xc001f80fc0})
	github.com/traefik/traefik/v3/pkg/middlewares/accesslog/logger.go:280 +0xff8
github.com/traefik/traefik/v3/pkg/server/middleware.(*ObservabilityMgr).BuildEPChain.WrapHandler.func4.1({0x708c4a0?, 0xc001fd75e0?}, 0xc00221a000?)
	github.com/traefik/traefik/v3/pkg/middlewares/accesslog/logger.go:75 +0x35
net/http.HandlerFunc.ServeHTTP(0xc002326750?, {0x708c4a0?, 0xc001fd75e0?}, 0xc00291a000?)
	net/http/server.go:2220 +0x29
github.com/traefik/traefik/v3/pkg/middlewares/capture.Wrap.func1({0x7547ba68e328, 0xc001aaf970}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/middlewares/capture/capture.go:50 +0x174
net/http.HandlerFunc.ServeHTTP(0xc001f81180?, {0x7547ba68e328?, 0xc001aaf970?}, 0x5acbce0?)
	net/http/server.go:2220 +0x29
github.com/traefik/traefik/v3/pkg/muxer/http.(*Muxer).ServeHTTP(0xc000a06f00, {0x7547ba68e328, 0xc001aaf970}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/muxer/http/mux.go:55 +0xa6
github.com/traefik/traefik/v3/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc0022121f0, {0x708c620, 0xc001aaf968}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:36 +0x14c
github.com/traefik/traefik/v3/pkg/middlewares/snicheck.SNICheck.ServeHTTP({{0x7032160?, 0xc0022121f0?}, 0xc00220df50?}, {0x708c620, 0xc001aaf968}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/middlewares/snicheck/snicheck.go:49 +0x163
github.com/traefik/traefik/v3/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0x413665?, {0x708c620, 0xc001aaf968}, 0xc00291a000)
	github.com/traefik/traefik/v3/pkg/middlewares/handler_switcher.go:23 +0x91
github.com/traefik/traefik/v3/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc00099e318, {0x708c620, 0xc001aaf968}, 0xc002903cc0, 0xc00142dab8)
	github.com/traefik/traefik/v3/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x2ee
github.com/traefik/traefik/v3/pkg/server.createHTTPServer.WrapHandler.func6.1({0x708c620?, 0xc001aaf968?}, 0xc002219e90?)
	github.com/traefik/traefik/v3/pkg/middlewares/requestdecorator/request_decorator.go:89 +0x53
net/http.HandlerFunc.ServeHTTP(0xc0007b0240?, {0x708c620?, 0xc001aaf968?}, 0x9?)
	net/http/server.go:2220 +0x29
github.com/traefik/traefik/v3/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc0007b0240, {0x708c620, 0xc001aaf968}, 0xc002903cc0)
	github.com/traefik/traefik/v3/pkg/middlewares/forwardedheaders/forwarded_header.go:201 +0xc2
github.com/traefik/traefik/v3/pkg/server.createHTTPServer.denyFragment.func7({0x708c620, 0xc001aaf968}, 0xc002903cc0)
	github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:761 +0xf6
net/http.HandlerFunc.ServeHTTP(0xc00123442f?, {0x708c620?, 0xc001aaf968?}, 0x477065?)
	net/http/server.go:2220 +0x29
github.com/traefik/traefik/v3/pkg/server.createHTTPServer.AllowQuerySemicolons.func9({0x708c620, 0xc001aaf968}, 0xc002903cc0)
	net/http/server.go:3234 +0x1cf
net/http.HandlerFunc.ServeHTTP(0x5f147e0?, {0x708c620?, 0xc001aaf968?}, 0xc?)
	net/http/server.go:2220 +0x29
github.com/traefik/traefik/v3/pkg/server.createHTTPServer.DisableAutoDetection.func10({0x708c620, 0xc001aaf968}, 0xc002903cc0)
	github.com/traefik/traefik/v3/pkg/middlewares/contenttype/content_type.go:56 +0xc5
net/http.HandlerFunc.ServeHTTP(0x696b495267372b34?, {0x708c620?, 0xc001aaf968?}, 0x5a33775a62793948?)
	net/http/server.go:2220 +0x29
net/http.serverHandler.ServeHTTP({0x55667762376a467a?}, {0x708c620?, 0xc001aaf968?}, 0x6d62596177333352?)
	net/http/server.go:3210 +0x8e
net/http.initALPNRequest.ServeHTTP({{0x70a2400?, 0xc002210d20?}, 0xc000ad3c08?, {0xc0001d2a50?}}, {0x708c620, 0xc001aaf968}, 0xc002903cc0)
	net/http/server.go:3819 +0x231
golang.org/x/net/http2.(*serverConn).runHandler(0x5775584838384a48?, 0x7a30593349737133?, 0x48455a664b465a4f?, 0x4a2f554869577063?)
	golang.org/x/[email protected]/http2/server.go:2475 +0xf5
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 58
	golang.org/x/[email protected]/http2/server.go:2409 +0x21d
 middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-01-28T17:16:59+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:51 > I have to go...
2025-01-28T17:16:59+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:52 > Stopping server gracefully
2025-01-28T17:16:59+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=traefik
2025-01-28T17:16:59+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=web
2025-01-28T17:16:59+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:298 > Waiting 10s seconds before killing connections entryPointName=websecure
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:80: use of closed network connection" entryPointName=web
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=web
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:240 > error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
2025-01-28T17:16:59+01:00 ERR github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:696 > Error while starting server error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
2025-01-28T17:16:59+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=traefik
2025-01-28T17:16:59+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=web
2025-01-28T17:17:00+01:00 DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:153 > Entrypoint closed entryPointName=websecure
2025-01-28T17:17:00+01:00 INF github.com/traefik/traefik/v3/pkg/server/server.go:76 > Server stopped
2025-01-28T17:17:00+01:00 INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:169 > Shutting down

@sevensolutions
Copy link
Owner

I do not use a self-signed certificate, but one from LetsEncrypt (production, not staging).

Interesting... If you're using a Letsencrypt certificate it should be publicly trusted and no CABundle should be needed. So yes, it should use the underlyings OS certificate store... But maybe there is something special going on with LXC containers.

{"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}
This means there is something wrong with the client configuration.

Client Authentication needs to be turned on, but i think thats already true in your case because you've got a client secret.

Image

@cowegner
Copy link
Author

Correct, Client Authentication is already turned on.
BTW: how can I post pictures here?

@sevensolutions
Copy link
Owner

Correct, Client Authentication is already turned on.

Ok. Can you please double check your client id and secret?
Because from the error message Invalid client or Invalid client credentials there must be something wrong here.

Image Image

BTW: how can I post pictures here?

You can just copy/paste them

@sevensolutions sevensolutions added the question Further information is requested label Feb 2, 2025
@cowegner
Copy link
Author

cowegner commented Feb 4, 2025

Success!

In the end it has been the client secret, which I copied at the beginning from Keycloak but must have hit regenerate somewhere during this course. Sorry for that.

Thank you very much!

@sevensolutions
Copy link
Owner

Perfect. Great to hear that its working for you now. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants