Latest next version of Woodpecker server fails to start #2370

rubenelshof · 2023-09-05T06:56:26Z

Component

server

Describe the bug

Woodpecker server fails to successfully start with the latest next version.
It will start up but the container remains unhealthy and eventually fails.

I have switched back to an earlier version next-3bee51dc7a which does work.

System Info

Latest next version.
Because it won't start up I can't get the exact version.

Compose:

    woodpecker:
        image: woodpeckerci/woodpecker-server:next
        networks:
            - dn-dev
        dns:
            - 1.1.1.1
            - 1.0.0.1
        ports:
            - "8502:8000"
            - "8504:9000"
        env_file:
            - "/opt/docker/configs/env-files/woodpecker.env"

Additional context

6:39AM ??? woodpecker/src/github.com/woodpecker-ci/woodpecker/cmd/common/logger.go:99 > LogLevel = debug
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET    /favicon.svg              --> github.com/woodpecker-ci/woodpecker/server/web.redirect.func1 (2 handlers)
[GIN-debug] GET    /favicons/*filepath       --> github.com/woodpecker-ci/woodpecker/server/web.serveFile.func1 (2 handlers)
[GIN-debug] GET    /assets/*filepath         --> github.com/woodpecker-ci/woodpecker/server/web.handleCustomFilesAndAssets.func2 (2 handlers)
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET    /web-config.js            --> github.com/woodpecker-ci/woodpecker/server/web.Config (12 handlers)
[GIN-debug] GET    /logout                   --> github.com/woodpecker-ci/woodpecker/server/api.GetLogout (12 handlers)
[GIN-debug] GET    /login                    --> github.com/woodpecker-ci/woodpecker/server/api.HandleLogin (12 handlers)
[GIN-debug] GET    /authorize                --> github.com/woodpecker-ci/woodpecker/server/api.HandleAuth (12 handlers)
[GIN-debug] POST   /authorize                --> github.com/woodpecker-ci/woodpecker/server/api.HandleAuth (12 handlers)
[GIN-debug] POST   /authorize/token          --> github.com/woodpecker-ci/woodpecker/server/api.GetLoginToken (12 handlers)
[GIN-debug] GET    /metrics                  --> github.com/woodpecker-ci/woodpecker/server/api/metrics.PromHandler.func1 (12 handlers)
[GIN-debug] GET    /version                  --> github.com/woodpecker-ci/woodpecker/server/api.Version (12 handlers)
[GIN-debug] GET    /healthz                  --> github.com/woodpecker-ci/woodpecker/server/api.Health (12 handlers)
[GIN-debug] GET    /api/user                 --> github.com/woodpecker-ci/woodpecker/server/api.GetSelf (13 handlers)
[GIN-debug] GET    /api/user/feed            --> github.com/woodpecker-ci/woodpecker/server/api.GetFeed (13 handlers)
[GIN-debug] GET    /api/user/repos           --> github.com/woodpecker-ci/woodpecker/server/api.GetRepos (13 handlers)
[GIN-debug] POST   /api/user/token           --> github.com/woodpecker-ci/woodpecker/server/api.PostToken (13 handlers)
[GIN-debug] DELETE /api/user/token           --> github.com/woodpecker-ci/woodpecker/server/api.DeleteToken (13 handlers)
[GIN-debug] GET    /api/users                --> github.com/woodpecker-ci/woodpecker/server/api.GetUsers (13 handlers)
[GIN-debug] POST   /api/users                --> github.com/woodpecker-ci/woodpecker/server/api.PostUser (13 handlers)
[GIN-debug] GET    /api/users/:login         --> github.com/woodpecker-ci/woodpecker/server/api.GetUser (13 handlers)
[GIN-debug] PATCH  /api/users/:login         --> github.com/woodpecker-ci/woodpecker/server/api.PatchUser (13 handlers)
[GIN-debug] DELETE /api/users/:login         --> github.com/woodpecker-ci/woodpecker/server/api.DeleteUser (13 handlers)
[GIN-debug] GET    /api/orgs                 --> github.com/woodpecker-ci/woodpecker/server/api.GetOrgs (13 handlers)
[GIN-debug] GET    /api/orgs/lookup/*org_full_name --> github.com/woodpecker-ci/woodpecker/server/api.LookupOrg (12 handlers)
[GIN-debug] GET    /api/orgs/:org_id/permissions --> github.com/woodpecker-ci/woodpecker/server/api.GetOrgPermissions (12 handlers)
[GIN-debug] DELETE /api/orgs/:org_id         --> github.com/woodpecker-ci/woodpecker/server/api.DeleteOrg (14 handlers)
[GIN-debug] GET    /api/orgs/:org_id         --> github.com/woodpecker-ci/woodpecker/server/api.GetOrg (13 handlers)
[GIN-debug] GET    /api/orgs/:org_id/secrets --> github.com/woodpecker-ci/woodpecker/server/api.GetOrgSecretList (13 handlers)
[GIN-debug] POST   /api/orgs/:org_id/secrets --> github.com/woodpecker-ci/woodpecker/server/api.PostOrgSecret (13 handlers)
[GIN-debug] GET    /api/orgs/:org_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.GetOrgSecret (13 handlers)
[GIN-debug] PATCH  /api/orgs/:org_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.PatchOrgSecret (13 handlers)
[GIN-debug] DELETE /api/orgs/:org_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.DeleteOrgSecret (13 handlers)
[GIN-debug] GET    /api/repos/lookup/*repo_full_name --> github.com/woodpecker-ci/woodpecker/server/api.LookupRepo (15 handlers)
[GIN-debug] POST   /api/repos                --> github.com/woodpecker-ci/woodpecker/server/api.PostRepo (13 handlers)
[GIN-debug] GET    /api/repos/:repo_id/permissions --> github.com/woodpecker-ci/woodpecker/server/api.GetRepoPermissions (14 handlers)
[GIN-debug] GET    /api/repos/:repo_id       --> github.com/woodpecker-ci/woodpecker/server/api.GetRepo (15 handlers)
[GIN-debug] GET    /api/repos/:repo_id/branches --> github.com/woodpecker-ci/woodpecker/server/api.GetRepoBranches (15 handlers)
[GIN-debug] GET    /api/repos/:repo_id/pull_requests --> github.com/woodpecker-ci/woodpecker/server/api.GetRepoPullRequests (15 handlers)
[GIN-debug] GET    /api/repos/:repo_id/pipelines --> github.com/woodpecker-ci/woodpecker/server/api.GetPipelines (15 handlers)
[GIN-debug] POST   /api/repos/:repo_id/pipelines --> github.com/woodpecker-ci/woodpecker/server/api.CreatePipeline (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/pipelines/:number --> github.com/woodpecker-ci/woodpecker/server/api.GetPipeline (15 handlers)
[GIN-debug] GET    /api/repos/:repo_id/pipelines/:number/config --> github.com/woodpecker-ci/woodpecker/server/api.GetPipelineConfig (15 handlers)
[GIN-debug] POST   /api/repos/:repo_id/pipelines/:number --> github.com/woodpecker-ci/woodpecker/server/api.PostPipeline (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/pipelines/:number/cancel --> github.com/woodpecker-ci/woodpecker/server/api.CancelPipeline (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/pipelines/:number/approve --> github.com/woodpecker-ci/woodpecker/server/api.PostApproval (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/pipelines/:number/decline --> github.com/woodpecker-ci/woodpecker/server/api.PostDecline (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/logs/:number/:stepId --> github.com/woodpecker-ci/woodpecker/server/api.GetStepLogs (15 handlers)
[GIN-debug] DELETE /api/repos/:repo_id/logs/:number --> github.com/woodpecker-ci/woodpecker/server/api.DeletePipelineLogs (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/secrets --> github.com/woodpecker-ci/woodpecker/server/api.GetSecretList (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/secrets --> github.com/woodpecker-ci/woodpecker/server/api.PostSecret (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.GetSecret (16 handlers)
[GIN-debug] PATCH  /api/repos/:repo_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.PatchSecret (16 handlers)
[GIN-debug] DELETE /api/repos/:repo_id/secrets/:secret --> github.com/woodpecker-ci/woodpecker/server/api.DeleteSecret (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/registry --> github.com/woodpecker-ci/woodpecker/server/api.GetRegistryList (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/registry --> github.com/woodpecker-ci/woodpecker/server/api.PostRegistry (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/registry/:registry --> github.com/woodpecker-ci/woodpecker/server/api.GetRegistry (16 handlers)
[GIN-debug] PATCH  /api/repos/:repo_id/registry/:registry --> github.com/woodpecker-ci/woodpecker/server/api.PatchRegistry (16 handlers)
[GIN-debug] DELETE /api/repos/:repo_id/registry/:registry --> github.com/woodpecker-ci/woodpecker/server/api.DeleteRegistry (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/cron  --> github.com/woodpecker-ci/woodpecker/server/api.GetCronList (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/cron  --> github.com/woodpecker-ci/woodpecker/server/api.PostCron (16 handlers)
[GIN-debug] GET    /api/repos/:repo_id/cron/:cron --> github.com/woodpecker-ci/woodpecker/server/api.GetCron (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/cron/:cron --> github.com/woodpecker-ci/woodpecker/server/api.RunCron (16 handlers)
[GIN-debug] PATCH  /api/repos/:repo_id/cron/:cron --> github.com/woodpecker-ci/woodpecker/server/api.PatchCron (16 handlers)
[GIN-debug] DELETE /api/repos/:repo_id/cron/:cron --> github.com/woodpecker-ci/woodpecker/server/api.DeleteCron (16 handlers)
[GIN-debug] PATCH  /api/repos/:repo_id       --> github.com/woodpecker-ci/woodpecker/server/api.PatchRepo (16 handlers)
[GIN-debug] DELETE /api/repos/:repo_id       --> github.com/woodpecker-ci/woodpecker/server/api.DeleteRepo (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/chown --> github.com/woodpecker-ci/woodpecker/server/api.ChownRepo (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/repair --> github.com/woodpecker-ci/woodpecker/server/api.RepairRepo (16 handlers)
[GIN-debug] POST   /api/repos/:repo_id/move  --> github.com/woodpecker-ci/woodpecker/server/api.MoveRepo (16 handlers)
[GIN-debug] GET    /api/badges/:repo_id_or_owner/status.svg --> github.com/woodpecker-ci/woodpecker/server/api.GetBadge (12 handlers)
[GIN-debug] GET    /api/badges/:repo_id_or_owner/cc.xml --> github.com/woodpecker-ci/woodpecker/server/api.GetCC (12 handlers)
[GIN-debug] GET    /api/badges/:repo_id_or_owner/:repo_name/status.svg --> github.com/woodpecker-ci/woodpecker/server/api.GetBadge (12 handlers)
[GIN-debug] GET    /api/badges/:repo_id_or_owner/:repo_name/cc.xml --> github.com/woodpecker-ci/woodpecker/server/api.GetCC (12 handlers)
[GIN-debug] GET    /api/pipelines            --> github.com/woodpecker-ci/woodpecker/server/api.GetPipelineQueue (13 handlers)
[GIN-debug] GET    /api/queue/info           --> github.com/woodpecker-ci/woodpecker/server/api.GetQueueInfo (13 handlers)
[GIN-debug] POST   /api/queue/pause          --> github.com/woodpecker-ci/woodpecker/server/api.PauseQueue (13 handlers)
[GIN-debug] POST   /api/queue/resume         --> github.com/woodpecker-ci/woodpecker/server/api.ResumeQueue (13 handlers)
[GIN-debug] GET    /api/queue/norunningpipelines --> github.com/woodpecker-ci/woodpecker/server/api.BlockTilQueueHasRunningItem (13 handlers)
[GIN-debug] GET    /api/secrets              --> github.com/woodpecker-ci/woodpecker/server/api.GetGlobalSecretList (13 handlers)
[GIN-debug] POST   /api/secrets              --> github.com/woodpecker-ci/woodpecker/server/api.PostGlobalSecret (13 handlers)
[GIN-debug] GET    /api/secrets/:secret      --> github.com/woodpecker-ci/woodpecker/server/api.GetGlobalSecret (13 handlers)
[GIN-debug] PATCH  /api/secrets/:secret      --> github.com/woodpecker-ci/woodpecker/server/api.PatchGlobalSecret (13 handlers)
[GIN-debug] DELETE /api/secrets/:secret      --> github.com/woodpecker-ci/woodpecker/server/api.DeleteGlobalSecret (13 handlers)
[GIN-debug] GET    /api/log-level            --> github.com/woodpecker-ci/woodpecker/server/api.LogLevel (13 handlers)
[GIN-debug] POST   /api/log-level            --> github.com/woodpecker-ci/woodpecker/server/api.SetLogLevel (13 handlers)
[GIN-debug] GET    /api/agents               --> github.com/woodpecker-ci/woodpecker/server/api.GetAgents (13 handlers)
[GIN-debug] POST   /api/agents               --> github.com/woodpecker-ci/woodpecker/server/api.PostAgent (13 handlers)
[GIN-debug] GET    /api/agents/:agent        --> github.com/woodpecker-ci/woodpecker/server/api.GetAgent (13 handlers)
[GIN-debug] GET    /api/agents/:agent/tasks  --> github.com/woodpecker-ci/woodpecker/server/api.GetAgentTasks (13 handlers)
[GIN-debug] PATCH  /api/agents/:agent        --> github.com/woodpecker-ci/woodpecker/server/api.PatchAgent (13 handlers)
[GIN-debug] DELETE /api/agents/:agent        --> github.com/woodpecker-ci/woodpecker/server/api.DeleteAgent (13 handlers)
[GIN-debug] GET    /api/signature/public-key --> github.com/woodpecker-ci/woodpecker/server/api.GetSignaturePublicKey (13 handlers)
[GIN-debug] POST   /api/hook                 --> github.com/woodpecker-ci/woodpecker/server/api.PostHook (12 handlers)
[GIN-debug] GET    /api/stream/logs/:repo_id/:pipeline/:stepId --> github.com/woodpecker-ci/woodpecker/server/api.LogStreamSSE (15 handlers)
[GIN-debug] GET    /api/stream/events        --> github.com/woodpecker-ci/woodpecker/server/api.EventStreamSSE (12 handlers)
[GIN-debug] GET    /api/debug/pprof/         --> github.com/woodpecker-ci/woodpecker/server/api/debug.IndexHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/heap     --> github.com/woodpecker-ci/woodpecker/server/api/debug.HeapHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/goroutine --> github.com/woodpecker-ci/woodpecker/server/api/debug.GoroutineHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/block    --> github.com/woodpecker-ci/woodpecker/server/api/debug.BlockHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/threadcreate --> github.com/woodpecker-ci/woodpecker/server/api/debug.ThreadCreateHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/cmdline  --> github.com/woodpecker-ci/woodpecker/server/api/debug.CmdlineHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/profile  --> github.com/woodpecker-ci/woodpecker/server/api/debug.ProfileHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/symbol   --> github.com/woodpecker-ci/woodpecker/server/api/debug.SymbolHandler.func1 (13 handlers)
[GIN-debug] POST   /api/debug/pprof/symbol   --> github.com/woodpecker-ci/woodpecker/server/api/debug.SymbolHandler.func1 (13 handlers)
[GIN-debug] GET    /api/debug/pprof/trace    --> github.com/woodpecker-ci/woodpecker/server/api/debug.TraceHandler.func1 (13 handlers)
[GIN-debug] POST   /hook                     --> github.com/woodpecker-ci/woodpecker/server/api.PostHook (12 handlers)
[GIN-debug] GET    /stream/events            --> github.com/woodpecker-ci/woodpecker/server/api.EventStreamSSE (12 handlers)
[GIN-debug] GET    /swagger/*any             --> github.com/swaggo/gin-swagger.CustomWrapHandler.func1 (12 handlers)
6:39AM INF woodpecker/src/github.com/woodpecker-ci/woodpecker/cmd/server/server.go:258 > Starting Woodpecker server with version 'next-503252df38'
6:40AM DBG woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59 > ip=127.0.0.1 latency=0.261191 method=GET path=/healthz status=204 user-agent=Go-http-client/1.1
6:40AM DBG woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59 > ip=127.0.0.1 latency=0.24957 method=GET path=/healthz status=204 user-agent=Go-http-client/1.1
6:41AM DBG woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59 > ip=127.0.0.1 latency=0.320063 method=GET path=/healthz status=204 user-agent=Go-http-client/1.1

Validations

Read the Contributing Guidelines.
Read the docs.
Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

The text was updated successfully, but these errors were encountered:

rubenelshof · 2023-09-05T10:20:33Z

I tried the next version again without success but I did get a additional log that I think could help.
This is from a failed container using docker inspect

[
    {
        "Id": "285d8d63b1482fef13d89f566030f21dd9fa5e66ac8c6da0f60a176d784c8941",
        "Created": "2023-09-05T10:16:39.722413654Z",
        "Path": "/bin/woodpecker-server",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 2,
            "Error": "",
            "StartedAt": "2023-09-05T10:16:41.378953859Z",
            "FinishedAt": "2023-09-05T10:18:13.878531523Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 3,
                "Log": [
                    {
                        "Start": "2023-09-05T12:17:11.379299268+02:00",
                        "End": "2023-09-05T12:17:11.486599771+02:00",
                        "ExitCode": 1,
                        "Output": "{\"level\":\"trace\",\"time\":\"2023-09-05T10:17:11Z\",\"message\":\"try to ping with url 'http://localhost:8000/healthz'\"}\nserver returned non-200 status code\n"
                    },
                    {
                        "Start": "2023-09-05T12:17:41.504028717+02:00",
                        "End": "2023-09-05T12:17:41.619115282+02:00",
                        "ExitCode": 1,
                        "Output": "{\"level\":\"trace\",\"time\":\"2023-09-05T10:17:41Z\",\"message\":\"try to ping with url 'http://localhost:8000/healthz'\"}\nserver returned non-200 status code\n"
                    },
                    {
                        "Start": "2023-09-05T12:18:11.636507324+02:00",
                        "End": "2023-09-05T12:18:11.749084744+02:00",
                        "ExitCode": 1,
                        "Output": "{\"level\":\"trace\",\"time\":\"2023-09-05T10:18:11Z\",\"message\":\"try to ping with url 'http://localhost:8000/healthz'\"}\nserver returned non-200 status code\n"
                    }
                ]
            }
        },

pat-s · 2023-09-08T21:28:53Z

Given that nobody else reported issues and all tests passed, it's likely a config issue.

Did you try starting it without custom DNS mappings? If those are important, can you explain them? Also which other env vars are set?

rubenelshof · 2023-09-08T21:38:00Z

Given that nobody else reported issues and all tests passed, it's likely a config issue.

Did you try starting it without custom DNS mappings? If those are important, can you explain them? Also which other env vars are set?

Yes I have tried starting it without the custom DNS mappings. Same result.

ENV vars set

WOODPECKER_OPEN=true
WOODPECKER_HOST=https://ci.example.com
WOODPECKER_GITEA=true
WOODPECKER_GITEA_URL=https://git.example.com
WOODPECKER_GITEA_CLIENT=
WOODPECKER_GITEA_SECRET=
WOODPECKER_AGENT_SECRET=
WOODPECKER_ADMIN=dionysussg
WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=true
WOODPECKER_DATABASE_DRIVER=mysql
WOODPECKER_DATABASE_DATASOURCE=username:password@tcp(databaseip:port)/woodpecker?parseTime=true
WOODPECKER_LOG_LEVEL=debug
WOODPECKER_DEBUG_PRETTY=true

rubenelshof · 2023-09-08T22:22:31Z

Having looked a bit further I personally think the issue is as follows:

I run Docker in Swarm mode.
woodpecker-server contains a health check to see if the container starts up correctly.
From my understanding in swarm mode a unhealthy container gets killed & a new container is created.
This is continuously repeated until it gets a healthy status back.
The current next version returns a unhealthy status.
Before the issues I have now, if I remember correctly, the container would return a healthy status.
The pinned version I have set now returns the healthy status.

As I mentioned every next version after next-3bee51dc7a fails.
The first PR after this version is #2366
It appears that in that PR the return status for the health endpoint has been changed from 200 to 204.
The problem I think is that the server expects a 200 response.
See server returned non-200 status code in the docker inspect message I posted.

This message is from this file: https://github.com/woodpecker-ci/woodpecker/blob/main/cmd/server/health.go
On line 60 you see the message.

Now I don't know much about golang code but I think it checks if the status is 200 and if not it will give an error and return this message.
By giving the error and returning this message it fails the health check and in return kills the container.

In addition to the above I have tried running Woodpecker on a different server with the following command:

docker run -it --rm -p 8092:8000 -e WOODPECKER_HOST=https://ci.test.com -e WOODPECKER_GITEA=true -e WOODPECKER_LOG_LEVEL=debug -v woodpecker-server-data:/var/lib/woodpecker/ woodpeckerci/woodpecker-server:next

Now the container doesn't get killed but still gets the unhealthy status & returns the same message I posted before from docker inspect

anbraten · 2023-09-09T06:23:56Z

We could check to use a status check like: statusOK := response.StatusCode >= 200 && response.StatusCode < 300

pat-s · 2023-09-09T07:25:17Z

@rubenelshof Thanks for the deep dive! This makes a lot of sense.

I wonder:

Don't we have any test for the health check that would have caught this? Are so few users running in native docker? (assuming that every docker users faces this issue).
Why isn't it an issue in any deployment kind?

pat-s · 2023-09-09T12:09:03Z

@qwerty287 FYI since you added #2366

qwerty287 · 2023-09-09T12:11:32Z

Hmm yes, indeed, I changed it to 204, didn't know I would have to change this as well. Fixing should be easy though by applying @anbraten's solution in #2370 (comment) to https://github.com/woodpecker-ci/woodpecker/blob/main/cmd/server/health.go#L59

Fix #2370

rubenelshof added the bug Something isn't working label Sep 5, 2023

qwerty287 mentioned this issue Sep 9, 2023

Fix health check #2412

Merged

qwerty287 added the regression fix a bug that was not released yet label Sep 9, 2023

qwerty287 closed this as completed in #2412 Sep 9, 2023

qwerty287 added a commit that referenced this issue Sep 9, 2023

Fix health check (#2412)

a06a626

Fix #2370

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Latest next version of Woodpecker server fails to start #2370

Latest next version of Woodpecker server fails to start #2370

rubenelshof commented Sep 5, 2023 •

edited

Loading

rubenelshof commented Sep 5, 2023 •

edited

Loading

pat-s commented Sep 8, 2023

rubenelshof commented Sep 8, 2023

rubenelshof commented Sep 8, 2023

anbraten commented Sep 9, 2023

pat-s commented Sep 9, 2023

pat-s commented Sep 9, 2023

qwerty287 commented Sep 9, 2023

Latest next version of Woodpecker server fails to start #2370

Latest next version of Woodpecker server fails to start #2370

Comments

rubenelshof commented Sep 5, 2023 • edited Loading

Component

Describe the bug

System Info

Additional context

Validations

rubenelshof commented Sep 5, 2023 • edited Loading

pat-s commented Sep 8, 2023

rubenelshof commented Sep 8, 2023

rubenelshof commented Sep 8, 2023

anbraten commented Sep 9, 2023

pat-s commented Sep 9, 2023

pat-s commented Sep 9, 2023

qwerty287 commented Sep 9, 2023

rubenelshof commented Sep 5, 2023 •

edited

Loading

rubenelshof commented Sep 5, 2023 •

edited

Loading