Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The deprecated runner version fails after job assignment not after start to listen for jobs #3366

Closed
enescakir opened this issue Jun 28, 2024 · 12 comments
Closed

The deprecated runner version fails after job assignment not after start to listen for jobs #3366

enescakir opened this issue Jun 28, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@enescakir
Copy link
Contributor

Describe the bug
In the previous versions, the runner used to fail prior to the job assignment and kept the job as queued if the runner version is deprecated. However, in the latest version, the failure occurs after the job has been picked, resulting in an immediate job failure. Waiting in queue for a runner with the new version was a better option.

√ Connected to GitHub
Current runner version: '2.316.1'
2024-06-27 15:51:23Z: Listening for Jobs
2024-06-27 15:51:26Z: Running job: test
2024-06-27 15:51:32Z: Job test completed with result: Canceled
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...

To Reproduce
Assign a job to runner with version 2.316.1

Expected behavior
Exit the runner due to deprecation before picking a job

Example log from older versions

√ Connected to GitHub
Current runner version: '2.310.2'
2024-02-03 15:06:22Z: Listening for Jobs
An error occured: Runner version v2.310.2 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...
@enescakir enescakir added the bug Something isn't working label Jun 28, 2024
@MuhammadElsaeed
Copy link

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

@enescakir
Copy link
Contributor Author

I am facing the same issue, how come the image is deprecated? it was released just last month. Are we supposed to update them every now and then?

There are lot of discussions on this topic. I used to version deprecation within a month

#3332
#3235

But the runner was failing before picking a job, so the job was waiting in the queued state for 24 hours. We were able to provision a new runner with the new version. But with the latest version, the runner started to fail after picking a job, so it causes job failure.

@luketomlinson
Copy link
Contributor

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

@enescakir
Copy link
Contributor Author

enescakir commented Jun 28, 2024 •
edited
Loading

Hi @enescakir thanks for raising this. Do you have a link to a failed run you could share?

@luketomlinson It's the run which I previously shared logs.

https://github.com/ubicloud/github-e2e-test-workflows/actions/runs/9699711981/job/26769354466

@enescakir
Copy link
Contributor Author

enescakir commented Jun 28, 2024 •
edited
Loading

@luketomlinson The interesting part is that not all jobs are failing. We use different GitHub Apps for local development, and they manage to run jobs with this older version. However, the production GitHub app can't run with this version. I don't know its details but the GitHub backend might have a certain check for the assigned job, and it may have determined that this job can't run with this runner version.

@luketomlinson
Copy link
Contributor

Thanks @enescakir, that's all helpful info, I'm looking into it.

@dt-napier
Copy link

Got same problem today. We have 3 runner scale sets in one namespace, with same docker image, but they target different repositories. And only one fails with this error:

[RUNNER 2024-06-28 15:03:27Z INFO MessageListener] VssConnection created
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

√ Connected to GitHub
[RUNNER 2024-06-28 15:03:27Z INFO Terminal] WRITE LINE: 

[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:27Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] Session created.
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: Current runner version: '2.316.1'
Current runner version: '2.316.1'
[RUNNER 2024-06-28 15:03:28Z INFO Terminal] WRITE LINE: 2024-06-28 15:03:28Z: Listening for Jobs
2024-06-28 15:03:28Z: Listening for Jobs
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-06-28 15:03:28Z INFO MessageListener] BrokerMigration message received. Polling Broker for messages...
[RUNNER 2024-06-28 15:03:28Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-06-28 15:03:28Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-06-28 15:03:28Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=138c0456-6361-41aa-9a21-28bdecdc5eaa&status=Online&runnerVersion=2.316.1&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-06-28 15:03:28Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-06-28 15:03:28Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-06-28 15:03:29Z ERR  Terminal] WRITE ERROR: An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2024-06-28 15:03:29Z ERR  Listener] --- End of stack trace from previous location ---
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxRetryAttemptsCount, Func`2 shouldRetry)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[RUNNER 2024-06-28 15:03:29Z ERR  Listener]    at GitHub.Runner.Listener.Program.MainAsync(IHostContext context, String[] args)
Runner listener exit with terminated error, stop the service, no retry needed.
Exiting runner...```

@ChristopherHX ChristopherHX mentioned this issue Jun 29, 2024
Closed
@luketomlinson
Copy link
Contributor

This issue should be fixed now. Old runners should no longer be able to be assigned jobs. @enescakir @MuhammadElsaeed @dt-napier Can you confirm you're no longer seeing this?

@enescakir
Copy link
Contributor Author

Thanks to your prompt solution, @luketomlinson, it's back to functioning as it used to. Was this problem related to the backend services?

√ Connected to GitHub
Current runner version: '2.316.1'
2024-07-02 07:01:03Z: Listening for Jobs
An error occured: Error: Forbidden Runner version v2.316.1 is deprecated and cannot receive messages.
Runner listener exit with terminated error, stop the service, no retry needed.Exiting runner...

@luketomlinson
Copy link
Contributor

Hey @enescakir, yep it was a backend issue. I'm going to close this out for now, but please open an issue if you see it again!

@Jungzhang
Copy link

Hi, I am facing the same issue @luketomlinson

@Jungzhang
Copy link

Upgrade to 2.317.0 is ok.

@mattnakama-skytap mattnakama-skytap mentioned this issue Jul 9, 2024
Closed
@jonapich jonapich mentioned this issue Jul 10, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@enescakir @Jungzhang @MuhammadElsaeed @luketomlinson @dt-napier