Agent is not responding in Windows tasks again

[ericoporto]

Expected Behavior

Windows tasks should build normally

Real Behavior

They don't start, I get an Agent is Not Responding issue

The Agent is not responding! message didn't appear before 08/09/2023, so it's probably something else that changed in the meantime.

Related Info

• Task issue
  • OS: Windows
  • Task Name build_windows
    https://cirrus-ci.com/task/5755161579094016

I know this looks similar to #1213 , but we tried to force a rerun of the docker build and it didn't fix.

Tried to force a re-cache by rebuilding the windows docker image changing the comments (which change the cache hash) but it didn't work either: adventuregamestudio/ags#2128

The error is happening in our recent builds since 08/09/2023
https://cirrus-ci.com/github/adventuregamestudio/ags

If it could have some better logging in case this is something wrong on our side, because right now we can't figure it out anyway to make it work.

I looked into Google Cloud Registry, and all our cached images for windows are there since 2021 which made things even more mysterious for me. I pulled the docker image locally in my Windows 10 and as far as I can tell it runs and it's normal?

What I did noticed is Container Registry is Deprecated!!! So we have until May 15, 2024 to migrate to Artifact Registry or something else!

Apparently if I try many times I can also get the Instance got rescheduled! message.

---

I see the same issue happen in other windows builds in CirrusCi.

• pypa/cibuildwheel: https://cirrus-ci.com/task/5514257299668992
• pyscaffold: https://cirrus-ci.com/task/4750682981400576
• zeek: https://cirrus-ci.com/task/6107088280092672
• pgbouncer: https://cirrus-ci.com/task/6075965571334144
• pyscaffoldext-markdown: https://cirrus-ci.com/task/6706836842217472

[ericoporto]

I found above that we are not the only repository where this issue is appearing - unfortunately a lot of repositories are using cirrus-ci only for the FreeBSD or arm64 offering, so I didn’t knew many repositories that included windows builds.

[fkorotkov]

The underlying issue seems to be that Cirrus Agent inside a container is not able to resolve grpc.cirrus-ci.com. Looks like microsoft/Windows-Containers#217 but there is not workaround that I can find yet. Continue looking...

[fkorotkov]

Fixed it by setting --dns flag explicitly when a container is getting run e.g. docker run --dns 8.8.8.8.

[ericoporto]

indeed it's fixed, thank you!

[ericoporto]

Oops, we have a new issue that says

Failed to get scheduled in timely manner!

Not sure what that means.

From here: https://cirrus-ci.com/task/6424294868123648

(it’s a windows task too…)

edit: triggering a rebuild made that error go away.

[ericoporto]

Getting Agent Not Responding issue again.

https://cirrus-ci.com/build/5763876357144576
https://cirrus-ci.com/task/5708568318443520
https://cirrus-ci.com/build/5049630409359360
https://cirrus-ci.com/task/5011195720105984

[fkorotkov]

This usually related to one of the layers of the prebuilt windows container expiring. Please try to re-run the Prebuilt task next time. Cirrus logic is only checking availability of the first few layers of an image in order to decide if it's there or not. Since Windows containers are so huge and have so many layers this logic fails more frequently.

[ericoporto]

It would be nice to get either a better error message or some documentation with things to try it out.

Right now it was as far as I can tell a 12hr outage that started to work again eventually after trying repeatedly to rerun things.