Docker Compose healthcheck between two containerized ASP.NET Core projs causes hang

[brandonh-msft]

I have two containers defined in my docker-compose:

docker-compose.yml

services:
  cis.copilot.orchestrator:
    image: ${DOCKER_REGISTRY-}ciscopilotorchestrator
    build:
      context: .
      dockerfile: ./CIS.CoPilot.Orchestrator/Dockerfile
    healthcheck:
      test: "curl -fLk --no-progress-meter http://localhost:8080/healthcheck"
      interval: 0s
      timeout: 3s
      retries: 1
      start_period: 10s
      start_interval: 5s

  cis.copilot.netagent:
    image: ${DOCKER_REGISTRY-}ciscopilotnetagent
    build:
      context: .
      dockerfile: CIS.CoPilot.NetAgent/Dockerfile
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      cis.copilot.orchestrator:
        condition: service_healthy

with an override file of

docker-compose.override.yml

services:
  cis.copilot.orchestrator:
    build:
      args:
      - BUILD_CONFIGURATION=Debug
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_HTTP_PORTS=8080
      - ASPNETCORE_HTTPS_PORTS=8081
      - Logging__LogLevel__Default=Trace
    entrypoint: ["dotnet", "CIS.CoPilot.Orchestrator.dll"]
    volumes:
      - ${APPDATA}/Microsoft/UserSecrets:/home/app/.microsoft/usersecrets:ro
      - ${APPDATA}/ASP.NET/Https:/home/app/.aspnet/https:ro
    expose:
      - 8080
      - 8081
    ports:
      - 8080:8080
      - 8081:8081
  cis.copilot.netagent:
    build:
      args:
      - BUILD_CONFIGURATION=Debug
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_HTTP_PORTS=8080
      - ASPNETCORE_HTTPS_PORTS=8081
      - Logging__LogLevel__Default=Trace
    volumes:
      - ${APPDATA}/Microsoft/UserSecrets:/home/app/.microsoft/usersecrets:ro
      - ${APPDATA}/ASP.NET/Https:/home/app/.aspnet/https:ro

I found that, when VS is used to build & deploy the containers, the entrypoint of the container is changed to use dotnet roll forward:

		"Entrypoint": [
			"dotnet",
			"--roll-forward",
			"Major",
			"/VSTools/DistrolessHelper/DistrolessHelper.dll",
			"--wait"
		],

Instead of respecting the endpoint defined in the Dockerfile for the orchestrator:

orchestrator DOCKERFILE

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS publish
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY . .
WORKDIR "/src/CIS.CoPilot.Orchestrator"
RUN dotnet dev-certs https --trust -q
RUN dotnet publish "./CIS.CoPilot.Orchestrator.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM mcr.microsoft.com/dotnet/aspnet:8.0

RUN apt-get update && \
	apt-get install -y curl

WORKDIR /app
COPY --from=publish /app/publish .
USER app

ENTRYPOINT ["dotnet", "CIS.CoPilot.Orchestrator.dll"]

This is the only diff I can tell between native docker compose up and using VS though, admittedly, I haven't done a full diff between both INSPECT areas of the resulting container groups.

I have tried augmenting my dcproj with

<ContainerDevelopmentMode>Regular</ContainerDevelopmentMode>
<DockerFastModeEnabled>false</DockerFastModeEnabled>

as well as forcing the entrypoint value in my docker-compose.override.yml but so far have had no luck.

When I docker compose up from terminal, everything works exactly as it should.

How do I fix this?

[NCarlsonMSFT]

@brandonh-msft I'm not able to reproduce a hang, but debugging is broken for me when using depends_on with the health check.

Work-around:
Use
Right-Click on the project->Add->VS Compose Override Files

This adds a docker-compose.vs.debug.yml and docker-compose.vs.release.yml
In those files you can add:

services:
  cis.copilot.netagent:
    depends_on:
      cis.copilot.orchestrator:
        condition: service_started

To change the depends_on condition, but only when running through VS

FYI The Entrypoint change isn't just to support Fast mode, it's also used in regular mode so that the service can be started under the debugger, but it does mean that depends_on doesn't work reliably.

[brandonh-msft]

That's interesting. What do your containers show when they start up? Mine never run the ASP.NET application - hence my mention of "hang" whereas when I remove the depends_on the application starts w/o issue.

I appreciate the "workaround" of having a couple of VS-specific files, however, without that health check in place I have to write code to handle the exceptions that occur when container 2 attempts to hit container 1. Code that I have to #ifdef around with necessary comments all because of this issue. I shouldn't have to do that... do you agree?

[NCarlsonMSFT]

We recognize it isn't ideal, and have an item on our back log to improve it. But I don't have a timeline you can expect improvement by.

[zharchimage]

That's interesting. What do your containers show when they start up? Mine never run the ASP.NET application - hence my mention of "hang" whereas when I remove the depends_on the application starts w/o issue.

I was developing an app and have the same problem as you do. My services never get started as long as I have the healthcheck and depends_on. Added an issue with a detailed description in here #446

[NCarlsonMSFT]

@brandonh-msft
Visual Studio 2022 v17.13 Preview 1 has added support for depends_on as a new opt-in feature. To turn it on set the msbuild property DependencyAwareStart to true in the .dcproj:

<PropertyGroup>
  <DependencyAwareStart>true</DependencyAwareStart>
</PropertyGroup>

If you can, please try out the preview and let me know if you're unblocked now!