Ingress questions/issues

[amolenk]

I’ve got some questions about how ingress actually works. I’m testing with a simple echo container (https://hub.docker.com/r/mendhak/http-https-echo) to view the HTTP request headers sent to the container application. I’m deploying the app with Bicep:

resource containerApp 'Microsoft.Web/containerApps@2021-03-01' = {
  name: 'echo'
  location: location
  properties: {
    kubeEnvironmentId: containerAppsEnvironmentId
    template: {
      containers: [
        {
          name: 'echo'
          image: 'mendhak/http-https-echo:23'
        }
      ]
      scale: {
        minReplicas: 1
      }
    }
    configuration: {
      activeResivionsMode: 'single'
      ingress: {
        external: true
        targetPort: 8080
      }
    }
  }
}

When I access the HTTPS application URL from the Azure Portal, I get the following response:

{
  "path": "/",
  "headers": {
    "host": "echo.gentlewater-47b7ce10.eastus.azurecontainerapps.io",
    "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"97\", \"Chromium\";v=\"97\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"macOS\"",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.76",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "sec-fetch-site": "cross-site",
    "sec-fetch-mode": "navigate",
    "sec-fetch-user": "?1",
    "sec-fetch-dest": "document",
    "referer": "https://sandbox-14-11.reactblade.portal.azure.net/",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "en-US,en;q=0.9,nl;q=0.8",
    "if-none-match": "W/\"6fc-O34NjIdkI94LhyKC1HLb0W4LNNU\"",
    "x-forwarded-for": "<redacted>",
    "x-forwarded-proto": "http",
    "x-envoy-external-address": "<redacted>",
    "x-request-id": "c3f69702-56db-40b1-9354-b4961728ead9",
    "x-envoy-expected-rq-timeout-ms": "1800000",
    "x-k8se-app-name": "echo--5b4q55x",
    "x-k8se-app-namespace": "k8se-apps",
    "x-k8se-protocol": "http1",
    "x-ms-containerapp-name": "echo",
    "x-ms-containerapp-revision-name": "echo--5b4q55x",
    "x-arr-ssl": "true"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "echo.gentlewater-47b7ce10.eastus.azurecontainerapps.io",
  "ip": "<redacted>",
  "ips": [
    "<redacted>"
  ],
  "protocol": "http",
  "query": {},
  "subdomains": [
    "eastus",
    "gentlewater-47b7ce10",
    "echo"
  ],
  "xhr": false,
  "os": {
    "hostname": "echo--5b4q55x-5c548475b6-snwpq"
  },
  "connection": {}
}

Why is the value of the x-forwarded-proto header http? I would’ve expected this to be https because the client calls the ingress endpoint over HTTPS.

Also, is the allowInsecure: true ingress option supported yet? When I use this option I still get redirected when making an HTTP call. Retested this and allowInsecure: true seems to work fine.

Finally, I also tried calling the HTTPS endpoint of the echo container (which is 8443) by changing my config:

ingress: {
  external: true
  targetPort: 8443
}

This gives me the following error when I try to call the app: upstream connect error or disconnect/reset before headers. reset reason: connection termination

[michielrutting]

I'm running into probably the same issue causing the redirect uri when logging into my web app using Azure AD and Microsoft Identity to be http instead of https. This happens even when using the forwarded headers middleware in my asp.net core 6.0 app.

[ecosgrave]

I had the same problem, after a lot of trial and error the workaround was to add a middleware to "fool" the OAuth middleware into constructing a https scheme. There may be other unintended consequences you need to consider, but this got me past the redirect_uri issue.

The idea is suggested here
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-6.0

// In Program.cs/Startup.cs when building your app
//
// OAuth middleware "notices" that we are running over HTTP not HTTPS when in a container which
// causes redirect_uri scheme to be "http" instead of "https".  This middleware forces incoming requests 
// to use the https scheme, so OAuth middleware can craft the correct redirect_uri.
if (app.Environment.IsRunningInContainer()) // doesn't happen outside of container environment.
{
  app.Use((context, next) =>
  {
    context.Request.Scheme = "https";
    return next(context);
  });
}

I spent time trying some of the following header forwarding ideas but they did not solve the redirect_uri problem, included here for reference of what wasn't helpful.
AzureAD/microsoft-identity-web#115
https://github.com/AzureAD/microsoft-identity-web/wiki/Deploying-Web-apps-to-App-services-as-Linux-containers#issues-with-load-balancing-across-multiple-regions-using-front-door

[kendallroden]

you can disable http to https redirection by passing in the --allow-insecure parameter. Would this solve your issue? @amolenk

[michielrutting]

Thanks for the reply @kendallroden. I'm not @amolenk, but I think the first part of the issue is that when accessing the ingress over HTTPS one would expect the value of the x-forwarded-proto header received by the application to be https, when currently it is http.

This could cause problems with middleware that now assumes the application is being accessed insecurely over http. It is possible to fool the application by inserting an extra middleware like @ecosgrave demonstrates, but in the linked documentation this is suggested for scenarios where forwarded headers are not present at all.

[amolenk]

I’ve solved my Identity Server issue by letting Identity Server generate HTTPS urls when it finds an x-arr-ssl: true HTTP header in the request. The team confirmed that the x-forwarded-proto header is being accidentally overwritten and should indeed contains https in my case. They'll make a fix for it.

For those interested, I'm adding Container Apps support to the eShopOnDapr sample app. Bicep files can be found here: https://github.com/dotnet-architecture/eShopOnDapr/tree/amolenk/containerapps/deploy/containerapps

[michielrutting]

Thanks for the confirmation @amolenk. I had briefly looked through the eShopOnDapr code before on GitHub and I want to dive deeper into it soon. What I've seen already showed me how some new and interesting ways of doing things, so thanks for that!

[kendallroden]

Hi all we are working on a fix and will update the thread when it is completed and available

[pricetula]

make sure the port specified is correct as well

[Hoang-Minh]

I run into same issue. Wonder if there is a fix for this issue yet ?

[zhenqxuMSFT]

This issue should be fixed. Correct proto can be returned from HTTP/HTTPS endpoint.
Accessing https://xxxxxx

 "headers": {
    ...
    "x-forwarded-proto": "https"
  },

Accessing http://xxxxxx

 "headers": {
    ...
    "x-forwarded-proto": "http"
  },

[zhenqxuMSFT]

@Hoang-Minh are you still having the issues? If so, could you provide repro steps?

[Hoang-Minh]

@zhenqxuMSFT I no longer have this issue after modifying my yml file.

[ahmelsayed]

Closing issue as fixed.

[kjeske]

@Hoang-Minh how you modified the yml file?

[Hoang-Minh]

@kjeske This is how I resolved the issue:

Set the ASPNETCORE_FORWARDEDHEADERS_ENABLED = true AND

Add these two lines suggested from @Venkatesan to my ingress yml

nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-regex: "true"

Everything works perfectly after that.
Link ref: SOF

[kjeske]

Thank you. I spent some time figuring out why ASPNETCORE_FORWARDEDHEADERS_ENABLED was the way and not configuring it in the Startup.cs when you have ASP.NET MVC Core app. But I found the answer. Here is the code that makes it working without having ASPNETCORE_FORWARDEDHEADERS_ENABLED set to true:

services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    options.KnownNetworks.Clear();
    options.KnownProxies.Clear();
});

The thing that makes a difference is this:

options.KnownNetworks.Clear();
options.KnownProxies.Clear();

Here explanation:
https://soapfault.com/2020/02/24/asp-net-core-reverse-proxy-and-x-forwarded-headers/

[Hoang-Minh]

@kjeske I tried your settings with my .net 6 application and it did not work. Maybe I was missing something.

[kjeske]

@Hoang-Minh that was only the configuration. It's important to also use it your application builder:

app.UseForwardedHeaders();