Getting 410 gone response.

[ManasaSathyanarayan]

I'm Manasa from configit. We are working on your Traefik for Service fabric, for one of our services.
Details:

1. Traefik version 1.6.0
2. Our service is stateless and all services are in same cluster.
3. We are deploying the services into Azure Cluster with our own deployment Powershell Scripts- not using VS Scripts Deploy-FabricApplication.ps1
4. We are using secure cluster.
5. The azure loadbalancer uses ports 80:8080 and we have not changed these defaults in the toml and servicemanifest.xml files.

What we see:

Traefik and our service is deployed without error. We can see backend and frontend listed in the dashboard. But calling cluster endpoint for our routed rule gets a 410 response. If we call Cluster endpoint with URI that doesn't match a rule it gives 404 error. So our conclusion is that traefik is matching correctly but not routing to the backend.

Below is the toml file after some manipulation in powershell:

debug=true
logLevel="INFO"
defaultEntryPoints=["http", "https"]
[entryPoints]
[entryPoints.http]
address=":80"
[entryPoints.traefik]
address=":8080"
[api]
entryPoint="traefik"
dashboard=true
debug=true
[servicefabric]
clustermanagementurl="https://localhost:19080"
apiversion="3.0"
[serviceFabric.tls]
key="Certs/servicefabric.key"
cert="Certs/servicefabric.crt"
insecureskipverify=true

Do you have any idea why we are getting this response ?

[jjcollinge]

Does your backend service use the ASPNET Service Fabric middleware i.e. UseServiceFabricIntegration?
if it does what do you pass in for the ServiceFabricIntegrationOptions parameter? What CommunicationListener are you using? The 410 behaviour of the middleware is explained here: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-communication-aspnetcore#using-unique-service-urls

Can you check that the endpoint displayed in the Service Fabric explorer for a service and that shown in the Traefik dashboard are the same.

[ManasaSathyanarayan]

Yes we are using ASPNET service fabric middleware with this option:

....UseServiceFabricIntegration( listener, ServiceFabricIntegrationOptions.UseUniqueServiceUrl )

We needed this option, or APIM also gave a similar response (currently we are using APIM in front of our service too.)

We are using the KestrelCommunicationListener object.

The endpoints in the explorer and in the dashboard are the same.

[lawrencegripper]

Hi,

I'll have some more time later this week to help out in more detail. In the meantime would it be ok to quickly try out something that might resolve this? I can't guarantee it will work.

Do you see the same behavior with APIM/Traefik when using .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.UseReverseProxyIntegration)?

I think this may help as it will assign a specific port per instance of the service rather than services sharing ports with appended unique urls.

[deanmccrae]

Hi guys,

Thanks for the help and getting back so quickly.

It appears from our tests so far that the use of UseReverseProxyIntegration is working for us with Traefik. I don't remember if we tried this option with APIM - I think we immediately used UseUniqueServiceUrl and found it worked fine with APIM and stopped at that. We will try UseReverseProxyIntegration with APIM also since we initially want both APIM and Traefik to work against the same service binary until we make the complete change-over to Traefik. Will write back when we've tried this.

[lawrencegripper]

Great, glad that has helped and fingers crossed it also works with APIM. Let us know how it goes?

On a side note I think this is probably a good bit to addd into the docs - would you be interested in adding a note about the middleware options to the Readme via a PR?

[deanmccrae]

This option appears also to work with APIM.

I would he happy to contribute to the docs: however I have no idea what UseReverseProxyIntegration actually does. It seems all of these enum flags are not documented anywhere - although there are some scatterings of documentation regarding the UseUniqueServiceUr option, buried in other articles. I don't know why these extension methods are not documented in the normal Microsoft docs.

For example, what is the difference between these two? And since they are flags and can be combined, what is the effect of combining them? I am in the dark about it.

On the other hand - they are massively important since they either make or break the use of APIM and Traefik.

[lawrencegripper]

Yes I had a similar experience and raised the feed back in this issue here. From my experimenting, while waiting for an answer, it appears to assign a unique port per service instance rather than a shared port with a unique url per service instance.

Lets give it a nudge and see if we can get clarification.

[JustinKaffenberger]

It appears I'm having the same problem. After looking at the Traefik logs, it appears the solution is to have the URL.Path value be respected when determining the final URI. So for example, the below log statement shows the correct URL.Host and URL.Path of my service in Service Fabric, so that all lines up. The RequestURI, which is correct ( /api/unittest/) gets combined with URL.Host, but URL.Path seems to be ignored by Traefik when building the final request. I think the solution is for the final request to be URL.Host + URL.Path + RequestURI. Full log statement is below:

{
        "Method": "GET",
        "URL": {
            "Scheme": "http",
            "Opaque": "",
            "User": null,
            "Host": "localhost:55023",
            "Path": "/0a94309c-c594-4770-97ea-e9978ac747e8/131890113202518360",
            "RawPath": "",
            "ForceQuery": false,
            "RawQuery": "",
            "Fragment": ""
        },
        "Proto": "HTTP/1.1",
        "ProtoMajor": 1,
        "ProtoMinor": 1,
        "Header": {
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Connection": [
                "keep-alive"
            ],
            "Cookie": [
                "ai_user=s9ou/|2018-11-07T13:46:02.480Z"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
            ],
            "X-Forwarded-Prefix": [
                "/integrationtest"
            ]
        },
        "ContentLength": 0,
        "TransferEncoding": null,
        "Host": "localhost",
        "Form": null,
        "PostForm": null,
        "MultipartForm": null,
        "Trailer": null,
        "RemoteAddr": "[::1]:56304",
        "RequestURI": "/api/unittest/",
        "TLS": null
    }