Traefik with Service Fabric -- failed to connect to Service Fabric server

Question

I have deployed Traefik on my Azure Service Fabric cluster with the following configuration:

# Enable Service Fabric configuration backend
[servicefabric]

# Service Fabric Management Endpoint
clustermanagementurl = "https://localhost:19080"

# Service Fabric Management Endpoint API Version
apiversion = "3.0"

insecureSkipVerify = true

However, when opening the Traefik dashboard, I'm getting an empty screen, as it fails to map all my Fabric applications.

Looking at the Traefik logs on one of my VMs, I'm seeing this error repeatedly:

level=error msg="failed to connect to Service Fabric server Get https://localhost:19080/Applications/?api-version=3.0: x509: certificate is valid for .eastus.cloudapp.azure.com, not localhost on https://localhost:19080/Applications/?api-version=3.0"

My Azure Service Fabric cluster has a SSL certificate signed by a trusted CA:

How can I solve this issue?

Edit 1:

If it helps, this is the configuration Traefik loads (according to the logs):

{
    "LifeCycle": {
        "RequestAcceptGraceTimeout": 0,
        "GraceTimeOut": 0
    },
    "GraceTimeOut": 0,
    "Debug": true,
    "CheckNewVersion": true,
    "AccessLogsFile": "",
    "AccessLog": null,
    "TraefikLogsFile": "",
    "TraefikLog": null,
    "LogLevel": "DEBUG",
    "EntryPoints": {
        "http": {
            "Network": "",
            "Address": ":80",
            "TLS": null,
            "Redirect": null,
            "Auth": null,
            "WhitelistSourceRange": null,
            "Compress": false,
            "ProxyProtocol": null,
            "ForwardedHeaders": {
                "Insecure": true,
                "TrustedIPs": null
            }
        }
    },
    "Cluster": null,
    "Constraints": [],
    "ACME": null,
    "DefaultEntryPoints": [
        "http"
    ],
    "ProvidersThrottleDuration": 2000000000,
    "MaxIdleConnsPerHost": 200,
    "IdleTimeout": 0,
    "InsecureSkipVerify": true,
    "RootCAs": null,
    "Retry": null,
    "HealthCheck": {
        "Interval": 30000000000
    },
    "RespondingTimeouts": null,
    "ForwardingTimeouts": null,
    "Docker": null,
    "File": null,
    "Web": {
        "Address": ":9000",
        "CertFile": "",
        "KeyFile": "",
        "ReadOnly": false,
        "Statistics": null,
        "Metrics": null,
        "Path": "/",
        "Auth": null,
        "Debug": false,
        "CurrentConfigurations": null,
        "Stats": null,
        "StatsRecorder": null
    },
    "Marathon": null,
    "Consul": null,
    "ConsulCatalog": null,
    "Etcd": null,
    "Zookeeper": null,
    "Boltdb": null,
    "Kubernetes": null,
    "Mesos": null,
    "Eureka": null,
    "ECS": null,
    "Rancher": null,
    "DynamoDB": null,
    "ServiceFabric": {
        "Watch": false,
        "Filename": "",
        "Constraints": null,
        "Trace": false,
        "DebugLogGeneratedTemplate": false,
        "ClusterManagementURL": "https://localhost:19080",
        "APIVersion": "3.0",
        "UseCertificateAuth": false,
        "ClientCertFilePath": "",
        "ClientCertKeyFilePath": "",
        "InsecureSkipVerify": true
    }
}

Edit 2:

One suggested to use the remote address of my cluster instead of localhost, doing so results in a different error:

Provider connection error: failed to connect to Service Fabric server Get https://.eastus.cloudapp.azure.com:19080/Applications/?api-version=3.0: stream error: stream ID 1; HTTP_1_1_REQUIRED on https://.eastus.cloudapp.azure.com:19080/Applications/?api-version=3.0; retrying in 656.765021ms

user_3454745 · Answer

Thanks to Diego's comment (under my question) I succeeded solving this issue with the following additions.

What was the problem?

My SF cluster is secured, requiring a client certificate to login -- which was not specified in the Traefik TOML file. (wish the error logged was more informative)
Looking at the Traefik logs, specifically on the SF part (look for the trace starting with Starting provider *servicefabric.Provider :
```
"Watch": false,
"Filename": "",
"Constraints": null,
"Trace": false,
"DebugLogGeneratedTemplate": false,
"ClusterManagementURL": "https://localhost:19080",
"APIVersion": "3.0",
"UseCertificateAuth": false,      <-------- Important
"ClientCertFilePath": "",         <-------- Important
"ClientCertKeyFilePath": "",      <-------- Important
"InsecureSkipVerify": false
```
- UseCertificateAuth -- indicates whether to use client certificate when Traefik queries the cluster's management endpoint.
- ClientCertFilePath -- the path of the file containing the public key of the client certificate.
- ClientCertKeyFilePath -- the path of the file containing the private key of the client certificate.

(both paths should be relative to the traefik.exe)

InsecureSkipVerify

The Traefik's SF config (above) includes a setting called InsecureSkipVerify

InsecureSkipVerify -- If set to false, then Traefik will reject the connection to the management endpoint unless the SSL certificate used is signed by a trusted CA.
This could be an issue if the certificate is signed for the remote address, while Traefik uses https://localhost as the cluster's endpoint -- as then Traefik would print an error similar to this:

failed to connect to Service Fabric server Get https://localhost:19080/Applications/?api-version=3.0: x509: certificate is valid for .eastus.cloudapp.azure.com, not localhost

To overcome this one, you can either

Set InsecureSkipVerify = true and redeploy
Set the management endpoint to the remote address: clustermanagementurl = "https://.eastus.cloudapp.azure.com:19080"

Thanks again Diego for giving me the hint that lead me to understand and share the above explanation.

user_10586744 · Answer

I know this is an old post but we just ran into this exact situation and this is the ONLY place i've seen the client settings mentioned. This is the provider section that finally seems to have worked for us:

################################################################
# Service Fabric provider
################################################################

# Enable Service Fabric configuration backend
[servicefabric]

# Service Fabric Management Endpoint
clustermanagementurl = "https://localhost:19080"
# Note: use "https://localhost:19080" if you're using a secure cluster

# Service Fabric Management Endpoint API Version
apiversion = "3.0"

# Enable TLS connection.
#
# Optional
#
[serviceFabric.tls]
  cert               = "certs/servicefabric.crt"
  key                = "certs/servicefabric.key"
  insecureskipverify = true

UseCertificateAuth    =  true
ClientCertFilePath    = "certs/traefik.crt"
ClientCertKeyFilePath = "certs/traefik.key"
InsecureSkipVerify    =  true

user_484222 · Answer

To authenticate to ServiceFabric APIs, you have to use a certificate, in your configuration you miss this detail.

In the Traefik settings you should have something like this:

# [serviceFabric.tls]
cert = "certs/servicefabric.crt"
key = "certs/servicefabric.key"
insecureskipverify = true

The following post describe it step by step

https://blog.techfabric.io/using-traefik-reverse-proxy-for-securing-microservices-on-azure-service-fabric/

Traefik with Service Fabric -- failed to connect to Service Fabric server

Edit 1:

Edit 2:

3 Answers

What was the problem?

InsecureSkipVerify