how to ignore server certificate check when connect to a self signed existing ES cluster

[yuecong]

I have one existing ES cluster( 7.x) and running in https mode with a self-signed cert.
I can connect it using

 curl https://13.x.x.x:9200/_cluster/health -u username:password -k

I configured the ES backend as

spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: "https://13.x.x.x:9200"
        username: username
        password: password
        tls:
          skip-host-verify: true
      es-provision: false

But I got the error as

"failed to create primary Elasticsearch client: health check timeout: Head https://xx.xxx.xxx.xxx:9200: x509: cannot validate certificate for xx.xx.xx.xx because it doesn't contain any IP SANs: no Elasticsearch node available

How can I config jaeger to connect to a self-signed ES server? My server also supports client cert authenticate if it is the only way to do so.

Thanks

[yuecong]

Also, how can I map the options from the cli options to the options to be specified inside the operator?
https://www.jaegertracing.io/docs/1.14/cli/#jaeger-collector-elasticsearch

[jpkrohling]

It should be sufficient to use es.tls.skip-host-verify as you did. Could you check whether the underlying deployment is being created with this option? Something like:

kubectl describe deployment {the-jaeger-collector-deployment-name}

The options block from the CR are translated as CLI parameters to the underlying component. Like:

options:
  es:
    server-urls: https://13.x.x.x:9200

becomes:

--es.server-urls="https://13.x.x.x:9200"

[yuecong]

Thanks for the tip. Here is the output when I describe deployment

  Containers:
   jaeger-query:
    Image:       jaegertracing/jaeger-query:1.14.0
    Ports:       16686/TCP, 16687/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      --es-archive.password=xxxx
      --es-archive.server-urls=https://13.xx.xxx.xx:9200
      --es-archive.tls.skip-host-verify=true
      --es-archive.username=xxxx
      --es.password=xxx
      --es.server-urls=https://13.xxx.xxx.xx:9200
      --es.tls.skip-host-verify=true
      --es.username=xxx
    Readiness:  http-get http://:16687/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SPAN_STORAGE_TYPE:    elasticsearch
      JAEGER_SERVICE_NAME:  jaeger-es.jaeger-tracing
      JAEGER_PROPAGATION:   jaeger,b3

But I still have the error as

it doesn't contain any IP SANs

[jpkrohling]

Are you able to replace the IP in the connection string with a hostname?

[yuecong]

yep. here is the error message now

"error":"failed to create primary Elasticsearch client: health check timeout: Head https://elasticsearch-external-svc.elasticsearch.svc.cluster.local:9200: x509: certificate is valid for Node, not elasticsearch-external-svc.elasticsearch.svc.cluster.local: no Elasticsearch node available"

I have one existing cluster in another namespace in the same k8s cluster. I think somehow the ignore certificate validation is not set, otherwise, it does not need to check the cert at all and there should not be one message as certificate is valid for Node

btw, I tried both cluster IP and the public IP from the same service and both have the same error as it doesn't contain any IP SANs.

[yuecong]

does this need to set --es.tls to be ture? But from the doc, it says it is for client cert authentication. If I do that, I have to pass client cert to do the auth rather than using username/password to do the auth.

[jpkrohling]

@pavolloffay looks like there's some inconsistency, as pointed out by @yuecong:

https://github.com/jaegertracing/jaeger/blob/fcc0adb751cb9e858a87e5f29ea393d0c8d813f5/pkg/es/config/config.go#L294-L302

Looks like there's a place where TLS=true means that client auth is enabled (which isn't the case).

[jpkrohling]

By the way, this seems to be a problem with Jaeger itself, not the operator.

[yuecong]

Agree this is related to Jaeger rather than the operator as it will be true for all type of deployment. I will work around it for now to use client cert. would like to help to test once it is fixed. Thanks

[pavolloffay]

I am closing this jaegertracing/jaeger#1918 will support insecure TLS without tls.enabled=true. It will be available after Jaeger 1.15