TLS version setting is not documented

[duj4]

What's wrong?

This is per a closed issue: grafana/agent#4698

We installed GA 0.37.2 on Windows Server without changing any configuration, error posted in the log files like below:

When I tried to curl the /metrics, it replied with error:

When I used openssl to connect the server:

Steps to reproduce

1. Install GA 0.37.2 on Windows Server 2019
2. Configure "windows_certificate_filter" with server and client enabled
3. Start GA with https enaled
4. curl server with

System information

Windows Server 2019

Software version

Grafana Agent 0.37.2

Configuration

server:
  log_level: debug
  http_tls_config: &tls_config
    windows_certificate_filter:
      server:
        system_store: LocalMachine
        store: My
        issuer_common_names:
          - "<common name>"
      client:
        issuer_common_names:
          - "<common name>"
  grpc_tls_config: *tls_config

Logs

No response

[duj4]

May I know where I could set the TLS version and how? As it stated that TLS version and cipher suites are not restricted in the release notes of 0.37.1.

[duj4]

Hi @mattdurham , could you please help check this one? Thanks.

[jkroepke]

Does it work, if you omit windows_certificate_filter.client

[duj4]

@jkroepke no, the same error

[jkroepke]

What about

openssl s_client -connect localhost:12345 -tls1_2

[duj4]

hi @jkroepke , I tried it on the windows server, please check the response:

I exported the cert from windows certstore and tried again:

[jkroepke]

And what happens, if you are using MS Edge?

Maybe RSA PSS certificates has some issue with go.

[mattdurham]

Apologies was at kubecon, it is an undocumented set of values you can see them in the code here. https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L17 . Under http_tls_config you would have a min_version and max_version flag. Valid values are listed here https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L115

[mattdurham]

If I recall correct you need a specific openssl to work with windows properly. I found it easier to import the certificates into firefox/chrome trust them and then hit the endpoints.

[duj4]

hi @mattdurham , thanks for the reply.

Apologies was at kubecon, it is an undocumented set of values you can see them in the code here.

https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L17

. Under http_tls_config you would have a min_version and max_version flag. Valid values are listed here
https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L115

Yes, I was using this config.

If I recall correct you need a specific openssl to work with windows properly. I found it easier to import the certificates into firefox/chrome trust them and then hit the endpoints.

The error posted when prometheus is trying to scrape it, not just from browser.

[duj4]

And what happens, if you are using MS Edge?

Maybe RSA PSS certificates has some issue with go.

I tried both Edge and Chrome, neither of them could be loaded, I am suspecting the code as well.

[duj4]

Apologies was at kubecon, it is an undocumented set of values you can see them in the code here.

https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L17

. Under http_tls_config you would have a min_version and max_version flag. Valid values are listed here
https://github.com/grafana/agent/blob/c2c40f8a27abe71e35eedf440ab25ae44bf8a9ff/pkg/server/tls.go#L115

Hi @mattdurham , no matter I set what TLS version is, error like "last octet invalid" will post out.

[mattdurham]

You may need to limit your suites to the one the certificate is using. The windows certificate store doesnt let you investigate the certificate so its very possible the default cipher suites are not suitable. Are you exporting your keys with the full trust chain?

I documented how to setup from a developer perspective but likely good documentation to review. I generally find the windows certificate store and testing it is extremely particular about what it accepts.

[duj4]

hi @mattdurham ,

Thanks for the doc, as the template is set up already by our security and which is widely used by other teams, I cannot modify it and here is the screenshot:

And here is the config file we are using now:

The cert had been imported to the Edge browser, but it is showing error as below:

Any further thought would be much appreciated.

[mattdurham]

I wonder if its related to golang/go#45990

[duj4]

I wonder if its related to golang/go#45990

Yes, I checked this one as well but found it is almost one year ago, so :P

[duj4]

@mattdurham would like to test or try anything else if there is any, thanks

[mattdurham]

Its so hard to test, since its certificates and you cant really share that. Does it work if you dont use the certificate store but instead export them and reference them via files?

[duj4]

Thanks @mattdurham , I tried it that way and which looks good:

config file snippet with TLS specified:

able to access the metrics/windows_exporter endpoint:

[duj4]

hi @mattdurham , may I know if there is any update on this issue?

[mattdurham]

No I cant recreate. So mostly at a dead end on my side. If its possible to share the certificates or dummy generated ones maybe I could dig into it from that angle.

[duj4]

@mattdurham sure, how could I send it to you? Though it is a test one, I would like to send it in a safe way, an email would be better.

[mattdurham]

Community slack channel is likely the best way, mattd there

[duj4]

Thanks @mattdurham , have sent the cert and key to ya via slack channel

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[duj4]

Hi All,
Is there any update on this issue? We are rolling out Grafana Agent to windows servers in production environment and this is becoming an blocker for now.
Appreciate if there is any further help.

[duj4]

We have tried 0.39.2, issue still persisted.

[duj4]

Thanks @mattdurham for the great help, the modified version (https://github.com/mattdurham/agent/releases/tag/cerstore2) fixed the cert store issue, but there is some limitation, here is the workable config:

If we set tls version to 1.3, even it is a max one, the connection will be broken:

Besides, though I set the cipher_suites in the config (), when I tried to connect it with another cipher, the connection is still set up:

The windows server we are running on is 2019, it does support TLS1.2 only, but how about the cipher suites? I think Grafana Agent should only support the connection with the listed ones but deny the connection with others.

Please help on this one.

Thanks.

[rfratto]

Hi there 👋

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[duj4]

@rfratto @mattdurham the TLS cipher issue is still pending for resolution, may I have any update on this? The deadline from our security team is around July 14th.

[mattdurham]

I never could satisfactorily recreate the issue, or resolve it in a manner that felt safe and acceptable. It is unlikely to be resolved without additional information or outside effort.

[duj4]

@mattdurham thanks for the update Matt, please feel free to let me know what kind of info you'd like to collect, another session would be fine.