Selecting SSL certificate dynamically

[ohadvano]

Assuming I am using SDS and I have on my disk a lot of certificates where the name of the certificate is a variable "x.cer".
I have a listener filter that would dynamically, per connection, resolve the value of x.
Would it be possible to apply downstream TLS context and dynamically select the presented certificate, based on the resolved value? Thanks

[mattklein123]

I'm sure you can do whatever you need if you write a custom handshaker. I'm not sure if this can be done out of the box. Your description is hard to understand so perhaps you could make it more clear on what exactly you are trying to achieve. cc @ggreenway @lizan

[ohadvano]

Basically, for a multi tenancy model, I would like to present each tenant with a different certificate. The tenancy information would be resolved in a listener filter, prior to the TLS handshake, so I would like to choose the matching certificate per tenant dynamically.

[mattklein123]

Yeah I think you will likely need to create a custom handshaker but it should be possible.

[ohadvano]

My concern is that custom handshaker would somehow mess the handshake and compromise security (even though I read it's possible to take the normal handshaker as base and modify the minimal parts needed).
Would it be possible to set a filter state object to override (if exists) the name of the required certificate per-connection? Seems like the most minimal change required to support this without creating a custom handshaker.

[ggreenway]

You can currently specify a number of certificates and have the correct one chosen based on the SNI presented in the handshake, but there's not support for looking on disk for them when the TLS handshake begins; all certificates are loaded when the config is loaded (or fetched via SDS).

[mattklein123]

Freel free to propose a change and we can take a look. If filter state is possible to get to in the handshaker this seems like something we could potentially support.

[ohadvano]

@ggreenway I'm afraid the SNI lookup would not be sufficient because two tenants might request the same SNI, although I would want to present each one with a different certificate

[ggreenway]

Sure, we could add some way for a listener filter similar to tls_inspector to indicate which certificate should be used.

But it would only work if the certificate(s) were already loaded; reading them from disk at this point would be very difficult and would introduce performance issues as the disk read could potentially block in the kernel for an arbitrary amount of time.

[ohadvano]

Yeah I agree, I didn't mean to directly access the disk, but use the SDS as a broker, the mechanism for certificates loading would be the same, only the name of the secret would be configured dynamically per-connection (if specified by filter state)

[ggreenway]

That sounds reasonable to me. Want to put together a really rough version of a PR so we can make sure we're on the same page?

[ohadvano]

Absolutely, I'll get to it in next few days, hopefully tomorrow

[ohadvano]

@ggreenway I had a thought about adding the matcher API to CommonTlsContext / SdsSecretConfig an option of "oneof" between "name" and "name_matcher", i.e:

{
  "name": ...,
  "name_matcher": ...
  "sds_config": {...}
}

The "oneof" should make this a non-breaking for those who use "name" already and enable the usage of Matching API for deciding upon the name of the secret to fetch, this way it could be applicable to both downstream and upstream TLS context. WDYT?

I still intend to start with the filter state override, I don't have enough capacity to add the matcher API right away, but I think it could be a more cleaner way to decide upon secrets name.

[ggreenway]

You can't wrap existing fields in oneof, unfortunately: https://github.com/envoyproxy/envoy/blob/main/api/API_VERSIONING.md?plain=1#L57

But I like the idea of using the matcher API for extending certificate selection.

[ohadvano]

How do you suggest to integrate this into the api config?

[ggreenway]

Thinking about this, I'm not sure it applies to upstream connections. The current SNI matching is only on the downstream side.

I think you could just add some config to DownstreamTlsContext to specify a matcher. But I've only given it a quick glance; I might be missing some details here.

[ohadvano]

I'm confused a little, how is SNI impacting name of secret to fetch using SDS?
As I see it, there's SDS integration for both downstream and upstream TLS context, I mean, in both cases you can dynamically fetch the certificate using SDS. Currently you need to provide a secret name for SDS to look, I'm suggesting adding matcher so the name for the secret to look for, would be dynamically resolved by the matcher in runtime. Have I missed something?

[ggreenway]

All of the secrets are fetched from SDS when the config is loaded; the name of the secrets to fetch is not dynamic.

Then, from the loaded certificates, we match the SNI of the incoming TLS handshake to one of those fetched certificates.

[ohadvano]

Have I completely misunderstood how SDS works then? When we discussed earlier in this thread about overriding the name by filter state, what was the meaning as you understood it? Because if all certificates are loaded by SDS at bootstrap, then the name in the config must be static, isn't it?

[ggreenway]

I understood that as the name of the certificate to use. But I assumed that the cert was already loaded.

[ohadvano]

The field "name" in SdsSecretConfig configures the name of the certificate to use (which was already loaded at bootstrap), or configures the name of the certificate that SDS should load at bootstrap?

[ggreenway]

I believe that field is sent to the sds server, so that it knows which secret to send back.

[ohadvano]

At which time is this field sent to the server? On-demand (when TLS handshake requested) or at bootstrap?

[ggreenway]

When the configuration is loaded.

There is not a way at this time to load additional secrets during the TLS handshake.

[ggreenway]

Although the SDS server can push a new version of named secrets to Envoy at any time. But I believe it can only push 1 certificate for each named request. I don't think there's currently a way for Envoy to ask SDS for "give me everything you have".

But it may be possible to implement what you're asking for. I haven't looked closely enough to know whether it's feasible.

[ohadvano]

So that means that if I have two listeners and each has downstream TLS context, one with name "x1" and the second "x2", then I'll end up with only two certificates in Envoy, for these names?

If that's true then I don't believe adding option to override used cert by filtet state is meaningful, because it could still be either "x1" or "x2".

Another question here is what happens in case LDS is configured? As I understand it, in LDS case the listeners are not static and can change throughout the lifetime of the Envoy process. In that case, each new listener might introduce a new requested secret that was previously unknown, so how the SDS handles this case?

I expected Envoy to fetch all the certificates the SDS wants to provide at bootstrap, and refresh the inventory from time to time, then the field name would be for selecting a certain certificate out of all of the certificates that the SDS has provided.

As I understand, step 1 for supporting this dynamic selection would be adding an option to fetch all certificates provided by SDS without requesting for specific names at fetch time?

[ggreenway]

Yes, I think your understanding is correct. If you cannot enumerate the SDS name of each certificate in the envoy configuration right now, there isn't a way to fetch all the certificates.

One possible workaround is to use LDS instead of SDS, and configure your listener with all the certs needed, and do a listener update whenever that list changes. It's not a great solution, but it would work.

[ohadvano]

@ggreenway so that means that if a new listener is loaded through LDS with a new secret, then it would request that delta secret from SDS accordingly?

[ggreenway]

Thanks Greg, couple of question per your example:

1. With your approach, does it mean that each listener/cluster will have its own "collection" of secrets that only the relevant listener/cluster can use?

Kind of. It depends on how you think about it.

For actually using a TLS certificate, it is ALWYS loaded specifically for the context in question as a BoringSSL SSLContext, no matter how we do things.

If multiple parts of the envoy configuration want to use the same secrets (ie the request to the SDS server is identical), those get de-duped and only fetched once.

2. What would happen if ListenerA has "fetch_all_the_secrets: true" and listenerB has "fetch_all_the_secrets: false"?

listenerA would issue a request that fetches all secrets, and listenerB would issue a request to fetch a single secret

3. How is the "name" field you were introducing supports dynamically populating the name of the secret to use? (as was presented before with filter_state_key/matcher tree)

There would be a filter-state with a fixed name. If it has been set, it will contain the name of the secret to use. This name will need to match the name that the SDS server associated with the secret.

4. How does LDS/CDS currently work in terms of utilizing multiple management servers? Can I get LDS config from two different xDS servers?

No, LDS/CDS can only be from one mgmt server, because there is a single list of listeners and a single list of clusters in envoy.

But TLS-using-SDS does support different mgmt servers for each TLS context; we should not discard that support here.

[ohadvano]

But TLS-using-SDS does support different mgmt servers for each TLS context; we should not discard that support here.

Ok, I guess that gives me good idea about why not using dynamic_resources config, thanks.

There would be a filter-state with a fixed name. If it has been set, it will contain the name of the secret to use. This name will need to match the name that the SDS server associated with the secret.

Could we instead make current name: field optional, and introduce filter_state_key: field or mater_tree: field that, if configured, will signal Envoy to use a filter state key to resolve the name of the secret? Seems to me that it's more user approaching. Envoy then needs to validate that both name and filter_state_key are not configured simultaneously.

Kind of. It depends on how you think about it.
For actually using a TLS certificate, it is ALWYS loaded specifically for the context in question as a BoringSSL SSLContext, no matter how we do things.
If multiple parts of the envoy configuration want to use the same secrets (ie the request to the SDS server is identical), those get de-duped and only fetched once.

Still not sure I get this part. Assuming Envoy has two listeners that both enabled fetch_all_the_secrets: true, Envoy would make two SDS requests at bootstrap, no? Practically making SDS respond twice with all its secrets.

Additional question, if fetch_all_the_secrets: true is set, what would be the named_resource field value in the discovery request made to SDS? wildcard?

[ggreenway]

There would be a filter-state with a fixed name. If it has been set, it will contain the name of the secret to use. This name will need to match the name that the SDS server associated with the secret.

Could we instead make current name: field optional, and introduce filter_state_key: field or mater_tree: field that, if configured, will signal Envoy to use a filter state key to resolve the name of the secret? Seems to me that it's more user approaching. Envoy then needs to validate that both name and filter_state_key are not configured simultaneously.

What I'm saying is that there's no need to make the filter-state key configurable. The key should be fixed, and if it is set, then it will be used. If it is not set, the current behavior will be used. Without something new to envoy (new listener filter, etc) to set it, it will never be present and so the current behavior will be used.

Filter-state is used for filters to pass information to each other. How this works is setup in code, not configuration. That's just how filter-state is used in Envoy.

But if you want to use a match-tree, that will definitely need configuration.

But one thing to keep in mind is that either a match-tree or filter-state is not directly related to SDS; both would work with other methods of adding certificates (such as pointing to a list of certs on the filesystem). The process for loading certificates (which happens at config-load time) and selecting a certificate for a connection (done at TLS handshake time) are completely separate processes. There are different ways to provide certs (SDS, filesystem) and ways to select a cert (built-in SNI-based selection, custom handshaker). These two concerns are not tightly coupled.

Kind of. It depends on how you think about it.
For actually using a TLS certificate, it is ALWYS loaded specifically for the context in question as a BoringSSL SSLContext, no matter how we do things.
If multiple parts of the envoy configuration want to use the same secrets (ie the request to the SDS server is identical), those get de-duped and only fetched once.

Still not sure I get this part. Assuming Envoy has two listeners that both enabled fetch_all_the_secrets: true, Envoy would make two SDS requests at bootstrap, no? Practically making SDS respond twice with all its secrets.

No, envoy would send one SDS request, receive the response, and update both TLS contexts from the same response.

Additional question, if fetch_all_the_secrets: true is set, what would be the named_resource field value in the discovery request made to SDS? wildcard?

I don't know yet. I want input from @adisuissa on how the SDS protocol should be setup for this use case. We just need some way to signal to the SDS server that this is the intent.

[ohadvano]

What I'm saying is that there's no need to make the filter-state key configurable

My motivation was having a more user-friendly interface, but I guess I can give this up.

But if you want to use a match-tree, that will definitely need configuration.

Yes, I think this is a cool feature to include and I can find some useful use cases with this. Would supporting this require making name field optional?

No, envoy would send one SDS request, receive the response, and update both TLS contexts from the same response.

Got it, so as long as there's at least one occasion that enabled fetch_all_secrets, Envoy fetches "all_secrets" just once. What am I missing here though? Feels to me like it contradict with the saying where each listener would have it's own bucket of secrets that it can use? I mean, if several listeners were using fetch_all_secrets, and Envoy ended up with fetching all secrets just once, then all those listeners can now have access to the same set of secrets, making the fetched secrets practically a global collection.

[ggreenway]

But if you want to use a match-tree, that will definitely need configuration.

Yes, I think this is a cool feature to include and I can find some useful use cases with this. Would supporting this require making name field optional?

No; the name field is part of the SDS config. The matcher is selecting which certificate to select for a specific TLS handshake. These are unrelated.

No, envoy would send one SDS request, receive the response, and update both TLS contexts from the same response.

Got it, so as long as there's at least one occasion that enabled fetch_all_secrets, Envoy fetches "all_secrets" just once. What am I missing here though? Feels to me like it contradict with the saying where each listener would have it's own bucket of secrets that it can use? I mean, if several listeners were using fetch_all_secrets, and Envoy ended up with fetching all secrets just once, then all those listeners can now have access to the same set of secrets, making the fetched secrets practically a global collection.

Actually, I misnamed it in the snippet above. It should actually be named something like allow_multiple_secrets. But there will still be a name specified. So different listeners could specify different names, and thus they'd may get different sets of secrets. But it could also be that the SDS server ignores the name field and just sends everything; that's a detail of a specific SDS server implementation.

[ohadvano]

Actually, I misnamed it in the snippet above. It should actually be named something like allow_multiple_secrets. But there will still be a name specified. So different listeners could specify different names, and thus they'd may get different sets of secrets. But it could also be that the SDS server ignores the name field and just sends everything; that's a detail of a specific SDS server implementation.

If that's the case, would it still be true that Envoy sends a single fetch request, for all secrets? It seems like now the name field has a practical meaning, because it would need to be part of the discovery request, so Envoy has to send out two different fetch requests to SDS ? But allowing SDS to respond with any count of secrets to choose from?

[ggreenway]

Envoy will send a single SDS request for each unique request. Any identical requests will be combined. So if you put the same name and cluster in each TLS context SDS config, they would all use identical requests.

[ohadvano]

So practically, to achieve dynamic secret selection from an unknown set of secrets, I would: set name field with the same arbitrary (but irrelevant) value for all listeners/clusters, set allow_multiple_secrets to signal Envoy that a response with more than one secret is fine, and set a filter state key or use matcher tree to specify which secret to use from what the SDS has returned?

[ggreenway]

Yes, I think that's correct.

[ohadvano]

Some more follow up questions then:

1. What if all tls_contexts use the same name value, but some set allow_multiple_secrets to true, and some are not?
2. If there are two listeners, both set allow_multiple_secrets, but with a different name, would in this case Envoy send two SDS requests?
3. How is the "buckets" idea currently implemented? A secret is associated with a name or a listener which requested it?

As a next step, I would like to support on-the-fly fetching of secret from SDS, meaning that if the secret isn't available at Envoy at the time of TLS handshake, then Envoy would fetch it from SDS and pause TLS handshake. Does this API model support this?

[ggreenway]

Some more follow up questions then:

1. What is all tls_contexts use the same name value, but some set allow_multiple_secrets to true, and some are not?

Then those are different requests, so you'll end up with 2 SDS server requests, one for each mode.

2. If there are two listeners, both set allow_multiple_secrets, but with a different name, would in this case Envoy send two SDS requests?

No, listener name is not related to SDS requests.

3. How is the "buckets" idea currently implemented? A secret is associated with a name or a listener which requested it?

Each TLS context has 1 or more certificates. When they are loaded, we figure out which SANs each one has. When a connection comes in, the correct one is chosen based on SNI.

I'm not sure what you mean exactly by "buckets".

As a next step, I would like to support on-the-fly fetching of secret from SDS, meaning that if the secret isn't available at Envoy at the time of TLS handshake, then Envoy would fetch it from SDS and pause TLS handshake. Does this API model support this?

That's a completely different feature than what we've been discussing here. The closest thing I can think of is on_demand config for tcpproxy. See https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto#envoy-v3-api-msg-extensions-filters-network-tcp-proxy-v3-tcpproxy for details.

[ohadvano]

Then those are different requests, so you'll end up with 2 SDS server requests, one for each mode.

I thought allow_multiple_secrets should only effect how Envoy handles the SDS responses? If not, does it mean that the request itself would look different if allow_multiple_secrets is set? Meaning the request to SDS includes some flag to indicate it respond with any count of secrets?

No, listener name is not related to SDS requests.

I meant different name value in tls_context, not the name of the listener.

I'm not sure what you mean exactly by "buckets".

Forget about buckets, I'm trying to understand the practice behind this API design. Envoy would request secrets from SDS, and let's say that the returned collection is relevant to both listenerA and listenerB, but not to listenerC. What would make those secrets available to be chosen only by listenerA/B but not C?

That's a completely different feature than what we've been discussing here. The closest thing I can think of is on_demand config for tcpproxy. See https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto#envoy-v3-api-msg-extensions-filters-network-tcp-proxy-v3-tcpproxy for details.

Referring to above comment about the full requirements I'm looking from this API change: #25958 (comment)

[ggreenway]

Then those are different requests, so you'll end up with 2 SDS server requests, one for each mode.

I thought allow_multiple_secrets should only effect how Envoy handles the SDS responses? If not, does it mean that the request itself would look different if allow_multiple_secrets is set? Meaning the request to SDS includes some flag to indicate it respond with any count of secrets?

That's an interesting point. I'm not sure if it ends up being a different request or not. If not, I don't think we even need an option; we could just allow multiple secrets anytime the SDS server sends them in a response (for TLS certificates; other secret types would need to be evaluated for whether that makes sense).

No, listener name is not related to SDS requests.

I meant different name value in tls_context, not the name of the listener.

Yes, the name is part of the request, so a different value is a different request.

I'm not sure what you mean exactly by "buckets".

Forget about buckets, I'm trying to understand the practice behind this API design. Envoy would request secrets from SDS, and let's say that the returned collection is relevant to both listenerA and listenerB, but not to listenerC. What would make those secrets available to be chosen only by listenerA/B but not C?

The SDS configuration in the TLS context specifies which secrets that context gets. If A and B should get the same secrets and C a different set, then A and B should use the same name in the SDS config, and C should use a different name. Then it's up to the SDS server to use those name values to respond with the correct set of secrets.

[ggreenway]

That's a completely different feature than what we've been discussing here. The closest thing I can think of is on_demand config for tcpproxy. See https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto#envoy-v3-api-msg-extensions-filters-network-tcp-proxy-v3-tcpproxy for details.

Referring to above comment about the full requirements I'm looking from this API change: #25958 (comment)

Right, but that is a different feature than either "fetch all" or "dynamically select a cert". That's a 3rd feature that we haven't discussed how to do yet.

[ohadvano]

The SDS configuration in the TLS context specifies which secrets that context gets. If A and B should get the same secrets and C a different set, then A and B should use the same name in the SDS config, and C should use a different name. Then it's up to the SDS server to use those name values to respond with the correct set of secrets.

I meant regarding internal implementation of Envoy - what enforces this so listenerA/B can't access C's secrets?

Right, but that is a different feature than either "fetch all" or "dynamically select a cert". That's a 3rd feature that we haven't discussed how to do yet.

How is it different than 2nd feature from that comment? "If it is not, then an on-demand request to the SDS will take place, asking for that secret, and the TLS handshake will be paused until the SDS responds with the required cert.".
Regardless if it's same feature or not, do you think it would be possible to implement with this API change?

[ggreenway]

The SDS configuration in the TLS context specifies which secrets that context gets. If A and B should get the same secrets and C a different set, then A and B should use the same name in the SDS config, and C should use a different name. Then it's up to the SDS server to use those name values to respond with the correct set of secrets.

I meant regarding internal implementation of Envoy - what enforces this so listenerA/B can't access C's secrets?

They're stored in the objects created for the TLS context. They just can't be found in the wrong place, because they're not there.

Right, but that is a different feature than either "fetch all" or "dynamically select a cert". That's a 3rd feature that we haven't discussed how to do yet.

How is it different than 2nd feature from that comment? "If it is not, then an on-demand request to the SDS will take place, asking for that secret, and the TLS handshake will be paused until the SDS responds with the required cert.". Regardless if it's same feature or not, do you think it would be possible to implement with this API change?

This API change doesn't help or prevent fetching secrets on-demand. They're not really related.

[ohadvano]

They're stored in the objects created for the TLS context. They just can't be found in the wrong place, because they're not there.

There would be reuse of memory though? Or will each TLS context have its own set of secrets? I mean, considering SDS responded with 1000 secrets that are relevant to both listenerA and B, would it require x2 memory?

[adisuissa]

That's a very long thread and thank you @ggreenway for shepherding this.

I just wanted to mention something that may be relevant to this discussion from the xDS protocol perspective.
The new xDS-Transport-Protocol (xds-TP) describes a Glob Collection, which essentially is a resource name of the following pattern: xdstp://<authority>/<type>/<path>/*.
For example: xdstp://acme.com/envoy.extensions.transport_sockets.tls.v3.Secret/group_listeners/group_A/* will return all resources of type envoy.extensions.transport_sockets.tls.v3.Secret with the group_listeners/group_A resource name "prefix". Each of the resources will have the same xDS-TP prefix in its name. For example, a specific resource name can be: xdstp://acme.com/envoy.extensions.transport_sockets.tls.v3.Secret/group_listeners/group_A/secret1.

[ggreenway]

They're stored in the objects created for the TLS context. They just can't be found in the wrong place, because they're not there.

There would be reuse of memory though? Or will each TLS context have its own set of secrets? I mean, considering SDS responded with 1000 secrets that are relevant to both listenerA and B, would it require x2 memory?

Yes, this is the case, but this will always be the case (for any of the proposed solutions in this thread).

My rough guess is that each TLS cert will cost around 10kb or so of memory per TLS context it is used in. If you're worried about memory use, I'd start by measuring the usage, which can be done without any code changes by configuring a bunch of certs using existing mechanisms.

[ggreenway]

@adisuissa I agree that glob collections look like a reasonable way to specify this.

Are there any compatibility issues we need to worry about? Or would it be ok to modify envoy to always accept multiple TLS certs in any SDS response for TLS certs?

[ohadvano]

Yes, this is the case, but this will always be the case (for any of the proposed solutions in this thread).

Why would this be the case if we were to use the approach of having a single global collection (as in LDS/CDS)?

The new xDS-Transport-Protocol (xds-TP) describes a Glob Collection, which essentially is a resource name of the following pattern: xdstp://<authority>/<type>/<path>/*.

Seems like it's an API proposal, is this pattern currently in use with Envoy?

Anyways, to use this pattern, we would have to somehow signal Envoy that a global collection is required, so there would be a requirement to add some boolean config to tls_context?

[ggreenway]

Yes, this is the case, but this will always be the case (for any of the proposed solutions in this thread).

Why would this be the case if we were to use the approach of having a single global collection (as in LDS/CDS)?

Read the code for the tls transport socket, and see how the secrets are used. There's not really a way to share them between contexts.

[ohadvano]

Read the code for the tls transport socket, and see how the secrets are used. There's not really a way to share them between contexts.

So no matter how certificates end up in Envoy, whether they are fetched by SDS, or statically referenced from file, etc.. each TLS context will always load all secrets? Is this something that can theoretically be changed or implemented in another way?

[ggreenway]

Read the code for the tls transport socket, and see how the secrets are used. There's not really a way to share them between contexts.

So no matter how certificates end up in Envoy, whether they are fetched by SDS, or statically referenced from file, etc.. each TLS context will always load all secrets? Is this something that can theoretically be changed or implemented in another way?

In theory, if ALL parameters of the tls context were identical, you may be able to share those between TLS contexts, but it would make the implementation more complicated, and there may be complicated lifetime issues to solve. If ANY parameter (explicit or implicit) of the TLS context is different, then I don't think they can be shared.

[ohadvano]

Bringing up again our discussion about scoped secrets with a small question. I can see that static_resources do reference secrets as a global resource. Following this definition, a static secret is not necessarily scoped to a listener/cluster, isn't it?

I'm trying to understand the difference when trying to apply the same to dynamic resources (which this discussion previously ruled out), would it be different simply because the secrets need to be assigned to a specific TLS context when they are loaded?

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.