-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Secret Discovery Service for v2 APIs #1194
Comments
We don't have a good story/plan for this right now. However I recognize that it's an important use case. For now is it possible to hot restart the entire Envoy process until we figure out the right approach/resources for solving this? |
Thanks. Our process monitoring currently does not support hot restarting the Envoy, but we can probably find a work-around. |
So, if we add KDS, we could have Envoy's client cert (presumably specified in |
But that doesn't implicitly handle rotation. If we want rotation KDS must be polling and able to update the context inline. |
(Which I guess is where we were going anyway) |
Yeah, it should support server initiated updates like the other APIs. I'm not if I fully grok the implementation complexity of this - can't we have a map somewhere from cert name to certificate and just have KDS map this essentially, with LDS consuming (for server certs) and also the xDS APIs consuming (for Envoy management server client certs). |
Basically the question is where the KDS polling lives and how it actually physically does the update in the code if there is a key material change. For example, there are a few concrete implementation options:
I think either could work. Whoever does the actual implementation of this will need to do some design work. |
Just to make it a little clear to me, in terms of triggering runtime cert rotation, I think there are two cases (please correct me if I misspoke):
Another question is (just thinking aloud), if the certificate is changed (SAN changed) in the middle of a connection, how to re-authenticate? This is like a corner case but I think we may need to cover it some time in the future. |
Would it be possible to continue using the old cert until the connection can be drained, and only use the new cert for new connections? |
Yes this is how it will work. |
I doubt that is a security issue. Suppose a client has a certificate with a specific SAN for only 10 min, if he uses that cert to establish a one-day connection, is that a problem? |
We could use this feature in Istio (istio.io). |
I've had multiple requests for an additional API to fetch key data since key data might want to come from a different management server. Probably worth keeping this open to track? |
OK, I will retitle to Key Discovery Service then. |
As an Envoy user, will I need to implement an KDS API in order to support key-rotation without restart? Or would there be a way to get new cert & key from the filesystem directly? |
We have concrete plans today to allow dynamic updates via the filesystem if the listeners are specified in a file on the filesystem (e.g. When we add KDS, this would probably work the same way, except the restriction this time would be no KDS server. |
I have a feeling we may need to add key-rotation directly via filesystem for Lyft use so it will come one way or the other. |
This issue came up again in the Istio community call today. Would you be open to a contribution of this feature? |
Yes. I would like to make progress on this. I've been discussing this off and on with various people. cc @PiotrSikora @myidpt @PiotrSikora What is currently plan for key delivery on your end? Ultimately I think we want to support dynamic reload via KDS API and also probably via watching FS file. |
Yes, the FS watch is specifically what we were interested in. |
FS watch is unlikely to be part of KDS API. I think it would be a direct addition to Listener/Cluster TLS context. @PiotrSikora can advise. |
I've been thinking about KDS (and cert rotation in general) this week. I'll write up a doc and share it either tomorrow or early next week, but it basically boils down to KDS pushing named FS watch, while nice in theory, is not only much more limited than KDS, but also prone to race conditions, since updates to the certificate chain and key files won't be done in a single transaction. |
@PiotrSikora I see something above (link) about
That should address the race-condition concern, no? |
I started working on the proof of concept implementation. |
@PiotrSikora, do you have any estimation when the support in envoy of the Secret Discovery Service will be implemented? |
@vadimeisenbergibm @mangchiandjjoe just started working on it. |
SDS uses the same style of xDS bidi stream of discovery requests and
responses. You can find a go stub in go control plane, for example.
…On Tue, Feb 27, 2018, 7:54 PM Piotr Sikora ***@***.***> wrote:
@vadimeisenbergibm <https://github.com/vadimeisenbergibm> @mangchiandjjoe
<https://github.com/mangchiandjjoe> just started working on it.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1194 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJGIxmuAY5wYtHvdcDLDynC8pL1gg2Ijks5tZM4NgaJpZM4OK1gB>
.
|
Thanks, @kyessenov, I see. |
Implement SDS API and dummy socket, and they are not in use. This is split from PR #4176. Risk Level: Low Testing: Unit tests Docs Changes: None Fixes #1194 Signed-off-by: JimmyCYJ <[email protected]>
Description: Use SDS api that fetches secrets from remote SDS server. Secrets are stored in Secret Provider. Listeners and Clusters are updated when secrets are received. Risk Level: Low Testing: Unit tests and integration tests Fixes #1194 Signed-off-by: Jimmy Chen [email protected]
Refactor SdsApi to support dynamic certificate validation context, and support Envoy to fetch certificate validation context from remote server via SDS API. Risk Level: Low Testing: Unit tests and integration tests. Fixes #1194 Signed-off-by: JimmyCYJ <[email protected]>
What is the status of this? I see this specific ticket is closed but there doesn't appear to be any published documentation regarding this feature. |
It has been fully implemented. Documentation is kind of hiding here But it may be easier to read integration test on how to use it. |
@qiwzhang it was on my list, but I forget before I went on leave, but we should have arch documentation on this feature. I'm going to open a doc issue on that. |
@qiwzhang that's awesome thanks. |
Description: Custom yaml will be treated as a configuration template, rather than a fixed set of configuration. It's still possible to pass fixed configuration, simply by not including any tokens for interpolation. This usage was already present and appeared supported by the API (e.g. see HttpBridgeTests), but simply wouldn't work, without any indication that it wouldn't. Risk Level: Low Testing: Local & CI Signed-off-by: Mike Schore <[email protected]> Signed-off-by: JP Simard <[email protected]>
Description: Custom yaml will be treated as a configuration template, rather than a fixed set of configuration. It's still possible to pass fixed configuration, simply by not including any tokens for interpolation. This usage was already present and appeared supported by the API (e.g. see HttpBridgeTests), but simply wouldn't work, without any indication that it wouldn't. Risk Level: Low Testing: Local & CI Signed-off-by: Mike Schore <[email protected]> Signed-off-by: JP Simard <[email protected]>
Hi,
We want to be able to rotate Envoy certs/keys without having to restart the Envoy. We're particularly interested in the client cert/key for connecting to upstream clusters. We recognize the CDS allows us to dynamically change clusters, but that doesn't let us rotate the cert used to connect to the CDS itself.
We see prior discussion about supporting cert rotation (#891) which was answered by pointing people towards the in-development LDS (#315). However, we don't see how the LDS solves the problem of rotating certs used to connect to the CDS.
What work would be required for support cert/key rotation for CDS without having to restart Envoy? This issue is a blocker for us using Envoy (we're looking at integrating it, and perhaps Istio, into Cloud Foundry).
We might be able to contribute the feature, if we had some guidance.
Cheers,
Angela and @rosenhouse
The text was updated successfully, but these errors were encountered: