Failed key preparations can result in extraneous keys being created in your KeyManager

[keeganwitt]

We had an AWS account in which SPIRE created 16,000 keys it wasn't using. If we had not caught this, we would have been charged $1 per key per month, so an additional $16,000 would have been charged to us. In this example we are using AWS KMS, but this can apply to other KeyManager plugins as well (such Azure or GCP).

The problem is that PrepareX509CA() and PrepareJWTKey() in manager.go have no logic to delete the newly created key if the function fails at some point after it's creation. For example, if it fails to publish the JWT or if getting the upstream server to sign the X.509 key fails.

[keeganwitt]

We do also have evidence of the server pod crashlooping (106 times). So, another question with this issue is how to make the key creation an atomic operation.

[amartinezfayo]

Thank you @keeganwitt for opening this issue and bringing it to our attention.
This is certainly an issue that we want to address in SPIRE. We need to improve the handling of errors while new X.509 and JWT keys are prepared. This will probably involve some refactor in the PrepareX509CA() and PrepareJWTKey() functions, and make changes needed so SPIRE does not discard the created keys even if some later step failed.
I'm putting this in the 1.11.1 milestone, so we can start planning the work on this soon.