Detect invalid padding parameters when setting up an RSA context

[gilles-peskine-arm]

Context

mbedtls_rsa_init takes two arguments padding and hash_id which can be invalid. However, as an initialization function, it must return void, since it is often called in contexts that cannot jump to error handling because other objects are not initialized yet.

In Mbed TLS 2.16+, there was validation through MBEDTLS_PARAM_FAILED if MBEDTLS_CHECK_PARAMS was enabled. With the removal of MBEDTLS_CHECK_PARAMS, there is no way to enable any validation anymore.

Goal

Change to the API of the RSA module. Important requirements:

• Make a small change. We don't have much time.
• mbedtls_rsa_init cannot return errors. (But mbedtls_rsa_set_padding can.)
• An invalid padding value must be caught, preferably in a way that is easy to debug. In the current API, the padding can be passed to either mbedtls_rsa_init or mbedtls_rsa_set_padding, and it is stored in the RSA context object.
• Preferably an invalid hash should be caught too. (But note that MBEDTLS_MD_NONE is ok for V15 padding.)
• Although it is good hygiene for a key to only be used with a single kind of padding, this is not a strong requirement. It may be possible to stop storing the padding parameters in the context altogether.

Possible solutions

This is not an exhaustive list.

• Stop storing the padding in the context altogether?
• Have mbedtls_rsa_init set the padding to some documented default, and require the caller to call mbedtls_rsa_set_padding if some other default is desired. Change mbedtls_rsa_set_padding to return MBEDTLS_ERR_RSA_BAD_INPUT_DATA if the arguments are invalid.
• Other, specify.