crypto: fix propagation of "memory limit exceeded"

When we throw ERR_CRYPTO_INVALID_SCRYPT_PARAMS after a call to
EVP_PBE_scrypt, check if OpenSSL reported an error and if so, append the
OpenSSL error message to the default generic error message. In
particular, this catches cases when `maxmem` is not sufficient, which
otherwise is difficult to identify because our documentation only
provides an approximation of the required `maxmem` value.

Fixes: #53291
PR-URL: #53300
Reviewed-By: Filip Skokan <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>

src/crypto/crypto_scrypt.cc

@@ -104,7 +104,17 @@ Maybe<bool> ScryptTraits::AdditionalConfig(
           params->maxmem,
           nullptr,
           0) != 1) {
-    THROW_ERR_CRYPTO_INVALID_SCRYPT_PARAMS(env);
+    // Do not use CryptoErrorStore or ThrowCryptoError here in order to maintain
+    // backward compatibility with ERR_CRYPTO_INVALID_SCRYPT_PARAMS.
+    uint32_t err = ERR_peek_last_error();
+    if (err != 0) {
+      char buf[256];
+      ERR_error_string_n(err, buf, sizeof(buf));
+      THROW_ERR_CRYPTO_INVALID_SCRYPT_PARAMS(
+          env, "Invalid scrypt params: %s", buf);
+    } else {
+      THROW_ERR_CRYPTO_INVALID_SCRYPT_PARAMS(env);
+    }
     return Nothing<bool>();
   }
 

test/parallel/test-crypto-scrypt.js

@@ -178,7 +178,8 @@ for (const options of bad) {
 
 for (const options of toobig) {
   const expected = {
-    message: /Invalid scrypt param/
+    message: /Invalid scrypt params:.*memory limit exceeded/,
+    code: 'ERR_CRYPTO_INVALID_SCRYPT_PARAMS',
   };
   assert.throws(() => crypto.scrypt('pass', 'salt', 1, options, () => {}),
                 expected);