Allow reading of secrets to environment variable for containerised lambdas

[kenoir]

What does this change?

Adds a lambda extension to read variables from secrets manager, this is required as there is not a drop in replacement for the mechanism used in ECS where AWS will convert secrets manager references to secrets for us.

This approach uses the Lambda Extension API, which is available to containerised Lambdas by adding files to /opt/extensions/.

For simplicity (by comparison to including some other language runtime to run extension code and/or providing a binary to package), we use a bash script extension, based on the example here..

How it works:

• In terraform the pipeline_lambda module uses the existing map of environment variable names to secret names to generate a map of environment variable names to secret names prefixed secret:, in addition we also provision permissions for the lambda to access those secrets.
• bash_secrets_extension.sh looks for environment variables values that have been passed to the lambda that have been prefixed with secret: and retrieves them from AWS Secrets Manager
• The extension creates a file in the containerised environment called /tmp/config that will persist between invocations, the file uses the Typesafe Config format (HOCON).
• Inside the Scala application we layer global, application and the config loaded from /tmp/config to resolve a final set of configuration in LambdaConfiguration, allowing the values from /tmp/config to substitute for environment variables where they don't already exist.

How to test

• Run the relation_embedder lambda using RIE and docker-compose with a valid .env file.
• Deploy this change, and terraform update to a non-production pipeline, observe the lambda reading the secrets.

Screen.Recording.2024-12-13.at.08.53.20.mov

How can we measure success?

This lambda, and all other ones that require secrets have a mechanism to do so that require no application or significant terraform changes.

Have we considered potential risks?

Yes, this change involves handling and providing secrets to our services. We must take care not to log or reveal these in code or application logs.

This change intends to keep secrets encrypted at rest within secrets manager until they are required by a lambda invocation. This follows an AWS recommended pattern, that unfortunately we can't take direct advantage of as we are using containerised lambdas, but follows the same mechanism.

[kenoir]

The .env file contains the remaining pipeline specific variables, we could have a script to generate a file like this:

es_apikey=secret:elasticsearch/pipeline_storage_2024-11-18/relation_embedder/api_key
es_host=secret:elasticsearch/pipeline_storage_2024-11-18/public_host
es_denormalised_index=works-denormalised-2024-11-18
es_merged_index=works-merged-2024-11-18
es_protocol=https
es_port=443

[kenoir]

Script added, as part of run_local.sh

[kenoir]

For use in other container images, this should be in a shared location eventually.

[kenoir]

This event loop seems to be necessary to keep the Lambda running happily. The extension is invoked as a separate process from the main lambda, but the extension API will fail the start-up if we exit before the lifetime of the lambda container itself.

There may be scope to simplify this!

[paul-butcher]

Would sleeping for 15 minutes work here, or does the extension have to explicitly listen for the shutdown event to avoid keeping the Lambda awake?

[kenoir]

Good question, I want to double check my assumptions here too. I'll check.

[kenoir]

After some experiments, removing the while loop here results in the following error at lambda initialization:

It looks like exiting early results in stopping the runtime interface emulator (RIE) from invoking the main lambda.

Would sleeping for 15 minutes work here

Putting a sleep 900 instead of the loop, does allow the lambda to be invoked and terminated (though the logs mutter about a forced stop). However a 2nd invocation then fails with the following error:

So I think that messes with some logic around invocation in the RIE in a way that may be reproduced in an in-situ invocation.

[paul-butcher]

That makes sense. I expect it starts when a lambda is first switched on and keeps going until there's nothing left to do, rather than running for each invocation. This allows warm starts to benefit the most from that warmth.

[kenoir]

This pattern is intended for re-use, in. combination with use of template.env and local.docker-compose.yml files. The intention is to work towards running any of the lambda services using the same script / pattern from the root with one script and a docker-compose.yml for all the services.

[paul-butcher]

I'd be tempted to call this a build script and end it here (or have a build and a run script).

Although recompiling with SBT is reasonably efficient, it still takes about 15 seconds to reach this point when there's nothing to do.

(not a blocker)

[kenoir]

I've added a --skip-build flag to do this optionally.

[paul-butcher]

This kind of thing makes me happy.

Could we even go one step further and hide val config from the Lambda, instead translating the relevant config to a case class or trait so that all the string-based references to properties are in one place?

I suspect it has too many tendrils to do cleanly as part of this change, but might be a worthy ticket to raise for a future improvement

[kenoir]

Something like:

// This is all shared
trait ApplicationConfig {}
case class MyApplicationConfig(someValue: String, someOtherValue: Int) extends ApplicationConfig

trait ConfigurationBuilder[C, T <: ApplicationConfig] {
  protected val rawConfig: C

  def build(rawConfig: C): T
  def config: T = build(rawConfig)
}

trait TypesafeConfigurable[T <: ApplicationConfig] extends ConfigurationBuilder[Config, T] {
    def build(rawConfig: Config): T
}

trait LambdaConfigurable extends TypesafeConfigurable[MyApplicationConfig] {
  private val defaultResolveFromFile: String = "/tmp/config"
  private val defaultApplicationConfig: String = "application.conf"

  private val lambdaConfigFile: File =
    new File(defaultResolveFromFile)

  protected val baseConfig: Config =
    ConfigFactory.load()

  protected val applicationConfig: Config =
    ConfigFactory.parseResources(defaultApplicationConfig)

  protected val lambdaConfig: Config = if (lambdaConfigFile.exists()) {
    ConfigFactory.parseFile(lambdaConfigFile)
  } else {
    ConfigFactory.empty()
  }

  lazy val rawConfig = lambdaConfig
    .withFallback(applicationConfig)
    .withFallback(baseConfig)
    .resolve()
}

// Then in an app do
trait MyAppConfigurable extends LambdaConfigurable {
  def build(rawConfig: Config): MyApplicationConfig = {
    MyApplicationConfig(
      someValue = rawConfig.getString("someValue"),
      someOtherValue = rawConfig.getInt("someOtherValue")
    )
  }
}

// And in LambdaMain
object LambdaMain
    extends RequestHandler[SQSEvent, String]
    with Logging
    with MyAppConfigurable {

      // config: MyApplicationConfig is available in this scope

}

Yes, I do prefer having one place where we extract the config from typesafe. I might have a go at this in a future PR 👍