Support internal Lambda extensions

[ramosbugs]

Internal Lambda extensions must be registered during the Lamba lifecycle Init phase, which ends when the runtime calls Next to await the first event. Since the Extension::run function both registers and executes the extension, there was previously no way for the runtime to determine that all internal extensions have been registered and that it's safe to proceed to the Invoke lifecycle phase.

This change introduces an Extension::register method that registers the extension and begins any logs/telemetry handlers. It then returns a new RegisteredExtension abstraction that can be used to invoke the extension's run loop concurrently with the runtime's run loop.

This change maintains backward compatibility by having the existing Extension::run method perform both steps. External Lambda extensions can use either API, and internal extensions should use the new API.

Resolves #743.

By submitting this pull request

• [X ] I confirm that my contribution is made under the terms of the Apache 2.0 license.
• [X ] I confirm that I've made a best effort attempt to update all relevant documentation.

[ramosbugs]

This new type seemed unavoidable since Extension::register has to pass ownership of all the logging and telemetry fields to the spawned background tasks. These are the remaining fields to be used in the invoke phase.

[calavera]

This is great, thanks for the changes! Do you mind creating an example in the examples directory? that way if anyone wants to create internal extensions they can use it as reference.

[ramosbugs]

The cargo lambda runtime emulator doesn't seem to support internal extensions, but I tested the example by deploying it to AWS via:

cargo lambda deploy
cargo lambda invoke extension-internal-flush --data-example sqs-event-obj --verbose --remote

This produced the following CloudWatch logs showing both the runtime and extension behaving as expected:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                 message                                                                                 |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1701561431609 | INIT_START Runtime Version: provided:al2023.v9 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:68313de216b90a01e19138b97177c40478ef50b63a22b29f152b960070428ae8  |
| 1701561431636 | EXTENSION Name: bootstrap State: Ready Events: [INVOKE]                                                                                                                 |
| 1701561431637 | START RequestId: 34bcc601-2f2d-4ad8-af6e-124968936a5a Version: $LATEST                                                                                                  |
| 1701561431637 | [runtime] received event Data { a: "Test", b: 123 }                                                                                                                     |
| 1701561431637 | [extension] waiting for event to be processed                                                                                                                           |
| 1701561431637 | [extension] flushing logs and telemetry                                                                                                                                 |
| 1701561431648 | END RequestId: 34bcc601-2f2d-4ad8-af6e-124968936a5a                                                                                                                     |
| 1701561431648 | REPORT RequestId: 34bcc601-2f2d-4ad8-af6e-124968936a5a Duration: 11.06 ms Billed Duration: 38 ms Memory Size: 128 MB Max Memory Used: 14 MB Init Duration: 26.63 ms     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[calavera]

Thanks!