Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda): add support for ephemeral storage #19552

Merged
merged 31 commits into from
Apr 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
3a70f66
feat(lambda): add support for ephemeral storage
robertd Mar 25, 2022
5a5ea1b
add missing semicolon
robertd Mar 25, 2022
986f671
throw on out of bound ephemeral storage size
robertd Mar 25, 2022
2b8e85a
update readme
robertd Mar 25, 2022
69a00ee
add tests
robertd Mar 25, 2022
447ad15
minor change
robertd Mar 25, 2022
ce255fd
Update packages/@aws-cdk/aws-lambda/lib/function.ts
robertd Mar 25, 2022
3f58ab3
Update packages/@aws-cdk/aws-lambda/lib/function.ts
robertd Mar 25, 2022
ef22c8f
Update packages/@aws-cdk/aws-lambda/test/function.test.ts
robertd Mar 25, 2022
5227a3a
Update packages/@aws-cdk/aws-lambda/README.md
robertd Mar 25, 2022
39ed82d
Update packages/@aws-cdk/aws-lambda/lib/function.ts
robertd Mar 25, 2022
f14691b
fix few minor suggestions
robertd Mar 25, 2022
e25f2d8
minor
robertd Mar 25, 2022
617a4fe
lint
robertd Mar 25, 2022
eec7cf2
Update packages/@aws-cdk/aws-lambda/README.md
robertd Mar 30, 2022
8340dbf
add tests using Size
robertd Mar 30, 2022
5be64bc
Merge branch 'lambda-ephemeral-storage' of github.com:robertd/aws-cdk…
robertd Mar 30, 2022
ed15836
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 30, 2022
f2cf8af
fix readme
robertd Mar 30, 2022
cd9bc2d
update readme
robertd Mar 30, 2022
9d51de0
fix rosetta
robertd Mar 30, 2022
b4172fb
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 31, 2022
3cc1850
Update packages/@aws-cdk/aws-lambda/lib/function.ts
robertd Mar 31, 2022
be67334
Update packages/@aws-cdk/aws-lambda/lib/function.ts
robertd Mar 31, 2022
ced6336
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 31, 2022
ffefc69
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 31, 2022
3168256
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 31, 2022
bdc6814
Merge branch 'master' into lambda-ephemeral-storage
robertd Mar 31, 2022
3f1fd81
add test for token value
robertd Apr 1, 2022
f0c5dd8
Merge branch 'master' into lambda-ephemeral-storage
robertd Apr 1, 2022
0a4bde8
Merge branch 'master' into lambda-ephemeral-storage
mergify[bot] Apr 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions packages/@aws-cdk/aws-lambda/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -710,6 +710,24 @@ const fn = new lambda.Function(this, 'MyLambda', {
});
```

## Ephemeral Storage

You can configure ephemeral storage on a function to control the amount of storage it gets for reading
or writing data, allowing you to use AWS Lambda for ETL jobs, ML inference, or other data-intensive workloads.
The ephemeral storage will be accessible in the functions' `/tmp` directory.

```ts
import { Size } from '@aws-cdk/core';

const fn = new lambda.Function(this, 'MyFunction', {
runtime: lambda.Runtime.NODEJS_14_X,
handler: 'index.handler',
code: lambda.Code.fromAsset(path.join(__dirname, 'lambda-handler')),
ephemeralStorageSize: Size.mebibytes(1024),
});
```

robertd marked this conversation as resolved.
Show resolved Hide resolved
Read more about using this feature in [this AWS blog post](https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/).

## Singleton Function

Expand Down
17 changes: 16 additions & 1 deletion packages/@aws-cdk/aws-lambda/lib/function.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import * as kms from '@aws-cdk/aws-kms';
import * as logs from '@aws-cdk/aws-logs';
import * as sns from '@aws-cdk/aws-sns';
import * as sqs from '@aws-cdk/aws-sqs';
import { Annotations, ArnFormat, CfnResource, Duration, Fn, Lazy, Names, Stack, Token } from '@aws-cdk/core';
import { Annotations, ArnFormat, CfnResource, Duration, Fn, Lazy, Names, Size, Stack, Token } from '@aws-cdk/core';
import { Construct } from 'constructs';
import { Architecture } from './architecture';
import { Code, CodeConfig } from './code';
Expand Down Expand Up @@ -95,6 +95,13 @@ export interface FunctionOptions extends EventInvokeConfigOptions {
*/
readonly memorySize?: number;

/**
* The size of the function’s /tmp directory in MB.
*
* @default 512 MiB
*/
readonly ephemeralStorageSize?: Size;

/**
* Initial policy statements to add to the created Lambda Role.
*
Expand Down Expand Up @@ -747,6 +754,11 @@ export class Function extends FunctionBase {
}
this._architecture = props.architecture ?? (props.architectures && props.architectures[0]);

if (props.ephemeralStorageSize && !props.ephemeralStorageSize.isUnresolved()
&& (props.ephemeralStorageSize.toMebibytes() < 512 || props.ephemeralStorageSize.toMebibytes() > 10240)) {
throw new Error(`Ephemeral storage size must be between 512 and 10240 MB, received ${props.ephemeralStorageSize}.`);
}

const resource: CfnFunction = new CfnFunction(this, 'Resource', {
functionName: this.physicalName,
description: props.description,
Expand All @@ -767,6 +779,9 @@ export class Function extends FunctionBase {
// Token, actually *modifies* the 'environment' map.
environment: Lazy.uncachedAny({ produce: () => this.renderEnvironment() }),
memorySize: props.memorySize,
ephemeralStorage: props.ephemeralStorageSize ? {
size: props.ephemeralStorageSize.toMebibytes(),
} : undefined,
vpcConfig: this.configureVpc(props),
deadLetterConfig: this.buildDeadLetterConfig(dlqTopicOrQueue),
tracingConfig: this.buildTracingConfig(props),
Expand Down
46 changes: 46 additions & 0 deletions packages/@aws-cdk/aws-lambda/test/function.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import * as cdk from '@aws-cdk/core';
import * as constructs from 'constructs';
import * as _ from 'lodash';
import * as lambda from '../lib';
import { Lazy, Size } from '@aws-cdk/core';

describe('function', () => {
test('default function', () => {
Expand Down Expand Up @@ -2584,6 +2585,7 @@ describe('function', () => {
architectures: [lambda.Architecture.X86_64, lambda.Architecture.ARM_64],
})).toThrow(/one architecture must be specified/);
});

test('Architecture is properly readable from the function', () => {
const stack = new cdk.Stack();
const fn = new lambda.Function(stack, 'MyFunction', {
Expand Down Expand Up @@ -2635,6 +2637,50 @@ describe('function', () => {
});
});

test('throws if ephemeral storage size is out of bound', () => {
robertd marked this conversation as resolved.
Show resolved Hide resolved
const stack = new cdk.Stack();
expect(() => new lambda.Function(stack, 'MyLambda', {
code: new lambda.InlineCode('foo'),
handler: 'bar',
robertd marked this conversation as resolved.
Show resolved Hide resolved
runtime: lambda.Runtime.NODEJS_14_X,
ephemeralStorageSize: Size.mebibytes(511),
})).toThrow(/Ephemeral storage size must be between 512 and 10240 MB/);
});

test('set ephemeral storage to desired size', () => {
const stack = new cdk.Stack();
new lambda.Function(stack, 'MyLambda', {
code: new lambda.InlineCode('foo'),
handler: 'bar',
runtime: lambda.Runtime.NODEJS_14_X,
ephemeralStorageSize: Size.mebibytes(1024),
});

Template.fromStack(stack).hasResource('AWS::Lambda::Function', {
Properties:
{
Code: { ZipFile: 'foo' },
Handler: 'bar',
Runtime: 'nodejs14.x',
EphemeralStorage: {
Size: 1024,
},
},
});
});

test('ephemeral storage allows unresolved tokens', () => {
const stack = new cdk.Stack();
expect(() => {
new lambda.Function(stack, 'MyLambda', {
code: new lambda.InlineCode('foo'),
handler: 'bar',
runtime: lambda.Runtime.NODEJS_14_X,
ephemeralStorageSize: Size.mebibytes(Lazy.number({ produce: () => 1024 })),
});
}).not.toThrow();
});

function newTestLambda(scope: constructs.Construct) {
return new lambda.Function(scope, 'MyLambda', {
code: new lambda.InlineCode('foo'),
Expand Down