-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: idempotency utility #447
Comments
Hi @walmsles thank you for the request. Feature parity with the Python version is definitely something that we have on our radars. Middy-compliant middlewares are only one of the usages that we want to cover at this stage (with the others being Class method decorators & manual instrumentation) so the feature deserves a closer look. |
Hi everyone, here is a design proposal. Would appreciate your comments on this (especially around the Utility interface as it's different from Python and Java which have no constraint on decorator usage.) Design proposal (Request for comments)1. SummaryThe goal of this document is to propose the scope and design of Idempotency utility for Powertools for TypeScript. The utility has been implemented in the Python and Java version. We will use the current Python implementation (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/) as a baseline, and describe only the differences we will make in TypeScript. Anything not discussed here will be the same as in Python version. This RFC assumes that you are familiar with Python’s implementation. If you aren’t, please check the documentation of Python Idempotency utility (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/) first. 2. MotivationIdempotency is a common pattern used by many customers. It guarantees that any retry with the same “idempotency key” should not be executed again. This utility aims to provide “out-of-the-box” idempotency on the top of Lambda with minimum code from library user. With this utility, customers only need to provide add a decorator, Middy middleware, or a function wrapper on the top of business logic functions. Customers may customize the utility behavior by providing different config, or provide a custom PersistentLayer to use different persistent storage for storing idempotency key. 3. Utility interfaceThere are two usages of using Idempotency utility.
The second option is useful for batch or multi-record processing. Imagine that we receive 10 records in a single request. We want the idempotency at the record level, not at the handler level. We can loop through the records and call the decorated function. We need to specify which Unlike existing utilities, the "manual" options are complex and expose a lot of implementation. Thus, the first release will have less usage options than those utilities. 3.1.1. Idempotent handler (via Middy middleware)import {
makeHandlerIdempotent,
DynamoDBPersistenceLayer,
IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
import middy from '@middy/core';
const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});
const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
/* ...Function logic here... */
}
export const handler = middy(lambdaHandler)
.use(makeHandlerIdempotent({
config: idempotencyConfig,
persistenceLayer: ddbPersistenceLayer,
}); 3.1.2. Idempotent handler (via Decorator)import {
idempotentHandler,
DynamoDBPersistenceLayer,
IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});
class Lambda implements LambdaInterface {
// Decorate your handler class method
@idempotentHandler(config, ddbPersistenceLayer)
public async handler(_event: any, _context: any): Promise<void> {
/* ...Function logic here... */
}
}
export const handlerClass = new Lambda();
export const handler = handlerClass.handler.bind(handlerClass); 3.2.1 Idempotent function (via Decorator)Note that this feature is typically used with Batch Utility (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/#idempotent_function-decorator). Given that we don’t have Batch Utility yet, we will give an example with a simple import {
idempotentFunction,
DynamoDBPersistenceLayer,
IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
import middy from '@middy/core'
const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});
class Lambda implements LambdaInterface {
public async handler(_event: any, _context: any): Promise<void> {
const records = /*..Extract SQS/DDB stream record, etc..*/
const results = []
for(record of records) {
results.push(this.process(record));
}
/* ...Format and return result... */
}
@idempotentFunction({
dataKeywordArgument = 'record', // Match with param name of decorated function below
config,
ddbPersistenceLayer
})
private process(record: any) {
/* ...Function logic here... */
return result;
}
}
export const handlerClass = new Lambda();
export const handler = handlerClass.handler.bind(handlerClass); 3.2.2. Idempotent function (via wrapper)Majority of TypeScript/JS Lambda code are not using a class. Middy middleware is not an option as we are dealing with a non-handler function. To support this major use case, we provide a wrapper function import {
makeFunctionIdempotent,
DynamoDBPersistenceLayer,
IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});
/**
* Function to process a single record
*/
function processRecord(record: any) {
/* ...Function logic here... */
return result;
}
/**
* Higher-order function to process a single record
*/
const processIdempotently = makeFunctionIdempotent(
processRecord,
{
dataKeywordArgument: 'record',
/*... other options...*/
}
);
const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
const records = /*..Extract SQS/DDB stream record, etc..*/
const results = [];
for (const record of records) {
results.push(processIdempotently(record));
}
/* ...Format and return result... */
}
//... 4. Requrements4.1 Feature breakdownThe first release will contain only high/medium priority features. All "Out of scope" features won’t be implement in the future releases unless there is a clear signal from customers.
4.2 Edge casesIdempotent function contains many edge cases and limitation. We will use the same behavior as Python and Java version. This section clarifies the behaviours of non-happy flows for reference during implementation.
5. Opened discussions and decisionsThis is a list of important decision points from maintainer discussion. Subject to changes from comments from community.
6. Implementation deviation from Python80-90% of the implementation will be based on the Python version. However, we will deviate where it’s appropriate as we don’t have a constraint of making breaking changes yet.
7. References
|
That sounds very nice and is indeed much needed 👍 We're also currently simply using a middy middleware to check if a request was already executed and add a flag to the
Would definitely help a lot. Thanks! |
This looks great! We are very excited that this utility has been prioritized! After reading the design proposal, we do have a question and a couple of comments. First, the question: We noticed that FR3 is "Return the same result when called with the same payload", which made us wonder about the scope of FR1. Is it just "don't re-process given the same payload"? The comments:
|
FR1 will stop Lambda process the same workload twice. FR3 is putting the result in the persistent layer, and return the same result when the new request comes in. This is just to break implementation into smaller parts. In FR1 implementation, it may simply return a hardcoded response saying that the given idempotency key has been processed. Then, we implement FR3 to save the result from 1st request, and do return the stored result for subsequent requests.
Totally agree. I actually didn't think about implementation order in my mind. Feel free to work in the order you prefer. But please focus on only high/medium priority first. May be also good to share with us your plan here.
Please let me discuss with other maintainers. I will get back to you on this. |
@jeffrey-baker-vg This issue is all yours and the team :) I don't see any comments that require a major change. I think we can start implementation. FR2 is a good candidate to start. Suggestions (optional) : Given that this feature is quite big and we have many contributors, should we could start with classes and their interfaces? It doesn't have to be detailed, just class names and public methods are sufficient. The aim is to align every contributor on boundary and responsibility of each classes. |
@ijemmy @jeffrey-baker-vg if that helps on 6.3, in Python, we chose not to make lambda context required for two reasons (idempotent_function):
I agree with @ijemmy that it is error prone for Lambda customers decorating sync functions but we couldn't find a better trade-off yet --- please let us know if you do in the future. |
I wanted to see what other's thoughts were here for using the /**
* Higher-order function to process a single record
*/
const processIdempotently = makeFunctionIdempotent(
processRecord,
{
dataKeywordArgument: 'record',
/*... other options...*/
}
);
const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
const records = /*..Extract SQS/DDB stream record, etc..*/
const results = [];
for (const record of records) {
results.push(processIdempotently(record));
}
/* ...Format and return result... */
} This seemed to be an adaptation of what was provided with the python version since it could use kwargs and therefore find specific arguments based on names. In javascript, keyword arguments don't really exist. There are mechanisms that allow for psuedo keyword arguments and I wanted to suggest we go the route below so that we can keep the concept of defining the payload we want to use for idempotency while fitting into the mold of javascript/typescript. /**
* Higher-order function to process a single record
*/
const processIdempotently = makeFunctionIdempotent(
processRecord,
{
dataKeywordArgument: 'field',
/*... other options...*/
}
);
const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
const records = {field: 'value'}
const results = [];
for (const record of records) {
results.push(processIdempotently(record));
}
/* ...Format and return result... */
} The difference is subtle but the key is that we have to enforce/ensure that the parameter into the method is in fact an object and the dataKeywordArgument is a field in that object; this way we can ensure the ability to look up specific keys within the object. So And then the user can define the Or we can assume that all arguments are the key to idempotency and not allow for this granular of a specification. Edit (by @ijemmy): Add code syntax highlight for TypeScript for ease of reading. No content changes. |
@dreamorosi @saragerion Let's discuss about this on tomorrow's maintainer sync. For reference:
|
Just talked with other maintainers (@dreamorosi , @flochaz ). Named parameter is definitely not what we desire. (We also got confused when looking at Python style). The trade-off is that we will enforce the signature of @flochaz proposed another option below. Let's discuss in this issue so we all can voice our opinions and find the most appropriate trade-off. It was difficult to wrap the discussion without examples. So let me sum up the 2 options we were discussing. Firstly, we assume that a client wants to make this method idempotent: interface Record {
[key: string]: any; // Note that we ONLY accept an object here. Client cannot passes a string or multiple arguments
}
function processRecord(record: Record) {
// do something
} Here are the two options: Option 1: Based @KevenFuentes9 's proposal, I expand it to cover the JMESPath. /**
* Higher-order function to process a single record
*/
const processIdempotently = makeFunctionIdempotent(
processRecord,
{
// There must a this field in the object passed into `processIdempotently` function
dataKeywordArgument: 'fieldToExtractIdempotency',
// (Optional) Used when the `dataKeywordArgument` contains an object and we want to extract key from a subset of fields
eventKeyJMESPath: "[userDetail, productId]"
/*... other options...*/
}
);
const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
const records = [
{
id: '1',
fieldToExtractIdempotency: {
userDetails: 'foo1',
productId: 'bar1'
otherFields: 'fizz1'
}
},
{
id: '2',
fieldToExtractIdempotency: {
userDetails: 'foo2',
productId: 'bar2',
otherFields: 'fizz2',
}
},
]
const results = [];
for (const record of records) {
// Note: the function will throw an error at run time if the records does not contain `field`
const result = processIdempotently(record);
results.push(result);
}
/* ...Format and return result... */
} Option 2: Let client specify how to extract idempotency key /**
* Higher-order function to process a single record
*/
const processIdempotently = makeFunctionIdempotent(
processRecord,
{
// Note: Client can specify how to create idempotency key from the passed record
extractKeyFunction: (record) => {
const { userDetails, productId } = record.fieldToExtractIdempotency;
return hash(userDetails + '#' + productId)
}
}
);
// Note: the rest here is the same as the option above... @flochaz Could you confirm if I understand your option 2 correctly? |
I've discussed with @dreamorosi and @flochaz. Let's go with your proposal. So far, it's the most appropriate one we can find for JavaScript/TypeScript. And it's more compatible with the JMESPath option that we'll implement later. |
Very interested in this as well. If that is of any help, we implemented a custom very basic version of this a while ago (inspired by the python implementation) Here is a gist |
@bboure Thank you! Is the |
Closing this issue since the Idempotency utility was released as beta preview in v1.11.1. We look forward to hear what you think of it, if you have any comment, question, or bug report please don't hesitate to open a new issue, start a discussion, or join us on Discord! |
|
Description of the feature request
Problem statement
Idempotency is a core Cloud issue that needs a solution to enable stable, fault-tolerant systems that can be affected by repeated transactions. A true idempotency solution as a utility for Powertools would be really useful for this project based on the function/features of the existing AWS Lambda Powertools for python
Summary of the feature
Link to Python Lambda Powertools documentation as a good example to follow: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency
Benefits for you and the wider AWS community
Supply a good idempotent solution "Out of the Box" with Powertools for Typescript developers.
Describe alternatives you've considered
Looked at this: https://www.npmjs.com/package/middy-idempotent but relies on redis for storage and the Idempotent Key generation is not ideal and looks like it will be problematic.
Additional context
https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/
Related issues, RFCs
None.
The text was updated successfully, but these errors were encountered: