Added event class MskFirehoseEvent.java for Firehose Lambda transformation when MSK is the source #490
Conversation
|
Thanks for the PR. Can you review some of the recent PRs / commits and model your class like those. Favour Lombok annotations over getters / setters / toString / equals. |
There was a problem hiding this comment.
In addition to what Mark's said, please add a test for this event to https://github.com/aws/aws-lambda-java-libs/tree/main/aws-lambda-java-tests
Changed packaging and used Lombok.
| /** | ||
| * Created by vermshas on 6/28/24. | ||
| * Event format is below: | ||
| * { |
| * Event format is below: | ||
| * { | ||
| * "invocationId": "", | ||
| * "sourceMSKArn": "", |
There was a problem hiding this comment.
Do we need to model the sourceMSKArn
There was a problem hiding this comment.
added sourceMSKArn.
| @Getter | ||
| @Setter | ||
| @ToString | ||
| @EqualsAndHashCode |
There was a problem hiding this comment.
use the following lombok annotations
@Data
@Builder(setterPrefix = "with")
@NoArgsConstructor
@AllArgsConstructor
Added MSKFirehoseEvent
This is a response class for the Lambda transformation function.
Adding test class.
Creating test event.
added MSKFirehoseResponse
|
Hi, I have added the test for the event, response class and updated Readme. |
Changed naming conventions.
|
|
||
| /** | ||
| * Created by vermshas on 6/28/24. | ||
| * { |
There was a problem hiding this comment.
Please remove this example and link the docs
There was a problem hiding this comment.
Removed the example, I am not able to find any document stating the specific event.
...a-events/src/main/java/com/amazonaws/services/lambda/runtime/events/MSKFirehoseResponse.java
Outdated
Show resolved
Hide resolved
...a-events/src/main/java/com/amazonaws/services/lambda/runtime/events/MSKFirehoseResponse.java
Show resolved
Hide resolved
Changed packaging for event handler.
...a-events/src/main/java/com/amazonaws/services/lambda/runtime/events/MSKFirehoseResponse.java
Show resolved
Hide resolved
| public Result result; | ||
| public ByteBuffer kafkaRecordValue; | ||
|
|
||
| public Record(String recordId, Result result, ByteBuffer kafkaRecordValue) { |
There was a problem hiding this comment.
Do you need this constructor?
There was a problem hiding this comment.
Removed the constructor by using Lombok.
Removed the event structure, we do not have documents stating the event structure for this case.
Expanded the Lombok import, updated java doc for the class and removed the record constructor.
Added new test.
...da-java-tests/src/test/java/com/amazonaws/services/lambda/runtime/tests/EventLoaderTest.java
Show resolved
Hide resolved
...da-java-tests/src/test/java/com/amazonaws/services/lambda/runtime/tests/EventLoaderTest.java
Outdated
Show resolved
Hide resolved
escaped quotes in JSON
Added values to ARN fields.
added MSKMetadata check and Approximate arrivaltimestamp.
|
use the following license text in all source files: |
| import lombok.NoArgsConstructor; | ||
|
|
||
| /** | ||
| * Created by vermshas on 6/28/24. |
There was a problem hiding this comment.
removed saw it added to some classes and thought its required. Added copyright/license text to MSKFirehoseEvent.java and MSKFirehoseResponse.java. Do we need to add license text to sample as well ?
There was a problem hiding this comment.
yes, to all source code, please
There was a problem hiding this comment.
Added license text to samples as well.
added copyright text.
Added Copyright text.
added license text.
Added license text.
…ation when MSK is the source (aws#490) * Create MskFirehoseEvent.java * Create MSKFirehoseResponse.java
Issue #, if available:
Not available :
Description of changes:
In case of the data flow like :
MSK -----> Firehose ---> invokes Lambda transformation with event -------> Lambda returns the transformed record back to Firehose ---------> Firehose Deliver's the data to S3 or any other destination.
The issue lies when Firehose tries to invoke the Lambda, with event [1], Lambda java event libraries does not have any event class to support this event.
[1] event :
{
"invocationId": "",
"sourceMSKArn": "",
"deliveryStreamArn": "",
"region": "us-east-1",
"records": [
{
"recordId": "00000000000000000000000000000000000000000000000000000000000000",
"approximateArrivalTimestamp": 1716369573887,
"mskRecordMetadata": {
"offset": "0",
"partitionId": "1",
"approximateArrivalTimestamp": 1716369573887
},
"kafkaRecordValue": ""
}
]
}
To handle these events, adding the class MskFirehoseEvent.
Target (OCI, Managed Runtime, both): both
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.