AWS Lambda S3Event deserialization fails

[goranopacic]

Describe the bug
AWS Lambda triggered by S3Event fails to start due to deserialization problem.
S3Event class is a part of AWS Lambda Java Events v3 library.

Expected behavior
Compiling exactly the same code with non-quarkus compiled AWS Lambda results in properly deserialized S3Event object.

Actual behavior

Logged Error in AWS CloudWatch log:

Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)

at [Source: (ByteArrayInputStream); line: 1, column: 14] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0]): com.fasterxml.jackson.databind.exc.InvalidDefinitionException
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (ByteArrayInputStream); line: 1, column: 14] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1615)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:400)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1077)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1332)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:331)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:290)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:293)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:156)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2079)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1453)
	at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder.handle(AmazonLambdaRecorder.java:65)
	at io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler.handleRequest(QuarkusStreamHandler.java:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator) at [Source: (ByteArrayInputStream); line: 1, column: 14] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0]): com.fasterxml.jackson.databind.exc.InvalidDefinitionException com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (ByteArrayInputStream); line: 1, column: 14] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0]) 
at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67) 
at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1615)
at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:400) 
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1077) 
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1332) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:331) 
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164) 
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:290) 
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249) 
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26) 
at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138) 
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:293) 
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:156) 
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2079) 
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1453) 
at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder.handle(AmazonLambdaRecorder.java:65) 
at io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler.handleRequest(QuarkusStreamHandler.java:58) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
at java.base/java.lang.reflect.Method.invoke(Unknown Source)

To Reproduce

1. create project using quarkus lambda Maven Archetype

mvn archetype:generate
-DarchetypeGroupId=io.quarkus
-DarchetypeArtifactId=quarkus-amazon-lambda-archetype
-DarchetypeVersion=1.10.0.Final

2. replace the TestLambda.java with the following code:

package com.replaceme;

import javax.inject.Inject;
import javax.inject.Named;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification.S3EventNotificationRecord;

@Named("test")
public class TestLambda implements RequestHandler<S3Event, String> {

    @Inject
    ProcessingService service;

    @Override
    public String handleRequest(S3Event s3event, Context context) {
        try {
            S3EventNotificationRecord record = s3event.getRecords().get(0);
            String srcBucket = record.getS3().getBucket().getName();
            String srcKey = record.getS3().getObject().getUrlDecodedKey();

            System.out.println(srcBucket + " " + srcKey);

            return "OK";

        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

3. deploy lambda, set trigger event to be S3, upload test file to related bucket and check cloudwatch log for errors

Configuration
no changes to default pom.xml generated by aws lambda quarkus Maven Archetype

application.properties:
quarkus.lambda.handler=test

Environment (please complete the following information):

• Output of uname -a or ver:
  Darwin xxxx.local 20.1.0 Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64 x86_64

• Java version
  java -version
  openjdk version "11.0.5" 2019-10-15
  OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
  Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.17.0, JRE 11 Mac OS X amd64-64-Bit 20191031_334 (JIT enabled, AOT enabled)
  OpenJ9 - 77c1cf708
  OMR - 20db4fbc
  JCL - 775c5eb03a based on jdk-11.0.5+10)

• Quarkus version or git rev:
  Quarkus 1.10.0.Final

• Build tool (ie. output of mvnw --version or gradlew --version):
  Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
  Maven home: /opt/apache-maven-3.6.3
  Java version: 11.0.5, vendor: Eclipse OpenJ9, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-11-openj9.jdk/Contents/Home
  Default locale: en_US, platform encoding: UTF-8
  OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

[ghost]

/cc @matejvasek, @patriot1burke

[geoand]

Is there in JVM mode or native mode?

[goranopacic]

jvm mode

[matejvasek]

@goranopacic Just guessing here: shouldn't be there S3EventNotification? https://stackoverflow.com/questions/36730266/aws-lambda-s3event-deserialization #7670

[goranopacic]

@matejvasek you can see this example: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-java
With no-quarkus lambdas I use S3Event, not, S3EventNotification and it works fine.

[goranopacic]

@matejvasek please be aware that this is SDK v2 and Lambda Java Events v3 library is a separate project that is included in pom.xml when you add quarkus lambda extension

[patriot1burke]

So, Quarkus integration with Lambda in JVM mode is done by using a RequestStreamHandler wrapper which uses the Jackson JSON parser to deserialize the stream to the target object of the Lambda. We do it this way so that Quarkus can control the creation of your Lambda class.

The problem is that S3Event is not a class that is deserializable by Jackson. We are going to have to have specific integration for this as the Quarkus Lambda integration assumed only user classes for input and output.

[patriot1burke]

Oh...and there would be a similar issue with native lambdas for S3Events.

[patriot1burke]

I'll work on a patch to support this.

[patriot1burke]

@goranopacic See linked PR.

[goranopacic]

superb! it works both ways - jvm and native.

Here is my log from simple lambda showing uploaded file name (Native). Init and second execution with 1ms billing :).

REPORT RequestId: c420634a-1e21-49b3-bb8e-9098667f451c Duration: 106.86 ms Billed Duration: 398 ms Memory Size: 256 MB Max Memory Used: 63 MB Init Duration: 290.92 ms
REPORT RequestId: 7ef6691f-e321-490b-ad0c-03f14c5832e0 Duration: 1.04 ms Billed Duration: 2 ms Memory Size: 256 MB Max Memory Used: 63 MB

I will post an Issue at AWS events project in order to make it more Jackson compatible so you don't have to do all these things. It is amazing that you managed to make it ready in such a short time. Thanks a lot!