`DeserializeError` when using `lambda_http` with `sam local start-api` #797

lmammino · 2024-01-27T16:12:11Z

Hello! 👋🏽

As usual, thanks for all the excellent work that you are doing to enable devs to ship Lambdas written in Rust.

I recently bumped into the following issue when running sam local start-api and then I made a call to the local gateway:

START RequestId: 55d7f053-2fb8-4381-9c15-567efca2fdd0 Version: $LATEST
ERROR Lambda runtime invoke{requestId="2ac8ff62-1a89-4c29-9801-ea7de20ec9db"}: DeserializeError { inner: Error { path: Path { segments: [] }, original: Error("this function expects a JSON payload from Amazon API Gateway, Amazon Elastic Load Balancer, or AWS Lambda Function URLs, but the data doesn't match any of those services' events", line: 0, column: 0) } }
END RequestId: 2ac8ff62-1a89-4c29-9801-ea7de20ec9db
REPORT RequestId: 2ac8ff62-1a89-4c29-9801-ea7de20ec9db	Init Duration: 0.52 ms	Duration: 43.70 ms	Billed Duration: 44 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

These are the versions I am using:

rustc 1.75.0 (82e1608df 2023-12-21)
cargo 1.75.0 (1d8b05cdd 2023-11-20)
SAM CLI, version 1.105.0
Docker version 24.0.7, build afdd53b
macOs silicon

Step-by-step reproduction guide

Follow below the steps to reproduce the issue

1. Start a new project with `cargo lambda`

cargo lambda new test-api
cd test-api

2. Update handler

Mostly to respond with a JSON body (shouldn't be strictly required)

// src/main.rs
use lambda_http::{run, service_fn, Body, Error, Request, RequestExt, Response};

async fn function_handler(event: Request) -> Result<Response<Body>, Error> {
    // Extract some useful information from the request
    let who = event
        .query_string_parameters_ref()
        .and_then(|params| params.first("name"))
        .unwrap_or("world");
    let message = format!("{{\"message\": \"Hello {who}, this is an AWS Lambda HTTP request\"}}");

    let resp = Response::builder()
        .status(200)
        .header("content-type", "application/json")
        .body(message.into())
        .map_err(Box::new)?;
    Ok(resp)
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        // disable printing the name of the module in every log line.
        .with_target(false)
        // disabling time is handy because CloudWatch will add the ingestion time.
        .without_time()
        .init();

    run(service_fn(function_handler)).await
}

3. Add a SAM template

# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  ApiHandler:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: rust-cargolambda
    Properties:
      CodeUri: .
      Handler: bootstrap
      Runtime: provided.al2023
      Architectures:
        - arm64
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Method: get
            Path: /test

4. Start sam local

sam local start-api

Now you can open your browser, for instance pointing it to http://127.0.0.1:3000/test?name=luciano, and you should see the error mentioned above in the sam local start-api process

The text was updated successfully, but these errors were encountered:

calavera · 2024-01-27T19:31:37Z

As the message says, SAM is sending something that doesn't match the structs that we've defined for those services. In this case path: Path { segments: [] } indicates something related to a segments field.

If you set the environment RUST_LOG=trace, you'll be able to see the JSON payload that SAM sends.

lmammino · 2024-01-28T11:18:39Z

Thanks for the quick reply, @calavera!

Somehow I was not able to get the full event by enabling RUST_LOG: trace (both in my shell where I start the SAM local server and as a function Env var, specified in the template.yaml)...

Then I realised that cargo lambda generates the following code in the main:

tracing_subscriber::fmt().with_max_lever(tracing::Level::INFO) // ...

Changing this to:

    tracing_subscriber::fmt()
        .with_max_level(
            env::var("RUST_LOG")
                .unwrap_or("info".to_string())
                .parse()
                .unwrap_or(tracing::Level::INFO),
        )

Did the trick, but maybe this could be a separate issue/PR to cargo lambda :) (happy to hear your thoughts on that).

Anyway, this is the JSON produced by SAM:

{
  "body": null,
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Cache-Control": "max-age=0",
    "Connection": "keep-alive",
    "Host": "127.0.0.1:3000",
    "Sec-Ch-Ua": "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"macOS\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "X-Forwarded-Port": "3000",
    "X-Forwarded-Proto": "http"
  },
  "httpMethod": "GET",
  "isBase64Encoded": false,
  "multiValueHeaders": {
    "Accept": [
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
    ],
    "Accept-Encoding": [
      "gzip, deflate, br"
    ],
    "Accept-Language": [
      "en-GB,en-US;q=0.9,en;q=0.8"
    ],
    "Cache-Control": [
      "max-age=0"
    ],
    "Connection": [
      "keep-alive"
    ],
    "Host": [
      "127.0.0.1:3000"
    ],
    "Sec-Ch-Ua": [
      "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\""
    ],
    "Sec-Ch-Ua-Mobile": [
      "?0"
    ],
    "Sec-Ch-Ua-Platform": [
      "\"macOS\""
    ],
    "Sec-Fetch-Dest": [
      "document"
    ],
    "Sec-Fetch-Mode": [
      "navigate"
    ],
    "Sec-Fetch-Site": [
      "none"
    ],
    "Sec-Fetch-User": [
      "?1"
    ],
    "Upgrade-Insecure-Requests": [
      "1"
    ],
    "User-Agent": [
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    ],
    "X-Forwarded-Port": [
      "3000"
    ],
    "X-Forwarded-Proto": [
      "http"
    ]
  },
  "multiValueQueryStringParameters": null,
  "path": "/dump",
  "pathParameters": null,
  "queryStringParameters": null,
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "1234567890",
    "domainName": "127.0.0.1:3000",
    "extendedRequestId": null,
    "httpMethod": "GET",
    "identity": {
      "accountId": null,
      "apiKey": null,
      "caller": null,
      "cognitoAuthenticationProvider": null,
      "cognitoAuthenticationType": null,
      "cognitoIdentityPoolId": null,
      "sourceIp": "127.0.0.1",
      "user": null,
      "userAgent": "Custom User Agent String",
      "userArn": null
    },
    "path": "/dump",
    "protocol": "HTTP/1.1",
    "requestId": "ea86b9eb-c688-4bbb-8309-a1671442bea9",
    "requestTime": "28/Jan/2024:11:05:46 +0000",
    "requestTimeEpoch": 1706439946,
    "resourceId": "123456",
    "resourcePath": "/dump",
    "stage": "Prod"
  },
  "resource": "/dump",
  "stageVariables": null,
  "version": "1.0"
}

At this point, I am still unsure which field is causing the deserialization issue...

lmammino · 2024-01-28T12:20:57Z

So I did more investigation to try to wrap my head around the issue... As a result, I opened #798, which might be useful and help with having more coverage in the future.

Here are my current findings:

This issue is only happening with REST ApiGw (changing the Event type from Api to HttpApi in my template.yaml example, makes it possible to make local calls through sam local start-api
In the additional tests that I added in Adds more tests to investigate #797 #798, I was able to deserialize both events coming from SAM (Rest and HTTP), so I am not really sure I am tracking the correct code path...

Any pointer on how I might be able to investigate this further?

* Adds more tests to investigate #797 * Update deserializer.rs

bnusunny · 2024-02-06T12:29:28Z

@lmammino I could reproduce this issue. And when I updated lambda-http crate to 0.9.2, the issue went away. Could you please verify if it works for you with the latest lambda-http crate?

lmammino · 2024-02-06T19:38:29Z

I just realised yesterday evening that i was running on

lambda_http = "0.8.3"
lambda_runtime = "0.8.3"

By updating to 0.9.1 that seems to have solved the issue for me. I haven't really had the time to investigate when that issue was actually fixed, but I belive this one can be closed now!

lmammino · 2024-02-06T19:42:46Z

Also proposed this change so whoever is using cargo lambda to scaffold a new project is more unlikely to bump on this one: https://github.com/cargo-lambda/default-template/pull/13/files

calavera · 2024-02-06T22:22:41Z

Thanks for the PR! I'm closing this issue as resolved 🎊

github-actions · 2024-02-06T22:22:59Z

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

lmammino added a commit to lmammino/aws-lambda-rust-runtime that referenced this issue Jan 28, 2024

Adds more tests to investigate awslabs#797

2b771eb

lmammino mentioned this issue Jan 28, 2024

Adds more tests to investigate #797 #798

Merged

2 tasks

lmammino mentioned this issue Jan 28, 2024

feat: adds support for RUST_LOG env var when configuring tracing_subscriber cargo-lambda/default-template#10

Merged

bnusunny pushed a commit that referenced this issue Jan 29, 2024

Adds more tests to investigate #797 (#798)

a837f3c

* Adds more tests to investigate #797 * Update deserializer.rs

calavera closed this as completed Feb 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`DeserializeError` when using `lambda_http` with `sam local start-api` #797

`DeserializeError` when using `lambda_http` with `sam local start-api` #797

lmammino commented Jan 27, 2024 •

edited

Loading

calavera commented Jan 27, 2024

lmammino commented Jan 28, 2024

lmammino commented Jan 28, 2024 •

edited

Loading

bnusunny commented Feb 6, 2024

lmammino commented Feb 6, 2024

lmammino commented Feb 6, 2024

calavera commented Feb 6, 2024

github-actions bot commented Feb 6, 2024

DeserializeError when using lambda_http with sam local start-api #797

DeserializeError when using lambda_http with sam local start-api #797

Comments

lmammino commented Jan 27, 2024 • edited Loading

Step-by-step reproduction guide

1. Start a new project with cargo lambda

2. Update handler

3. Add a SAM template

4. Start sam local

calavera commented Jan 27, 2024

lmammino commented Jan 28, 2024

lmammino commented Jan 28, 2024 • edited Loading

bnusunny commented Feb 6, 2024

lmammino commented Feb 6, 2024

lmammino commented Feb 6, 2024

calavera commented Feb 6, 2024

github-actions bot commented Feb 6, 2024

`DeserializeError` when using `lambda_http` with `sam local start-api` #797

`DeserializeError` when using `lambda_http` with `sam local start-api` #797

lmammino commented Jan 27, 2024 •

edited

Loading

1. Start a new project with `cargo lambda`

lmammino commented Jan 28, 2024 •

edited

Loading