Lambda Snapstart database connection

[RickVerkuijlen]

Describe the bug

Hi all,
Recently in one of our projects, we introduced Lambda Snapstart with the help of the Quarkus Snapstart optimisation. The problem is that, with using Hibernate ORM together with Panache, I couldn't find a way to re-establish the database connection for my entities. Snapstart stores the state of my Lambda, with the database connection. This connection goes stale after some time. When using the state created by Snapstart, the connection is used again resulting in connection problems. After a few seconds, when the connection is re-established, I can do another request and everything works as expected.

Without Snapstart, it takes around 10 seconds to start up the Lambda and finish the request. With Snapstart, this is reduced to 2 seconds, but the first connection with the database cannot be made. This wasn't a problem without Snapstart.

Any tips?

Expected behavior

Executing calls from a Snapstart state to a database for the first time.

Actual behavior

A stale database connection which fails the first time, but succeeds after a few seconds of being warm.

How to Reproduce?

1. Enable Snapstart on a Lambda that makes a call do a database
2. Use the Snapstart state on next request
3. First request should fail and after a few seconds, requests are succeeding

Output of uname -a or ver

No response

Output of java -version

openjdk version "17.0.5" 2022-10-18

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.16.3.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)

Additional information

No response

[quarkus-bot]

/cc @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda)

[codecr]

Hi!

I'm also having the same problem. Any tips will be appreciated.

[patriot1burke]

Quarkus integration with SnapStart does a full quarkus boot during the init phase. So if connection pooling is enabled and a DB interaction is done during startup, then the connections will be stale. You'll also want quarkus.datasource.jdbc.min-size to be zero.

I think you can also register a callback to flush the connection pool on Snapstart restore:

https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html

Try this maybe:

import org.crac.Resource;
import org.crac.Core;

@ApplicationScoped
public class FlushPool implements Resource {

   @Inject
   AgroalDataSource dataSource;

   void onStart(@Observes StartupEvent ev) {    
        Core.getGlobalContext().register(this);           
    }

    @Override
    public void beforeCheckpoint(org.crac.Context<? extends Resource> context)
        throws Exception {
      // noop
    }
    @Override
    public void afterRestore(org.crac.Context<? extends Resource> context)
        throws Exception {
           dataSource.flush()
    }
}

[RickVerkuijlen]

Yes, this is exactly what solves my issue! Thank you for the support!!

[codecr]

Thank you! I also confirm the solution.

[patriot1burke]

Reopening issue, because IMO, Quarkus should be automatically flushing the pool.

[RickVerkuijlen]

@patriot1burke Is there any updates on the support from Quarkus?