Quarkus 3 application on AWS Lambda- Part 5 Measuring Lambda cold and warm starts with GraalVM Native Image

Introduction

In the part 1 of our series about how to develop, run and optimize Quarkus web application on AWS Lambda, we demonstrated how to write a sample application which uses the Quarkus framework, AWS Lambda, Amazon API Gateway and Amazon DynamoDB. We also made the first Lambda performance (cold and warm start time) measurements and observed quite a big cold start time.

In the part 2 of the series, we introduced Lambda SnapStart and measured how its enabling reduces the Lambda cold start time by more than 50%.

In the part 3 of the series, we introduced how to apply Lambda SnapStart priming techniques by starting with DynamoDB request priming with the goal to even further improve the performance of our Lambda functions. We saw that by doing this kind of priming by writing some additional code we could significantly further reduce the Lambda cold start times compared to simply activating the SnapStart.

In the part 4 of the series, we introduced how to apply Lambda SnapStart priming techniques we called API Gateway event request priming with the goal to even further improve the performance of our Lambda functions compared to the DynamoDB request priming. We saw that by doing this kind of priming by writing a some amount of additional code we could further reduce the Lambda cold start times.

In this part of our series, we'll introduce how to adjust our sample application to one from which we can build the GraalVM Native Image and deploy it as a Lambda Custom Runtime. We'll then measure the Lambda performance with it and compare the results with other already introduced approaches.

Quarkus 3 on AWS Lambda with GraalVM Native Image (rebuilt) sample application

This article assumes prior knowledge of GraalVM and its native image capabilities. For a concise overview about them and how to get both installed, please refer to the following articles: Introduction to GraalVM, GraalVM Architecture and GraalVM Native Image or read my article Introduction to GraalVM and its native image capabilities.

Let's take a look at the rebuilt sample application and the differences to our previous sample application. As far as the source code of the application is concerned, the ReflectionConfig class has been added. In this class, we use @RegisterForReflection annotation to define the classes that are only loaded at runtime.

@RegisterForReflection(targets = {
    APIGatewayProxyRequestEvent.class,
    HashSet.class, 
    APIGatewayProxyRequestEvent.ProxyRequestContext.class, 
    APIGatewayProxyRequestEvent.RequestIdentity.class,
        DateTime.class,
        Product.class,
        Products.class,
})

Since GraalVM uses Native Image Ahead-of-Time compilation, we need to provide such classes in advance, otherwise ClassNotFound errors will be thrown at runtime. This includes custom entity classes like Product and Products, some AWS dependencies to APIGateway Proxy Event Request (from the artifact id aws-lambda-java-events from pom.xml), DateTime class to convert timestamp from JSON to Java object and some other classes. It sometimes takes several attempts and running the application first to find all such classes.

There are other ways to register classes for Reflection, which are described in this article Tips for writing native applications.

To prevent multiple errors when building GraalVM Native Image like

Caused by: com.oracle.svm.core.util.UserError$UserException: 
Class initialization of org.apache.commons.logging.impl.Log4JLogger failed. 
This error is reported at image build time because class 
org.apache.commons.logging.impl.Log4JLogger is registered for
linking at image build time by command line and command line. 
Use the option 
'--initialize-at-run-time=org.apache.commons.logging.impl.Log4JLogger'

when using the default Apache HTTP Client when creating DynamoDB Client in the DynamoProductDao class, I switched to using UrlConnectionHttpClient as follows:

  private final DynamoDbClient dynamoDbClient = DynamoDbClient.builder()
    .credentialsProvider(DefaultCredentialsProvider.create())
    .region(Region.EU_CENTRAL_1)
    .overrideConfiguration(ClientOverrideConfiguration.builder()
      .build())
    .httpClient(UrlConnectionHttpClient.builder().build())
    .build();

You can try to fix those errors and stick to the Apache HTTP Client by generating the correct configuration by running GraalVM Tracing Agent, like described in the article Configure Native Image with the Tracing Agent and Collect Metadata with the Tracing Agent but I decided against it. Generally speaking the usage of Log4 logging with GraalVM Native Image through some of the dependencies (like Apache HTTP Client) introduces a lot of pain which only be fixed starting from 2.25 version, see the article Building GraalVM native images. So, in case you stick to Apache HTTP Client, declare this (or higher) version of all Log4j related artifacts in the pom.xml.

In the pom.xml there are a few more additional declarations necessary. First we need to use Amazon DynamoDB Client Quarkus extension from Quarkiverse as follows:

 <dependency>
   <groupId>io.quarkiverse.amazonservices</groupId>
   <artifactId>quarkus-amazon-dynamodb</artifactId>
    <version>3.2.0</version>
 </dependency>

With out it, I ran into different errors when building GraalVM Native Image :

Fatal error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected an instance of Random/SplittableRandom class in the image heap. Instances created during image generation have cached seed values and don't behave as expected. If these objects should not be stored in the image heap, you can use 

    '--trace-object-instantiation=java.util.Random'

to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 

    '--initialize-at-run-time=<culprit>'

to prevent the instantiation of the object.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Object was reached by
  reading field software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy.random of constant 
    software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy@1121cb59: FullJitterBackoffStrategy(baseDelay=PT0.025S, maxBackoffTime=PT20S)
  scanning root software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy@1121cb59: FullJitterBackoffStrategy(baseDelay=PT0.025S, maxBackoffTime=PT20S) embedded in
    software.amazon.awssdk.services.dynamodb.DynamoDbRetryPolicy.retryPolicyFor(DynamoDbRetryPolicy.java:109)
  parsing method software.amazon.awssdk.services.dynamodb.DynamoDbRetryPolicy.retryPolicyFor(DynamoDbRetryPolicy.java:106) reachable via the parsing context
    at software.amazon.awssdk.services.dynamodb.DynamoDbRetryPolicy.resolveRetryStrategy(DynamoDbRetryPolicy.java:94)
    at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbBaseClientBuilder.finalizeServiceConfiguration(DefaultDynamoDbBaseClientBuilder.java:110)
    at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:171)
    at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:202)

Generally speaking, Quarkus 3 currently doesn't work well with GraalVM Native Image due to these described workarounds which I didn't have to do when working with Spring Boot 3 on Lambda.

This extension takes effect automatically, we do not have to change the source code of the application. We also need to activate the native profile as follows:

<profile>
   <id>native</id>
   <activation>
      <property>
          <name>native</name>
      </property>
   </activation>
   <properties>
      <quarkus.native.additional-build-args>
          --initialize-at-run-time=software.amazonaws.example.product.dao
       </quarkus.native.additional-build-args>
       <quarkus.native.enabled>true</quarkus.native.enabled>
   </properties>
 </profile>

Here we initialize all classes in the software.amazonaws.example.product.dao package at runtime. Otherwise we run into the error when creating the GraalVM Native Image.

To build our application as a GraalVM native image, we have to pass the native profile as a parameter. The call then looks like this: mvn clean package -Dnative. This will save GraalVM Native Image as a file named bootstrap archived in function.zip.

The last part of the changes concerns AWS SAM template.yaml. Since there is no managed Lambda GraalVM runtime environment, the question arises how we can deploy our native GraalVM image on AWS Lambda. This is possible if we choose Lambda Custom Runtime as the runtime environment (this currently only supports Linux) and deploy the built .zip file as a deployment artefact. You can find out more about this in the article Building a custom runtime for AWS Lambda. This is exactly what we define in template.yaml as follows:

Globals:
  Function:
    Handler: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest
    CodeUri: target/function.zip
    Runtime: provided.al2023  
    ....

With Runtime provided.al2023 we define Lambda runtime environment as Amazon Linux 2023 Custom Runtime and with CodeUri target/function.zip we define the path to the deployment artifact compiled with Maven in the previous step. Deployment works the same way with sam deploy -g. The API is secured with the API key. We have to send the following as HTTP header: "X-API-Key: 6ZbcDefQW12BN56WES318", see MyApiKey definition in template.yaml. To create the product with ID=1, we can use the following curl query:

curl -m PUT -d '{ "id": 1, "name": "Print 10x13", "price": 0.15 }' -H "X-API-Key: 6ZbcDefQW12BN56WES318" https://{$API_GATEWAY_URL}/prod/products

For example, to query the existing product with ID=1, we can use the following curl query:

curl -H "X-API-Key: 6ZbcDefQW12BN56WES318" https://{$API_GATEWAY_URL}/prod/products/1

In both cases, we need to replace the {$API_GATEWAY_URL} with the individual Amazon API Gateway URL that is returned by the sam deploy -g command. We can also search for this URL when navigating to our API in the Amazon API Gateway service in the AWS console.

Measurements of the Lambda cold and warm start times of our application with GraalVM Native Image

In the following, we will measure the performance of our GetProductByIdFunction Lambda function, which we will trigger by invoking curl -H "X-API-Key: 6ZbcDefQW12BN56WES318" https://{$API_GATEWAY_URL}/prod/products/1. I used GraalVM version 23 as I did my measurements, you can use the recent version which is at the time of publishing already 24.

The results of the experiment are based on reproducing more than 100 cold starts and about 100,000 warm starts with the Lambda function GetProductByIdFunction (we ask for the already existing product with ID=1 ) for the duration of about 1 hour. We give Lambda function 1024 MB memory, which is a good trade-off between performance and cost. We also use (default) x86 Lambda architecture. For the load tests I used the load test tool hey, but you can use whatever tool you want, like Serverless-artillery or Postman.

Cold (c) and warm (w) start time with in ms:

Conclusion

In the this part of our series, we introduced how to adjust our sample application to one from which we can build the GraalVM Native Image and deploy it as a Lambda Custom Runtime. Comparing to the measurements done with the managed Java 21 Lambda runtime including enabling Lambda SnapStart and applying priming summarized in the part 4 we see that we achieves the lowest cold start times with GraalVM Native Image and but warm start times are higher comparing to using managed Java 21 Lambda runtime. So there is a trade-off here. You can als try to give Lambda less memory than 1024 MB (something between 512 and 768 MB) when using GraalVM Native Image and re-measure performance because this approach offers quite stable results also with the lower Lambda memory setting.

As you saw above, using GraalVM with Quarkus 3 (I use the version 3.18) required a lot of workarounds to be able to compile a native image. So, there is definitely an improvement potential for the Quarkus framework to make it work with GraalVM Native Image without such big changes in the source code and dependency management.