Asynchronous HTTP request in AWS Lambda

Question

I am wanting to execute a http request inside of a lambda function, invoked by API Gateway. The problem is, the request takes a bit of time to complete (<20 seconds) and don't want the client waiting for a response. In my research on asynchronous requests, I learned that I can pass the X-Amz-Invocation-Type:Event header to make the request execute asynchronously, however this isn't working and the code still "waits" for the http request to complete.

Below is my lambda code:

'use strict';

const https = require('https');

exports.handler = function (event, context, callback) {
    let requestUrl;
    requestUrl = event.queryStringParameters.url;

    https.get(requestUrl, (res) => {
        console.log('statusCode:', res.statusCode);
        console.log('headers:', res.headers);

        res.on('data', (d) => {
            process.stdout.write(d);
        });
    }).on('error', (e) => {
        console.error(e);
    });

    let response = {
        "statusCode": 200,
        "body": JSON.stringify(event.queryStringParameters)
    };
    callback(null, response);
};

Any help would be appreciated.

Changing the invocation type means that the calling code, the code invoking the Lambda function, won't wait for the Lambda function to finish. The Lambda function itself is still going to wait for the HTTP call to finish, but whatever called the Lambda function doesn't have to wait for that. — Mark B
– Mark B, Commented Jan 19, 2018 at 21:36
"I learned that I can pass the X-Amz-Invocation-Type:Event header" Please define "pass." You need to set it in the Integration Request. Also, with an Async Lambda invocation, the response returned by your Lambda function is discarded, since there is nowhere for it to go. Is that what you intended? — Michael - sqlbot
– Michael - sqlbot, Commented Jan 19, 2018 at 21:42
@Michael-sqlbot This is what I want as I do not care about the response, and I can handle any errors within the lambda function (log to cloudwatch or something). I would like the http request to "fire and forget", if that makes sense. When I say "pass", I am setting that header in the client request. I did not see a place to add this header (and the value) in the Integration Request UI. — Bobby Bruce
– Bobby Bruce, Commented Jan 20, 2018 at 14:09
@MarkB Understood. I am setting that header in my client request, but Michael below you insists that it should be set in the Integration Request in API Gateway's UI. I don't see a field to add Header/Value. — Bobby Bruce
– Bobby Bruce, Commented Jan 20, 2018 at 14:11
@BobbyBruce Michael is correct, you don't pass that value to your API Gateway endpoint, you pass it to the AWS API Lambda Invocation endpoint. I'm not sure it's possible to do this via API Gateway since it will always want a response from the Lambda function. This answer suggests chaining 2 Lambda functions in order to achieve the desired result: stackoverflow.com/questions/34294693/… — Mark B
– Mark B, Commented Jan 20, 2018 at 15:20

Noel Llevares · Accepted Answer · 2018-01-20 00:10:06Z

3

You can use two Lambda functions.

Lambda 1 is triggered by API Gateway then calls Lambda 2 asynchronously (InvocationType = Event) then returns a response to the user.

Lambda 2, once invoked, will trigger the HTTP request.

answered Jan 20, 2018 at 0:10

Noel Llevares

16.2k5 gold badges60 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bobby Bruce Over a year ago

Sounds like it could work. I am wondering how to call the 2nd lambda with that invocation type. If you have documentation on that, that would be extremely helpful. Thank you

aleksikallio Over a year ago

You can find the info in AWS JavaScript SDK docs. Make sure that you use the InvocationType Event, don't use a callback and call AWS.Request.send() on the request object.

Warren Parad · Accepted Answer · 2021-12-04 11:54:12Z

Whatever you do, don't use two lambda functions.

You can't control how lambda is being called, async or sync. The caller of the lambda decides that. For APIGW, it has decided to call lambda sync.

The possible solutions are one of:

SQS
Step Functions (SF)
SNS

In your API, you call out to one of these services, get back a success, and then immediately return a 202 to your caller.

If you have a high volume of single or double action execution use SQS. If you have potentially long running with complex state logic use SF. If you for someone reason want to ignore my suggestions, use SNS.

Each of these can (and should) call back out to a lambda. In the case that you need to run more than 15 minutes, they can call back out to CodeBuild. Ignore the name of the service, it's just a lambda that supports up to 8 hour runs.

Now, why not use two lambdas (L1, L2)? The answer is simple. Once you respond that your async call was queued (SQS, SF, SNS), to your users (202). They'll expect that it works 100%. But what happens if that L2 lambda fails. It won't retry, it won't continue, and you may never know about it.

That L2 lambda's handler no longer exist, so you don't know the state any more. Further, you could try to add logging to L2 with wrapper try/catch but so many other types of failures could happen. Even if you have that, is CloudWatch down, will you get the log? Possible not, it just isn't a reliable strategy. Sure if you are doing something you don't care about, you can build this arch, but this isn't how real production solutions are built. You want a reliable process. You want to trust that the baton was successfully passed to another service which take care of completing the user's transaction. This is why you want to use one of the three services: SQS, SF, SNS.

Collectives™ on Stack Overflow

Asynchronous HTTP request in AWS Lambda

2 Answers 2

2 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Linked

Related