Error handling in AWS Lambda with SQS event source

In an earlier post, we have seen how SQS can be used as an event source to trigger AWS Lambda function. But we had ignored error handling and focused only on the happy path. In this post we will extent that and see how Dead Letter Queue(DLQ) can be used for error handling.

Default behavior

First lets see default behavior of a Lambda function triggered by SQS message in case of error.

• Induce an error in the Lambda python code in earlier post

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

import json def lambda_handler(event, context): body = { "message": "Hello SQS", "event": event } print(json.dumps(body)) raise Exception("Dummy Exception") response = { "statusCode": 200, "body": json.dumps(body) } return response

• Send a sample message from SQS queue "testLambdaQueue". This will trigger Lambda function (testSQSFunction) and the error will be thrown.
• Check CloudWatch log: As seen in the CloudWatch Log, Lambda functions encounters the error and exits. Thus the message is not deleted from the queue (testLambdaQueue) and will again be pulled by the Lambda function once Visibility Timeout (30 second by default) is over. This cycle will go on till message is remove once Retention Period (4 days by default) is over. 

CloudWatch Log Stream

• Purge testLambdaQueue to remove the In-Flight message 

Using DLQ for error handling

Lets enhance the current setup to include a DLQ.

• Create a new Standard SQS Queue named "LambdaDLQ". This is the DLQ that we will use for error handling.
• Attach the DQL with Lambda function: Go to Lambda > Functions > testSQSFunction > Configuration and "Debugging and error handling" section, select LambdaDLQ and save.

 

• Change Redrive Policy of SQS queue that triggers the Lambda function: Select testLambdaQueue > Queue Actions > Configure Queue. Select Use Redrive Policy, set Dead Letter Queue to LambdaDLQ and Maximum Receives to 2. This setting will push testLambdaQueue message to LambdaDQL after 2 retry  by the Lambda function.

 

• Send a test message from testLambdaQueue: As we can see below 1 message is in In-Flight for testLambda queue in SQS console

• Check CloudWatch log: It shows that Lambda function has tried to process the message twice. Post that the message to be pushed to DLQ

• And as expected we can see in SQS console that the message has moved from testLambdaQueue to LambdaDLQ

In this post we have seen how Dead Letter Queue (DLQ) can be effectively used for error handling in Lambda functions triggered by SQS events. An advantage of this approach is that it decouples application logic from error handling logic.