How to debug AWS Lambda functions in production

Serverless is pretty cool but native debugging has always been a burden for developers. There have been many attempts to debug Lambdas, including simulating the cloud environment and mocking managed services. However, these solutions lack two crucial capabilities. They can not simulate the security permissions of AWS Lambda functions and they can not keep up with the authenticity of event data flowing between resources. In other words, they can not help you debug Lambdas in production.

In this article, I looked into a VSCode plugin which makes it possible to debug AWS Lambda functions live on AWS. It is possible to put breakpoints in your Lambdas and debug just one invocation while not blocking the whole flow. This way, you will have the exact debugging data because you debug on the real AWS environment with the real authentic data. It doesn't require adding code, mocking your resources, or simulating the app.

Introduction

Start online debug sessions while the AWS Lambda functions run on AWS Cloud. The debugger sets up an easy and secure connection between your AWS Lambda functions and VSCode. You can debug your serverless applications natively with their permissions. When you have an erroneous invocation, you can put breakpoints and pause the execution of the Node.js and Python functions to see the stack trace.

Installation

To use the AWS Lambda Debugger, you need to complete all installation steps.

• Install the extension from the marketplace.
• Sign up for Thundra and get your THUNDRA_AGENT_LAMBDA_DEBUGGER_AUTH_TOKEN. If you've already signed up, you can get your auth token from the start page.
• Add this key to both your VS Code configuration and your AWS Console:

1. Open the Command Palette (⇧⌘P) and select Thundra: Edit Configuration command to paste the auth token to the authToken field.

2. Go to your AWS Console and set your auth token value with the THUNDRA_AGENT_LAMBDA_DEBUGGER_AUTH_TOKEN key as your environment variable.

• After installing the extension, instrument your AWS Lambda. You can do it in two ways.
  
  • Using Layers (easier)
    1. Guide for NodeJS
    2. Guide for Python

How to use

• Click on the “Start Thundra Debugger” button on the Status Bar. You can also execute the command Thundra: Start Debugger to start the debugging session.

• Set a debug point on your VS Code. You can set multiple debug points as well.
• Now invoke your AWS Lambda function to hit on the debug point.
• When there is a hit on your breakpoint, the status bar below turns red.
• You can start debugging when the variables and the call stack are loaded.
• Debug session ends when your AWS Lambda function times out. You can update the timeout of your function for longer debug sessions.

Commands

• Thundra: Start Debugger to start the debugger with the current profile's settings
• Thundra: Start Debugger (with a session name) to start the debugger with the given session name
• Thundra: Change active profile to change the active profile to the one given in your config file
• Thundra: Edit configuration to edit the JSON file containing your debugger configuration

You can access all of the above commands from the command palette (Cmd+Shift+P or Ctrl+Shift+P).

Self-hosted Broker

A self-hosted instance of the broker that provides the necessary functionality for online debugging can be installed on your AWS account using our CloudFormation template or our CDK application.

You can configure the self-hosted broker with different access settings as described in the below image:

CloudFormation Template

The Amazon CloudFormation template provides all the required resources for setting up the online debugging broker. You can easily install it on your own AWS account following the steps below. You will have both a public and an internal endpoint for the broker as the output. You can choose to have DNS mappings done automatically as subdomains under a Route53 hosted zone that you specify.

Installation Steps

1. SSL Certificate Preparation If you want to enable a secure connection via HTTPS, you will need an SSL certificate signed for the domains you are planning to use as the debugger endpoints. If you don’t have an available SSL certificate on AWS Certificate Manager, you can upload your existing certificate or request a new one by following this guide.
2. Parameter Settings Create a CloudFormation stack by providing the parameters in this link using the template:
3. DNS Configuration

This configuration is created automatically if you have provided a “Hosted Zone Name” parameter.

This configuration is not mandatory if you don’t provide an SSL certificate. Without the certificate, you can directly use the endpoints in the “Outputs” section.

You will see two URLs named BrokerInternalURL and BrokerPublicURL under the “Output” section after the stack creation is completed. You will need to update your DNS records to direct your reserved domains for the Thundra debugging broker to these endpoints.

For example: thundra-debugger-public.yourdomain.com -> BrokerPublicURL thundra-debugger-internal.yourdomain.com -> BrokerInternalURL

Debug Tool Configuration

Your debugging environment most likely has access to the internet and the best option at this point is to configure it to connect to the public broker endpoint. A secure connection is provided if you have an SSL certificate for the installation. If this is not the case or you want to use the internal endpoint anyway, you can access it from the debugging environment as long as it is connected to your VPC via a VPN tool.

You can use values under the “Output” section of the stack for the Thundra debugging client configuration. Use “BrokerPubliclUrl” or “BrokerInternalUrl” in order to debug with the public endpoint or the internal endpoint. Use “BrokerPort” as the broker host parameter.

Please note that if you make a manual DNS configuration, you should use the respective DNS record values as the broker host parameter instead of the values under the “Output” section.

Lambda Configuration

Using the public endpoint is also the best option for your Lambda functions if they have internet access, but this may not be possible all the time. If you need to configure your Lambda functions to connect to the internal endpoint, you also need to be sure that you have an appropriate security configuration in order to access the debugging broker from your Lambda functions.