Really long coldstart

Hellos! I’m having an issue where the Lambda cold start is really long, something like 90 seconds. The first few API Gateway calls timeout (since it’s 30sec max timeout). After the cold start the API calls return very quickly, eg 150ms for simple call, 1000ms for a large DB query.

My setup has RDS, so the Lambdas are in a VPC (not sure that makes a difference).

My main question is how to even troubleshoot this?

I have the prewarm scripts running but I need to troubleshoot if they’re working.

I’m using Jets version 2.2.5.

Update: looks like the prewarm Lambda weren’t working because they were in the private subnets so couldn’t receive Lambda invocations. That’s now fixed following this guide:

VPC makes a difference. See: It’s crazy that you’re seeing 90s. Feel like it’s another issue for it to be that long. When I tested, the cold start was about 6-9s.

There’s a note in the doc about recent AWS Lambda VPC upgrades. The upgrades are rolling out. Some AWS accounts have already seen the VPC speed improvements. There’s no specific timeline for all accounts.

Cool. Thanks for the update. Here’s some context that may help others. Jets prewarming works by calling a Lambda function that calls all the other controller-based lambda functions in the application. It uses the AWS Lambda Invoke API. The Lambda Invoke API is an outbound call out to the internet. Hence, the function needs internet access. This is why private subnets need to have NAT gateways. NAT Gateways provide private subnets with outbound internet access.

Note: There is also something called VPC Endpoints. At this time, it does not look like private endpoints support Lambda yet.