Rate Limit Exceeded?

We are running into this more and more on deployments…ugghhhh AWS and Cloudformation.

Anyone else experiencing this??? Any ideas to mitigate?

Your request has been throttled by EC2, please make sure you have enough API rate limit. EC2 Error Code: RequestLimitExceeded. EC2 Error Message: Request limit exceeded. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: ea231a7b-cc32-42fe-8584-e78751cbea85)

Released a deploy stagger option https://github.com/tongueroo/jets/pull/364 It may help. Docs:

https://rubyonjets.com/docs/extras/deploy-stagger/

Wondering how many lambda functions your app creates. Thinking the rate limit error corresponds to the number of functions getting created.

Also wondering how often you run into the rate limit error. Been testing without staggering enabled and haven’t been able to trigger it yet. This is with a test app that has 200+ functions. :face_with_monocle:

Sorry this project fell of my map for a little while…we aren’t near 200 functions, only around 60ish…the problem was only occurring when deploying to VPC…maybe the network connections being setup cause the issue??? Anyways, let me pull in the deploy stagger option and see if that mitigates…as always, thanks for the quick response.

I’ll update you with what I see.

All good.

RE: VPC…maybe the network connections being setup cause the issue???

Oh interesting, that makes more sense now. With VPC the current lambdas are creating an ENI card for each lambda function. So the rate limit is reached much more quickly. The extra ec2 API calls to create the ENIs don’t happen in non-vpc.

Recently, AWS announced Lambda VPC speed improvements support. It should also help because only a few ENIs will be created instead of one for each function. Details:

Wondering if after the Lambda VPC speed improvements are fully rolled out, the rate limit issues you’re seeing will also go away. AWS says they are partially rolling it out already on some AWS accounts and will be eventually fully rolled out. No definite timeframe though.

Same error here, also running inside VPC.

Follow the stack trace:

    Calling lambda function pixugu-homolog-ApiV1Provi-LogInfoStartLambdaFuncti-SQ0PID11UUIA on AWS
    Function name: pixugu-homolog-ApiV1Provi-LogInfoStartLambdaFuncti-SQ0PID11UUIA
    #<Thread:0x0000564c2b039670@/opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/preheat.rb:35 run> terminated with exception (report_on_exception is true):
    /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Rate exceeded (Aws::CloudFormation::Errors::Throttling)
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-core-3.109.1/lib/seahorse/client/request.rb:72:in `send_request'
    from /opt/ruby/gems/2.5.0/gems/aws-sdk-cloudformation-1.44.0/lib/aws-sdk-cloudformation/client.rb:2327:in `describe_stack_resources'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call/base_guesser.rb:54:in `stack_resources'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call/base_guesser.rb:43:in `generated_function_name'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call/base_guesser.rb:33:in `function_name'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call/guesser.rb:30:in `function_name'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call.rb:22:in `function_name'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call.rb:51:in `remote_run'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/commands/call.rb:29:in `run'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/preheat.rb:26:in `warm'
    from /opt/ruby/gems/2.5.0/gems/jets-2.3.17/lib/jets/preheat.rb:36:in `block (2 levels) in warm_all'