I did try the sample apps you provided. Didn’t see any differences between demo13 & demo17 with or without prewarming enabled.
We are not using the Jets AWS clients. We create our own for use within the application code.
I think you are on to something wrt prewarmer making AWS api calls & the rate-limit changes.
By default Async lambda invocations will retry on failure. Might not want that behavior in the prewarmer.
It’s worth mentioning our app has 175 lambdas (just recounted) … but we have two deployments (staging, production) with prewarming enabled.
So that’s 350 lambdas in the account prewarming. Perhaps we are hitting rate limits on the SDK and the lambdas are also retrying.
I checked and we are getting a lot of errors on the jets-preheat_job-warm lambda. More errors than invocations (looks like retry’s happening) Fewer errors on the hcr-prod-jets-preheat_job-torch lambda. Sadly, retry’s will just make the problem worse in this case.
So if a warm job fails after the back-off strategy is exhausted, the lambda will be retried 2 more times before giving up, adding even more load on the SDK.