Jets deploy doesn't work after deleting AWS functions: UPDATE_FAILED AWS::CloudFormation::Stack

Jets
2

Thanks for the kind words.

Yup, the CloudFormation stack is out of sync with the actual resources. In these cases, it’s a bit of a pain. But the silver lining is that you get to learn how out to get out of these situations. I’ve learned a lot about CloudFormation and all sorts of situations with experience, not just Jets related. I usually end up deleting everything to get back to a clean slate.

General Steps

Go to the CloudFormation console and delete stacks manually to get unstuck.

  • Start by manually to deleting the parent stack (it will fail)
  • Particularly, the nested child stack with the out-of-sync functions will fail.
  • Then manually delete each child stack by clicking it and deleting each one that is failing.
  • If you delete the same stack multiple times, then CloudFormation will eventually (usually the 2nd time) give you a checkbox to “keep” the resources CloudFormation cannot successfully delete. In your case, it will be the functions that have already been deleted that CloudFormation cannot find. So you’re not really “keeping” the functions, you’re just telling CloudFormation to “ignore” it and move on to delete the other resources it knows about.
  • You repeat this for until the misbehaving child stacks are gone.
  • Eventually you’ll only have the parent stack left and in-sync child stacks left.
  • You can try a jets delete again and might be done.
  • If you keep going until you get all the way to just the parent stack by itself. When you try to delete that the parent stack, it will eventually say that it cannot delete the stack because the s3 bucket that is created by the parent stack and managed by Jets is not empty.

At this point you have to 2 options:

  • Run jets delete. It should 1. empty out the s3 bucket 2. delete the parent stack 3. and clean out the CloudWatch logs.

Or:

  • Manually empty the S3 bucket
  • Manually delete the parent stack.
  • Manually delete the CloudWatch logs associated with that Jets environment.

Here’s a loop might be helpful:

for i in $(aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev) ; do echo $i ; aws logs delete-log-group --log-group-name $i ; done

Note the grep demo-dev. Replace it with the project namespace that you are using.

You can build up to the command by running it separately first:

aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev # separately first

Then run the loop:

for i in $(aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev) ; do echo $i ; aws logs delete-log-group --log-group-name $i ; done

Hope that helps.

2 Likes