Unable to deploy with "already exists in stack" upgrading from Jets 4 -> 5

I’m finally getting around to updating my oldest Jets app from Jets 4 to Jets 5. I had a few challenges getting everything working again locally, but nothing major. Now, I’m unable to deploy with CloudFormation errors “already exists in stack” on basically all of my resources. How can I get past this?

Seems like Jets is thinking it is creating new resources inside the nested stacks when they actually already exist. Is there a safe way to just re-create the whole stack?

➭ bundle exec jets deploy
Deploying stack bot-dev ...
The /tmp/jets/bot/cache folder exists. Incrementally re-building the jets using the cache.  To clear the cache: rm -rf /tmp/jets/bot/cache
=> Copying current project directory to temporary build area: /tmp/jets/bot/stage/code
=> Bundling: running bundle install in cache area: /tmp/jets/bot/cache.
=> cd /tmp/jets/bot/cache && env bundle install
Bundle complete! 27 Gemfile dependencies, 146 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/gems`
Bundle install completed
Tidying project: removing ignored files to reduce package size.
=> rsync -a --links /tmp/jets/bot/stage/code/vendor/gems/ruby/3.2.0/ /tmp/jets/bot/stage/opt/ruby/gems/3.2.0/
=> Replacing compiled gems with AWS Lambda Linux compiled versions: /tmp/jets/bot/stage/opt
Checking project for compiled gems...
=> Generating shims in the handlers folder.
Checking s3://bot-dev-s3bucket-4z9iwp6ke1ru/jets/code/opt-c9691013.zip
Already exists: s3://bot-dev-s3bucket-4z9iwp6ke1ru/jets/code/opt-c9691013.zip
Checking s3://bot-dev-s3bucket-4z9iwp6ke1ru/jets/code/code-82d98f0e.zip
=> Creating zip file for /tmp/jets/bot/stage/code
=> cd /tmp/jets/bot/stage/code && chmod -R 755 . && zip --symlinks -rq code.zip .
Zip file created at: /tmp/jets/bot/stage/zips/code-82d98f0e.zip (160 KB)
Building CloudFormation templates
Built CloudFormation templates at /tmp/jets/bot/templates
Uploading CloudFormation templates to S3.
Uploading code zip files to S3.
Uploading /tmp/jets/bot/stage/zips/code-82d98f0e.zip (160 KB) to S3
Uploaded to s3://bot-dev-s3bucket-4z9iwp6ke1ru/jets/code/code-82d98f0e.zip
Time to upload code to s3: 0s
Checking for modified public assets and uploading to S3.
Time for public assets to s3: 0s
Deploying CloudFormation stack with jets app!
Waiting for stack to complete
05:57:16PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack bot-dev User Initiated
05:57:19PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
05:57:19PM UPDATE_IN_PROGRESS AWS::IAM::Policy IamPolicy
05:57:19PM UPDATE_IN_PROGRESS AWS::S3::Bucket S3Bucket
05:57:19PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiGateway
05:57:20PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1
05:57:31PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiResources1
05:57:35PM UPDATE_COMPLETE AWS::IAM::Policy IamPolicy
05:57:41PM UPDATE_COMPLETE AWS::S3::Bucket S3Bucket
05:57:42PM UPDATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Requested update requires the creation of a new physical resource; hence creating one.
05:57:53PM UPDATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
05:57:53PM UPDATE_COMPLETE AWS::Lambda::LayerVersion GemLayer
05:57:54PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack AmazonFulfillmentJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack AttributionJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack OrderUpdaterJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack InventoryJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack EmailJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack PartnerFulfillmentJob
05:57:54PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ReturnLabelJob
05:57:55PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController Resource creation Initiated
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack EmailJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-EmailJob-1KRD7LZMLA143/3ecedc00-25b0-11ee-8991-128d4a17231b was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [EmailJobInboundLambdaFunction, EmailJobS3LambdaFunction].
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack OrderUpdaterJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-OrderUpdaterJob-1AHA4SQXGB1W/fea95c30-ff75-11ea-bfd8-0e69ad9c0ded was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [OrderUpdaterJobSendTaxExemptEmailLambdaFunction, OrderUpdaterJobProcessOrdersLambdaFunction, OrderUpdaterJobSendIntlReturnInstructionsLambdaFunction, OrderUpdaterJobAmazonAdditionalVatReportLambdaFunction, OrderUpdaterJobReportKatanaAmazonOrdersLambdaFunction, OrderUpdaterJobPartnerSalesReportLambdaFunction, OrderUpdaterJobS3LambdaFunction, OrderUpdaterJobDeleteCancelledAmazonFromKatanaLambdaFunction, OrderUpdaterJobReportKatanaSyncErrorsLambdaFunction, OrderUpdaterJobFulfillInKatanaLambdaFunction, OrderUpdaterJobTaxuallyVatReportLambdaFunction, OrderUpdaterJobSendDomesticReturnLabelLambdaFunction].
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack ReturnLabelJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-ReturnLabelJob-IIN9PK9F0FWY/55b69040-04cf-11e9-ae8c-0aaa9d7b8aca was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [ReturnLabelJobS3LambdaFunction, ReturnLabelJobDomesticReturnLambdaFunction, ReturnLabelJobGenerateLambdaFunction, ReturnLabelJobIntlReturnLambdaFunction].
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack AmazonFulfillmentJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-AmazonFulfillmentJob-NFGVVF1DEWUR/c5012cc0-d79c-11ea-9f9d-124aa02dc2ed was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [AmazonFulfillmentJobS3LambdaFunction, AmazonFulfillmentJobAssignBuyWithPrimeOrderLambdaFunction, AmazonFulfillmentJobFulfillOrdersLambdaFunction, AmazonFulfillmentJobRefreshOrdersLambdaFunction, AmazonFulfillmentJobRefreshOrderLambdaFunction, AmazonFulfillmentJobSyncFbaSalesLambdaFunction].
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack InventoryJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-InventoryJob-DC8L112BK9BY/0b349460-8e29-11ea-9a60-0e167b726915 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [InventoryJobFulfillmentKpisLambdaFunction, InventoryJobDailySnapshotLambdaFunction, InventoryJobCostUpdaterLambdaFunction, InventoryJobDailyReportLambdaFunction, InventoryJobAmazonRestockReportLambdaFunction, InventoryJobS3LambdaFunction, InventoryJobReorderNudgeLambdaFunction, InventoryJobFulfillmentKpisEomReportLambdaFunction].
05:58:05PM UPDATE_FAILED AWS::CloudFormation::Stack PartnerFulfillmentJob Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-PartnerFulfillmentJob-OAFOD3TGUV6H/c61452b0-3bc9-11ee-bb04-12408ce141f5 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [PartnerFulfillmentJobSendInvoiceLambdaFunction, PartnerFulfillmentJobFulfillLambdaFunction, PartnerFulfillmentJobSendFoToPartnerLambdaFunction, PartnerFulfillmentJobUpdateTrackingLambdaFunction, PartnerFulfillmentJobS3LambdaFunction, PartnerFulfillmentJobRequestFulfillmentsLambdaFunction].
05:58:05PM CREATE_FAILED AWS::CloudFormation::Stack JetsController Resource creation cancelled
05:58:06PM UPDATE_FAILED AWS::CloudFormation::Stack AttributionJob Resource update cancelled
05:58:06PM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack bot-dev The following resource(s) failed to create: [JetsController]. The following resource(s) failed to update: [EmailJob, AmazonFulfillmentJob, InventoryJob, AttributionJob, OrderUpdaterJob, PartnerFulfillmentJob, ReturnLabelJob].
05:58:09PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
05:58:09PM UPDATE_IN_PROGRESS AWS::IAM::Policy IamPolicy
05:58:09PM UPDATE_IN_PROGRESS AWS::S3::Bucket S3Bucket
05:58:09PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiGateway
05:58:09PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1
05:58:20PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiResources1
05:58:25PM UPDATE_COMPLETE AWS::IAM::Policy IamPolicy
05:58:31PM UPDATE_COMPLETE AWS::S3::Bucket S3Bucket
05:58:31PM UPDATE_COMPLETE AWS::Lambda::LayerVersion GemLayer
05:58:31PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack AmazonFulfillmentJob
05:58:31PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack OrderUpdaterJob
05:58:32PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack PartnerFulfillmentJob
05:58:32PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack InventoryJob
05:58:32PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack AttributionJob
05:58:32PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ReturnLabelJob
05:58:32PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack EmailJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack AmazonFulfillmentJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack OrderUpdaterJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack InventoryJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack ReturnLabelJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack AttributionJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack EmailJob
05:58:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack PartnerFulfillmentJob
05:58:54PM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack bot-dev
05:58:55PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsController
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack AmazonFulfillmentJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack OrderUpdaterJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack ReturnLabelJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiResources1
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack InventoryJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack EmailJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack PartnerFulfillmentJob
05:59:05PM UPDATE_COMPLETE AWS::CloudFormation::Stack AttributionJob
05:59:05PM DELETE_COMPLETE AWS::CloudFormation::Stack JetsController
05:59:06PM DELETE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer
05:59:06PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiGateway
05:59:07PM DELETE_COMPLETE AWS::Lambda::LayerVersion GemLayer
05:59:07PM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack bot-dev
Stack rolled back: UPDATE_ROLLBACK_COMPLETE
The Jets application failed to deploy. Jets creates a few CloudFormation stacks
to deploy your application. The logs above show the CloudFormation parent stack
events. You can go to the CloudFormation console and look for the nested stack
with the error. The specific nested stack usually shows more detailed information
and can be used to resolve the issue.
Example of checking the CloudFormation console:

    https://docs.rubyonjets.com/docs/debug/cloudformation/

--------------------------------------------------------------------------------
Here's the nested stack error details: Embedded stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-EmailJob-1KRD7LZMLA143/3ecedc00-25b0-11ee-8991-128d4a17231b was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [EmailJobInboundLambdaFunction, EmailJobS3LambdaFunction].
Stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-EmailJob-1KRD7LZMLA143/3ecedc00-25b0-11ee-8991-128d4a17231b Status: UPDATE_ROLLBACK_COMPLETE
Stack events:
05:57:55PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack bot-dev-EmailJob-1KRD7LZMLA143 User Initiated
05:57:57PM CREATE_IN_PROGRESS AWS::Lambda::Function EmailJobS3LambdaFunction
05:57:57PM CREATE_IN_PROGRESS AWS::Lambda::Function EmailJobInboundLambdaFunction
05:57:58PM CREATE_FAILED AWS::Lambda::Function EmailJobInboundLambdaFunction bot-dev-email_job-inbound already exists in stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-EmailJob-1KRD7LZMLA143/3ecedc00-25b0-11ee-8991-128d4a17231b
05:57:58PM CREATE_FAILED AWS::Lambda::Function EmailJobS3LambdaFunction bot-dev-email_job-s3 already exists in stack arn:aws:cloudformation:us-east-1:652695076726:stack/bot-dev-EmailJob-1KRD7LZMLA143/3ecedc00-25b0-11ee-8991-128d4a17231b
05:57:58PM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack bot-dev-EmailJob-1KRD7LZMLA143 The following resource(s) failed to create: [EmailJobInboundLambdaFunction, EmailJobS3LambdaFunction].
05:58:43PM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack bot-dev-EmailJob-1KRD7LZMLA143
05:58:56PM DELETE_COMPLETE AWS::Lambda::Function EmailJobInboundLambdaFunction
05:58:56PM DELETE_COMPLETE AWS::Lambda::Function EmailJobS3LambdaFunction
05:58:56PM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack bot-dev-EmailJob-1KRD7LZMLA143
Here's also the AWS Console link to the failed nested stack:

I’m still stuck here and dug into this a bit more this morning. I think I found the problem, but not sure how to solve it yet.

It looks like somewhere along the line between Jets 4 and Jets 5 the logical_id of the CloudFormation resource for the lambda functions was changed. So for example, where previously in Jets 4 the logical ID for my EmailJob#inbound job would have been named InboundLambdaFunction in the EmailJob stack, now the CloudFormation template logical ID is named EmailJobInboundLambdaFunction with the same name for the lambda function resource.

This is problematic because CloudFormation recognizes it as a new resource with a new Logical Id and tries to create the lambda function with the same name, but fails because it already exists.

I noticed in the commit message of the Jets 5 release one of the bullets there is “Fully qualify LambdaFunction logical id.” which seems to confirm what I’m observing.

I’m not sure of the best way around this. I supposed I could nuke the whole stack and recreate it, but that makes me a bit nervous.

@tung are you around? Haven’t heard from you lately and would love your thoughts/advice here.

Ok I got past the deploy problem by doing a blue-green deploy. I guess this was needed to create a fresh new environment with the new job resource naming scheme. I should have figured this earlier.

The rest of the upgrade from Jets v4 to v5 wasn’t totally smooth, although I have it working now. I had to make a couple of small code changes to Jets to get past a few bugs that I found, I think related to minor changes in 3rd-party dependencies.

Here is my fork of Jets v5 with changes: Comparing rubyonjets:v5.0.14...konnected-io:v5 · rubyonjets/jets · GitHub

@tung would you consider a v5 branch in the main Jets repo and allowing a PR to patch these issues?