Intermittent "A conflicting conditional operation is currently in progress against this resource" error with terraspace all

jalam · February 17, 2022, 2:07am

Hi Tung and BoltOps community,

It’s been a while and trust you are keeping well and safe.

I’m intermittently encountering an issue when running terraspace all plan or terraspace all up in where I have

many stacks
many batches (up to 7 batches)
each batches run a couple of stacks (up to 4 stacks per batch)

The behaviour is somewhat intermittent, though from experience, it seems to throw the following error when at around the 4th stack of the first batch of doing a terraspace all plan (and also usually during execution of the stacks in the first batch of a terraspace all up). Below is an example of the error message

/var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': A conflicting conditional operation is currently in progress against this resource. Please try again. (Aws::S3::Errors::OperationAborted)
	from /var/lib/gems/2.7.0/gems/aws-sdk-s3-1.112.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-s3-1.112.0/lib/aws-sdk-s3/plugins/dualstack.rb:27:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-s3-1.112.0/lib/aws-sdk-s3/plugins/accelerate.rb:56:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	from /var/lib/gems/2.7.0/gems/aws-sdk-core-3.126.2/lib/seahorse/client/request.rb:72:in `send_request'
	from /var/lib/gems/2.7.0/gems/aws-sdk-s3-1.112.0/lib/aws-sdk-s3/client.rb:10682:in `put_bucket_tagging'
	from /var/lib/gems/2.7.0/gems/terraspace_plugin_aws-0.3.7/lib/terraspace_plugin_aws/interfaces/backend/bucket/tagging.rb:12:in `tag'
	from /var/lib/gems/2.7.0/gems/terraspace_plugin_aws-0.3.7/lib/terraspace_plugin_aws/interfaces/backend/bucket.rb:37:in `tag'
	from /var/lib/gems/2.7.0/gems/terraspace_plugin_aws-0.3.7/lib/terraspace_plugin_aws/interfaces/backend/bucket.rb:15:in `create'
	from /var/lib/gems/2.7.0/gems/terraspace_plugin_aws-0.3.7/lib/terraspace_plugin_aws/interfaces/backend.rb:10:in `call'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner/backend.rb:25:in `create'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner.rb:49:in `block in terraform'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/hooks/builder.rb:25:in `run_hooks'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/hooks/concern.rb:6:in `run_hooks'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner.rb:48:in `terraform'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner.rb:15:in `block in run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner.rb:89:in `time_took'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/terraform/runner.rb:14:in `run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/init.rb:23:in `block in init'
	from /usr/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
	from /usr/lib/ruby/2.7.0/timeout.rb:33:in `block in catch'
	from /usr/lib/ruby/2.7.0/timeout.rb:33:in `catch'
	from /usr/lib/ruby/2.7.0/timeout.rb:33:in `catch'
	from /usr/lib/ruby/2.7.0/timeout.rb:110:in `timeout'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/init.rb:22:in `init'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/init.rb:12:in `run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/commander.rb:10:in `run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:122:in `run_terraspace'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:46:in `block (3 levels) in deploy_batch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:44:in `fork'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:44:in `block (2 levels) in deploy_batch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:43:in `each'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:43:in `block in deploy_batch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:42:in `each'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:42:in `each_slice'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:42:in `deploy_batch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:35:in `block in deploy_batches'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:33:in `each'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:33:in `each_with_index'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:33:in `deploy_batches'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:16:in `block in run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:172:in `time_took'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/all/runner.rb:12:in `run'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/all.rb:43:in `plan'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/command.rb:61:in `dispatch'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/command.rb:61:in `dispatch'
	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/lib/terraspace/cli/concern.rb:65:in `start'
	from /var/lib/gems/2.7.0/gems/terraspace-1.1.1/exe/terraspace:14:in `<top (required)>'
	from /usr/local/bin/terraspace:23:in `load'
	from /usr/local/bin/terraspace:23:in `<main>'

Admittedly, I’ve not checked the logs (as these are run from CI/CD pipelines, and they get removed at the end of each run).

I’ve been looking around for answers to the error message (as it seems to be AWS related rather than Terraspace related), and most of the answers I’ve seen relates to the deletion/recreation of an S3 bucket (which I imagine won’t normally be done from doing a “plan”). A bit of a long shot and I might be way off the mark here… But I am wondering if this would be related to how terraspace is dealing with s3 buckets (e.g. checking for existence of remote backend s3 buckets/applying bucket policies/tags, etc)?

P.S. I am using terraspace v1.1.1

Many thanks,
James

tung · February 17, 2022, 3:41am

Thanks for the stack trace. Think what’s happening is that multiple processes are updating the s3 bucket tags at the same time. https://github.com/boltops-tools/terraspace_plugin_aws/blob/199bbcec4ad89bb48e05ce7af1774ced07d2bc5d/lib/terraspace_plugin_aws/interfaces/backend/bucket/tagging.rb#L12

Hence the:

A conflicting conditional operation is currently in progress against this resource

As a quick fix, try disable config.tag_existing

Docs: https://terraspace.cloud/docs/plugins/aws/

TerraspacePluginAws.configure do |config|
  config.tag_existing = false
end

Newly created state buckets will be tagged properly but not existing buckets.

Think will have to be smarter about tagging the bucket. Guessing most of the time it’s the same bucket and same tags. Perhaps some caching. IE: For calls with the same bucket and tags, it only needs to be performed once across all terraspace up forked processes.

Also, can add some exponential back off and retry logic.

Will consider PRs. Of course, no sweat either way. Hope to get to it in time

jalam · February 17, 2022, 5:53am

Hi Tung,

Thanks for the prompt and useful response as always!

Just an update, I’ve tried as you’ve suggested. i.e. putting the following in config/plugins/aws.rb

TerraspacePluginAws.configure do |config|
  config.tag_existing = false
end

and have not seen the errors again so far

Cheers,
James