New s3 backend fails due to ACL rule

Terraspace
1

I have a fairly conventional s3 backed config which has being working for 18 months +, however I recently tried to move this config to a new AWS account and got an ACL error

It appears that AWS no longer allow ACLs on s3 buckets by default, which is addressed by hashicorp here: https://github.com/hashicorp/terraform-provider-aws/issues/28353

This states that the default (either ommiting acl or setting it to private) should work, however this doesn’t work with terraspace.

My config:

terraform {
  backend "s3" {
    bucket         = "<%= expansion('terraform-state-:ACCOUNT-:REGION') %>"
    key            = "<%= expansion(':MOD_NAME/:APP/:ROLE/dev/:EXTRA/terraform.tfstate') %>"
    region         = "<%= expansion(':REGION') %>"
    encrypt        = true
    dynamodb_table = "terraform_locks"
  }
}

The bucket actually gets created, but then it fails instantly when trying to do something with the ACLs:

opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': The bucket does not allow ACLs (Aws::S3::Errors::AccessControlListNotSupported)
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.128.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.128.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.128.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.177.0/lib/seahorse/client/request.rb:72:in `send_request'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.128.0/lib/aws-sdk-s3/client.rb:9484:in `put_bucket_acl'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/s3-secure-0.6.1/lib/s3_secure/access_logs/enable.rb:16:in `add_bucket_acl'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/s3-secure-0.6.1/lib/s3_secure/access_logs/enable.rb:5:in `run'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/terraspace_plugin_aws-0.6.0/lib/terraspace_plugin_aws/interfaces/backend/bucket/secure.rb:33:in `secure'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/terraspace_plugin_aws-0.6.0/lib/terraspace_plugin_aws/interfaces/backend/bucket.rb:14:in `create'
        from /opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/terraspace_plugin_aws-0.6.0/lib/terraspace_plugin_aws/interfaces/backend.rb:10:in `call'

Any ideas on where to start with debugging this?

I’ve tried setting the acl = private, but it gets the same error

2

I see. This is related to Amazon S3 Block Public Access – Another Layer of Protection for Your Accounts and Buckets | AWS News Blog

Released a fix https://github.com/boltops-tools/s3-secure/pull/8 To update:

cd your-terraspace-project
bundle update s3-secure
bundle info s3-secure

To fix it manually, you can edit the bucket permissions manually:

  1. Under the S3 Console. Go the Permissions Tab. In the Object Ownership section, click Edit
  2. Select ACLs enabled, check I acknowledge that ACLs will be restored. and select Object writer
  3. Click Save Changes
3

Thanks for that!

Does this take a while to work its way through to the rubygems website? I ran the update but it stays at s3-secure 0.6.1…

4

actually it seems like the terraspace_plugin_aws is to blame, I set the s3-secure version to 0.7.0 in the Gemfile

like:

gem "s3-secure", "~> 0.7.0"

and running bundle update downgraded terraspace_plugin_aws to v0.3.4

Fetching s3-secure 0.7.0 (was 0.6.1)
Installing s3-secure 0.7.0 (was 0.6.1)
Fetching terraspace_plugin_aws 0.3.4 (was 0.6.0)
Installing terraspace_plugin_aws 0.3.4 (was 0.6.0)
Bundle updated!

seems a bit weird?

5

Oh I see. My bad. Need to loosen and fix the dependency version in the terraspace_plugin_gem to allow bundle to resolve the dependencies correctly.

On the road right now. Will take a look tomorrow. Cheers.

1 Like
6

Ok. Fixed by https://github.com/boltops-tools/terraspace_plugin_aws/pull/26 Try:

cd your-terraspace-project
bundle update terraspace_aws_plugin
bundle info terraspace_aws_plugin
bundle info s3-secure
7

that has sorted it, thanks!