Customized layering support?

Terraspace
1

I’m still trying at a high level to see if terraspace might be a good direction for us to migrate from terragrunt to solve some pains we are experiencing.

Does terraspace support customizing the way layering is done?

For example, a typical terragrunt project file for us might look like this:

include {
  path = "${find_in_parent_folders("account_remote_state.hcl")}"
}

terraform {
  source = "git@github.com:org/repo//modules/module"

  extra_arguments "required_files" {
    commands = [
      "apply",
      "plan",
      "destroy",
      "import",
      "push",
      "refresh",
    ]

    required_var_files = [
      "${find_in_parent_folders("global.tfvars")}",
      "${find_in_parent_folders("team.tfvars")}"
      "${find_in_parent_folders("env.tfvars")}",
      "${find_in_parent_folders("account.tfvars")}",
      "${find_in_parent_folders("region.tfvars")}",
    ]
  }
}

Context:

  • we have multiple teams
  • we have multiple envs – for example, qa, stg, prd
  • each team has at least one account per environment of a certain account type
  • a few teams might have multiple accounts used as different account types.
  • within an account, multiple regions are supported.

I see from the https://terraspace.cloud/docs/tfvars/full-layering/ doc that there seems to be some support for specifying per aws account configs, such as:

provider/namespace/region aws/112233445566/us-west-2/{base,dev,instance}.tfvars

Ideally we could avoid having to use account ids in the path to tfvars files as we have numerous accounts and its not immediately obvious the context behind the account.

We would be looking for a path akin to this:

aws/team_a/qa/service/us-east-2/base.tfvars

where: 

team = team_a
env = qa
account_type = service
region = us-east-2

Thanks for any suggestions on how to handle this. Happy to provide more context as it possible I didn’t explain things right.

Cheers.

2

I’m digging in a bit more here Seems there’s the concept of a namespace, and by default the AWS plugin has the namespace be the accountid. I wonder if there was a way I could define my own plugin which extends the AWS plugin but changes the namespace to be something else for me, such as ENV-TEAM-TYPE

I’m extremely rusty on my ruby skills to make this happen. I did try to run a generator for a new plugin, but ran into this bug – https://github.com/boltops-tools/terraspace/issues/73

I’ll keep pluging away. Thanks.

3

Today I learned about changing the default build cache directory here https://terraspace.cloud/docs/config/reference/ which allows me to add :NAMESPACE to the directory.

I also discovered ‘hooks.on_boot’ config option with the suggestion that this might be able to set env variables dynamically. I wonder if this is something I might be able to do to dynamically change what the NAMESPACE is to something other than the account_id?

Going to explore that idea.

4

Yeah, I think I’m flailing a bit here not really knowing enough about ruby to be confident in a direction.
I thought maybe I could introduce some monkey-patching in the hooks.on_boot ruby block to change the behavior of the aws terraspace plugins namespace method, but that didn’t do anything.

I think I might be able to write my own plugin which essentially derives from the terraspace_plugin_aws gem, but overwrites the namespace to some other method, but I’m not sure how terraspace would know how to use my plugin vs the aws one, as I think both gems would be available. It seems there some concept of plugin autoloading

There’s not much documentation or example on plugins – presumably its not yet bubbling up on the priority list.

Would love to hear from someone who might have a suggestion on how to customize the aws plugin’s namespace function. Thanks.

5

Appreciative of your updates and research. Been also thinking about how to approach this myself. Your updates have been helpful :+1:

RE: custom layering

Terraspace doesn’t support customizing how layering is performed yet. From reading this, I would like this ability added.

Thoughts

Here are thoughts on your research for now.

Recapping to help. It sounds like you’re trying to map your current structure to terraspace.

RE: your structure example

aws/team_a/qa/service/us-east-2/base.tfvars

where:

team = team_a
env = qa
account_type = service
region = us-east-2

As you’ve researched, there’s no current way to customize layering to map to your structure: team, env, account_type, region.

RE: plugin to extend namespace

Can see how it may make sense to try there. You can try to piggyback off the namespace method, and try to make it more “smart” with multiple “variables” like team, env, etc. It feels a little bit messy.

A plugin encapsulates a lot more than just layering. So a plugin is overkill.

RE: changing default build cache directory

Yup, the default build cache can be changed and controlled. The build cache directory is decoupled from the layering. While the build cache directory can be customized, it’s not enough. IE: It has no effect on layering and won’t customize it.

RE: hooks.on_boot config option

Yup. This a good spot to set env vars super early on. It’s useful for things like automatically setting AWS_PROFILE based on TS_ENV. IE: So TS_ENV=dev always points to AWS_PROFILE=dev-profile. It won’t help much with customizing layering, though. Unless, again, you squeeze multiple variables into one.

Your questions and research provide useful context. Think it may lead to solving this case. Right now, the default layering is flexible and handles most cases, but not this particular case. Customized layering is needed.

So, all that being said, don’t think it’s possible to map your current structure to terraspace until there’s the ability to customize layering. Unsure what the interface will look like yet. Think that it belongs in terraspace-core vs at the plugin-level, though. The current area of code where layering is performed is here tfvar/layer.rb Will have to think about this one a bit more. :face_with_monocle:

6

One additional thought to consider. Here’s an article from Hashicorp Terraform talking about Terraform Mono Repo vs. Multi Repo: The Great Debate

Think most people tend to think in terms of a mono repo, and it makes sense to start off that way. It so easy to think about things as one big “simple” button to press, and that’s it. The code starts to get complex, though. That’s the hidden cost. So at some point, folks break it out to multiple repos.

In the case of terraspace, this would be multiple terraspace projects. So you could consider:

  • team = team_a - maps to a terraspace project
  • env = qa - maps to TS_ENV
  • account_type = service - maps to a terraspace stack
  • region = us-east-2 - maps to AWS_REGION

Know this may not be a structure that most start with. It has some pros and cons.

The projects are decoupled. So you have to think about how to orchestrate multiple projects. If the projects are naturally decoupled, IE: the teams have been set up, so they operate independently, then there’s no orchestration, and it’s actually a pro.

But there’s likely a “core” or “platform” team that’s responsible for shared infrastructure components like VPC or Clusters. Depending on where you’re at, the share components can be managed as separate terraspace projects, and the coordination can be minimized. Some folks like the mono approach, and some like the multiple repo approach. :man_shrugging:t2:

7

To help achieve the structure, custom layering support has been added.

As a part of this, project-level layering has also been added. Docs: Project-level Layering Docs

Project-level may be enough without the need for custom layering. And custom layering handles extra edge cases.

Also, the ability to make namespace layer names be friendly has been added. Docs: Friendly Names Layering

Thinking project-level layering and custom layering should help those who prefer the mono-repo setup. Think will add some more thoughts about the tradeoffs between a mono-repo vs micro-repos setup later.

8

This sounds really great. Thanks. I will try to schedule some time to evaluate and try out this change within the week. Cheers.

9

I briefly tried out the new version with custom layering and friendly namespaces, and realized it doesn’t address another related aspect of our implementation that I neglected to bring up in this discussion.

Currently in our terragrunt mono-repo, when run we terragrunt, we do something akin to the following:

$ export AWS_PROFILE=shared-iam
$ cd teamA/qa/service-01/us-east-1/demo
$ terragrunt plan

The terragrunt.hcl file within that directory has code like the following:

locals {
  # i.e, arn:aws:iam::ACCOUNT_ID:role/ProvisionerRole
  role_arn  = << CODE_TO_GET_ROLE  >>
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = "\n# Provider for the service account\nprovider \"aws\" {\n  region = \"us-east-1\"\n  assume_role {\n    role_arn = \"${local.role_arn}\"\n  }\n}\n"
}

We use one set of AWS credentials mapped to one account, and then our terraform provider will assume a role into the account we want the project provison resources in based on what directory we’re running out of.

The trouble I’m having with terraspace is that when it builds up the cache dir and defines the aws namespace, it has no knowledge of the role that will be assumed when terraform runs, so it defines the namespace and the custom layering based off our iam-account (what we have AWS_PROFILE set to - the initial account in our role chain)

Assume we have the following accounts:

11111111 = iam-account
22222222 = teamA-qa-account
33333333 = teamA-stg-account
44444444 = teamA-prd-account
55555555 = teamB-qa-account

our terraspace app.rb:

config.build.cache_dir = ":CACHE_ROOT/:NAMESPACE/:REGION/:BUILD_DIR"

# TODO dynamically look these up
config.layering.names = {
  "111111111111":"shared-iam",
  "222222222222":"teamA-qa-service-01"
  ...
}

Our provider.rb ERB:

# Provider for the service account
provider "aws" {
  region  = <%= ENV['TS_ENV'] %>
  assume_role {
   role_arn = "arn:aws:iam::<%= lookup_account_id %>:role/ProvisionerRole"
  }
}

Our seed dir:

seed
└── tfvars
    └── stacks
        └── demo
            ├── base.tfvars
            └── teamA-qa-service-01
                └── base.tfvars

Run terrspace plan:

AWS_PROFILE=shared-iam TS_TEAM=teamA TS_ENV=qa terraspace plan demo

Cache gets generated like this:

Building .terraspace-cache/shared-iam/us-east-1/stacks/demo
...

$ cd .terraspace-cache/shared-iam/us-east-1/stacks/demo
$ ls -1
1-stacks-demo-base.auto.tfvars
provider.tf
...

Contents of provider.tf generated file:

cat provider.tf
# Provider for the service account
provider "aws" {
  region  = "us-east-1"
  assume_role {
   role_arn = "arn:aws:iam::222222222222:role/ProvisionerRole"
  }
}

Terraform is going to use the role to provision the teamA-qa-service-01 account, BUT its not picking up custom tfvars associated with the teamA-qa-service-01 account.

I think what we need is a way to customize how the aws terraspace plugin resolves the namespace.

Is that possible?

Thanks.

10

This is because Terraspace detects info like AWS account early on during the build phase, not during run/apply phase. You can use the standard AWS profile instead of the Terraform role_arn. The AWS profile switches earlier, so the same role and AWS account will be used by both terraspace and terraform.

Note, it’s a decent effort for Terraspace to parse for terraform role_arn. Will consider PRs, though. The recommendation is to use ~/.aws currently.

Docs: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html

Steps:

  1. Use aws role_arn in your .aws/config and .aws/credentials instead.
  2. Remove the role_arn in your current code

RE: Custom Layering, Complexity, Sweeping Under the Rug

If you’re going down the route of custom layering and porting things as-is, it won’t get rid of code complexity. You’re trading one set of complexities for another. Instead of a complex hierarchical path structure, there’s complexity with env vars. So:

$ export AWS_PROFILE=shared-iam
$ cd teamA/qa/service-01/us-east-1/demo
$ terragrunt plan

Becomes:

$ TEAM=teamA \
  TS_ENV=qa \
  SERVICE=service-01 \
  AWS_REGION=us-east-1 \
  AWS_PROFILE=profile1 \
  terraspace up demo

It’s actually worse to have to remember all the env vars.

This is the typical course for hierarchical mono-repo setup because they naturally result in more complex systems. There are simply more moving parts. To be clear, this doesn’t necessarily mean micro-repos are better. It depends. Try to find a good balance between the 2 approaches that fit your needs.

You can consider using one env var and using a boot hook to set the rest of them. Note, use version Terraspace v0.6.2. Made some changes:

Terraspace offers a lot of control in that you can tap into a full programming language. Example:

config/boot.rb

app = ENV['APP'] || ''
team, ts_env, service, region, profile = app.split('/').map { |x| x.blank? ? nil : x }

ENV['TEAM']        = team    || 'teamA'
ENV['TS_ENV']      = ts_env  || 'dev'
ENV['SERVICE']     = service || 'service1'
ENV['AWS_REGION']  = region  || 'us-east-1'
ENV['AWS_PROFILE'] = profile || 'dev-profile' # or logic to handle the different profiles

This “sweeps the garbage under the rug” to mask the complexity.

APP=teamB/prod/service2/us-east-2/prod-profile terraspace build demo

Generally

Consider reviewing your current structure to see if there are opportunities to remove unnecessary or little-used parts. Sometimes removing even the littlest thing can dramatically decrease complexity.

Of course, it may not be feasible to refactor because of existing statefiles which would require a major “lift-and-shift” migration. So maybe you’ll just have to use the learnings for future setups. That’s what Terraspace’s defaults encapsulate. It’s not perfect, nothing ever is, but it should hopefully account for 80-90% uses.

11

Thanks for sharing your thoughts on options for finding a path forward for terraspace for our setup.

The attractiveness of terraspace is that it could collapse our deep nested hiearchy we have today with terragrunt and the redundancy of our hcl/tfvars files and work with our existing tfstate files as they are without the need for any manipulation.

Getting to terraspace would probably make things simpler for smaller refactorings with less files to touch going forward. We might discover that with less duplication of code, it easier to see we only have a small handful of cases where we take advantage of our layering and could probably eliminate pieces. Right now its hard to see the forest through the trees.

You bring up a great idea of maybe turning the problem on its head by first defining secondary profiles in ~/.aws/config that assume our roles and use that as AWS_PROFILE, eliminating the need to specify the role in our terraform provider. I hadn’t thought of tha - we just hadn’t done that as there hadn’t been a need using pure terraform or terragrunt, but it could be a quick easy compromise to getting things working. It might even be that our boot hook could generate the entry in ~/.aws/config, or we can just have a prereq script we require running to populate the entries, wrapping calls to the AWS cli.

I will give this a try next.

I definitely like and had similar ideas to reducing the number of env variables needed. I’m getting ahead of myself but down the line there’s idea of maybe a lightweight bash script or Makefile around calls to terraform. Makefiles come to mind for its ability to run targets in parallel. That can evolve over time if and when needs arise.

Thanks again!

12

Unfortunately, I ran into a snag trying out the suggestion of pre-creating profiles in my aws-config.

I get the following error message:

Traceback (most recent call last):
  66: from HOME/.rbenv/versions/2.7.2/bin/bundle:23:in `<main>'
  65: from HOME/.rbenv/versions/2.7.2/bin/bundle:23:in `load'
  64: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/exe/bundle:37:in `<top (required)>'
  63: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors'
  62: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/exe/bundle:49:in `block in <top (required)>'
  61: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli.rb:24:in `start'
  60: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  59: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli.rb:30:in `dispatch'
  58: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  57: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
  56: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  55: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli.rb:494:in `exec'
  54: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli/exec.rb:28:in `run'
  53: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli/exec.rb:63:in `kernel_load'
  52: from HOME/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.11/lib/bundler/cli/exec.rb:63:in `load'
  51: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/bin/terraspace:23:in `<top (required)>'
  50: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/bin/terraspace:23:in `load'
  49: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/exe/terraspace:14:in `<top (required)>'
  48: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
  47: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/command.rb:59:in `dispatch'
  46: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
  45: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
  44: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
  43: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/cli.rb:53:in `build'
  42: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/builder.rb:13:in `run'
  41: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/builder.rb:77:in `clean'
  40: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/cleaner.rb:9:in `clean'
  39: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/cleaner.rb:16:in `backend_change_purge'
  38: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/cleaner/backend_change.rb:10:in `purge'
  37: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/cleaner/backend_change.rb:20:in `purge?'
  36: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/cleaner/backend_change.rb:34:in `current_backend'
  35: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/memoist-0.16.2/lib/memoist.rb:169:in `cache_dir'
  34: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/mod.rb:131:in `cache_dir'
  33: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/compiler/expander.rb:3:in `expansion'
  32: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/plugin/expander/interface.rb:40:in `expansion'
  31: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/plugin/expander/interface.rb:40:in `each'
  30: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/plugin/expander/interface.rb:41:in `block in expansion'
  29: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace-0.6.2/lib/terraspace/plugin/expander/interface.rb:72:in `var_value'
  28: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/terraspace_plugin_aws-0.3.0/lib/terraspace_plugin_aws/interfaces/expander.rb:6:in `account'
  27: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/memoist-0.16.2/lib/memoist.rb:169:in `account'
  26: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws_data-0.1.1/lib/aws_data.rb:59:in `account'
  25: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/memoist-0.16.2/lib/memoist.rb:169:in `sts'
  24: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws_data-0.1.1/lib/aws_data.rb:88:in `sts'
  23: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/base.rb:102:in `new'
  22: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-sts/client.rb:332:in `initialize'
  21: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/base.rb:22:in `initialize'
  20: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/base.rb:65:in `build_config'
  19: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:152:in `build!'
  18: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:179:in `apply_defaults'
  17: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `resolve'
  16: from HOME/.rbenv/versions/2.7.2/lib/ruby/2.7.0/set.rb:328:in `each'
  15: from HOME/.rbenv/versions/2.7.2/lib/ruby/2.7.0/set.rb:328:in `each_key'
  14: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `block in resolve'
  13: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:207:in `value_at'
  12: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:214:in `resolve_defaults'
  11: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each'
  10: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each'
   9: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:215:in `block in resolve_defaults'
   8: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:72:in `call'
   7: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/credentials_configuration.rb:70:in `block in <class:CredentialsConfiguration>'
   6: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/credential_provider_chain.rb:12:in `resolve'
   5: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/credential_provider_chain.rb:12:in `each'
   4: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/credential_provider_chain.rb:13:in `block in resolve'
   3: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/credential_provider_chain.rb:139:in `assume_role_credentials'
   2: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/credential_provider_chain.rb:172:in `assume_role_with_profile'
   1: from HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/shared_config.rb:119:in `assume_role_credentials_from_config'
HOME/bld/scratch/terraspace/iac/devops-iac/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.112.1/lib/aws-sdk-core/shared_config.rb:219:in `assume_role_from_profile': Profile teamA-qa-service-01-myrole has a role_arn, and source_profile, but the source_profile does not have credentials.

If I try to run aws CLI directly using it, it works fine:

$ AWS_PROFILE=teamA-qa-service-01-myrole aws sts get-caller-identity
{
    "UserId": "AAAAAAAA:botocore-session-1614980795",
    "Account": "222222222222",
    "Arn": "arn:aws:sts::222222222222:assumed-role/MyRole/botocore-session-1614980795"
}

This looks to be related to this issue I found googling – https://github.com/newcontext-oss/kitchen-terraform/issues/368

Maybe I can work around this by figuring out how to update the aws-sdk library being used. Our authentication is a bit complicated involving role_chaining and mfa configuration using aws-vault as an external credential_process. I wonder if the ruby library is too old to handle it.

I will continue to poke around. Cheers.

13

Interesting, can tell from the stack trace that aws-sdk-core-3.112.1 is being used. It’s the current latest version of the aws-sdk-core:

$ gem search -e -r aws-sdk-core
aws-sdk-core (3.112.1)

So don’t think upgrading will help here.

RE: Our authentication is a bit complicated involving role_chaining and mfa configuration using aws-vault as an external credential_process.

Guessing it’s probably the MFA. Thinking that’s probably why the original setup calls export AWS_PROFILE=shared-iam and then uses terraform role_arn. So you answer the MFA token prompt outside of terraform first. Then terraform role_arn is able to assume the role.

Have dealt with MFA before and wrote another tool to help with it: aws-mfa-secure. It’s integrated to other boltops-tools already, but not yet for terraspace. Even though it’s not integrated, you can easily use it as a standalone tool.

One feature of the tool is aws-mfa-secure exports. It generates short-lived AWS_* credentials that you can export to the current terminal shell. You’ll do something like this:

$ eval `aws-mfa-secure exports`
Please provide your MFA code: 123456

See the boltops-tools/aws-mfa-secure README for more details.

After providing your MFA token, the AWS_* env vars should be set. Both the AWS SDKs and Terraform AWS provider will treat and respect AWS_* env vars as the highest precedence. So it should work. Note, haven’t tested it specifically for your auth chain with assumed roles and vault. So a little unsure if it’ll work. Letting you know in case, though. In theory, you can work around MFA with short-lived temp AWS_* creds before calling terraspace and terraform.

RE: I definitely like and had similar ideas to reducing the number of env variables needed. I’m getting ahead of myself but down the line there’s idea of maybe a lightweight bash script or Makefile around calls to terraform. Makefiles come to mind for its ability to run targets in parallel. That can evolve over time if and when needs arise.

Think that’s a good approach :+1: Sometimes, folks tend to try to fit everything into one tool to come up with a “god” command. It may be impossible to ever fit the god criteria. As the linux saying goes, “Use the right tool for the right job”. Did an interview with Anton B, he explains it pretty clearly: “We still have makefiles, we still have shell”. Here’s the video at the specific time: https://youtu.be/J_-XPfFlsbU?t=6420

14

Prior to seeing your response, I came up with a solution, or at least a way forward that works for now to continue with a POC – It’s to call the AWS Secure Token Service from within the terraspace boot hook to extract some temporary creds. I can then set those in the environment for use during the terraspace config phase. Code looks something like this (pardon my ruby…):

# config/boot.rb

require 'aws-sdk-sts'
require 'json'

# dynamically build up a role to assume and assume it
# for the terraspace config phase

region = ENV['AWS_REGION']  || 'us-east-1'
team   = ENV['TS_TEAM']
type   = ENV['TS_TYPE']  || 'service'
env    = ENV['TS_ENV'] || 'qa'
count  = Integer(ENV['TS_COUNT'] || 1)

account_alias = ENV['TS_ALIAS'] || "#{team}-#{env}-#{type}-#{sprintf('%02d',count)}"

# TODO - dynamically look this information up. For now have a local config
account_data = JSON.parse(File.read('config/account_data.json'))
account_id = account_data.find { |act| act['alias'] == account_alias }['account_id']

role="arn:aws:iam::#{account_id}:role/MyRole"

ENV['AWS_REGION'] = region

params = {
  role_arn: role,
  role_session_name: ENV['USER'],
}

res = Aws::STS::Client.new.assume_role(params)

ENV['AWS_ACCESS_KEY_ID'] = res.credentials.access_key_id
ENV['AWS_SECRET_ACCESS_KEY'] = res.credentials.secret_access_key
ENV['AWS_SESSION_TOKEN'] = res.credentials.session_token

# remove reference to outer profile -- use above creds instead
ENV.delete('AWS_PROFILE')

I gave that a try and it works! Terraspace namespace gets set to the my assume_role account, and the friendly name feature you add worked nicely to replace it with my account alias.

With this authentication piece out of the way, I look forward next week to experimenting more with building out our various tfvars layers and see if we can iterate over the combination of teams/env/accounts/regions and and get our plans running clean via terraspace.

Thanks again for continued help and suggestions. Very clever framework. Cheers.

P.S - I’m curious how your aws-mfa-secure tool works, and will take a look, but most likely we’d continue using aws-vault which works well for us. It’s handy to store creds in desktop keychain, popup modal window for entering MFA, and caching session creds as well. We also have some workflows using SAML and maybe eventually AWS SSO. It’ll be interesting to see how that plays out with terraspace but it seems with the boot hook there’s probably a way through any hurdles.

1 Like
15

Awesome! Glad that you worked it out. With all that vault does for you:

desktop keychain, popup modal window for entering MFA, and caching session creds as well

It’s sounds like a better solution and you’re already using it. It’s smart to stick to what works! aws-mfa-secure doesn’t do much and wrote it only because needed it for a project at the time to get around MFA.

RE: Thanks again for continued help and suggestions. Very clever framework. Cheers.

:grin:

16

I was slightly premature in declaring victory friday night getting the custom boot.rb script to work. It did work in that during the terraspace build phase, terraspace correctly used the correct account as the terraspace namespace, but the problem was now pushed to the terraform execution phase. I have some terraform provider configuration which doesn’t work don’t work when AWS_ACCESS_KEY_ID is set.

What was happening was the project fail failing to plan when run through terraspace, but if I changed into the cache-dir and ran terraform directly, the project would plan clean

Luckily I did find a workaround in terraspace via terraform hooks! I can simply undo the env variables changes made in the boot hook.

boot.rb:

# config/boot.rb

require 'aws-sdk-sts'
require 'json'

# dynamically build up a role to assume and assume it
# for the terraspace config phase

region = ENV['AWS_REGION']  || 'us-east-1'
team   = ENV['TS_TEAM']
type   = ENV['TS_TYPE']  || 'service'
env    = ENV['TS_ENV'] || 'qa'
count  = Integer(ENV['TS_COUNT'] || 1)

account_alias = ENV['TS_ALIAS'] || "#{team}-#{env}-#{type}-#{sprintf('%02d',count)}"

# TODO - dynamically look this information up. For now have a local config
account_data = JSON.parse(File.read('config/account_data.json'))
account_id = account_data.find { |act| act['alias'] == account_alias }['account_id']

role="arn:aws:iam::#{account_id}:role/MyRole"

ENV['AWS_REGION'] = region

params = {
  role_arn: role,
  role_session_name: ENV['USER'],
}

res = Aws::STS::Client.new.assume_role(params)

ENV['AWS_ACCESS_KEY_ID'] = res.credentials.access_key_id
ENV['AWS_SECRET_ACCESS_KEY'] = res.credentials.secret_access_key
ENV['AWS_SESSION_TOKEN'] = res.credentials.session_token

# save this, you'll need it later when executing terraform
ENV['ORIG_AWS_PROFILE'] = ENV['AWS_PROFILE']

# remove reference to outer profile -- use above creds instead
ENV.delete('AWS_PROFILE')

config/hooks/terraform.rb:

class EnvOverride
  def call
	ENV.delete('AWS_ACCESS_KEY_ID')
	ENV.delete('AWS_SECRET_ACCESS_KEY')
	ENV.delete('AWS_SESSION_TOKEN')

	ENV['AWS_PROFILE'] = ENV['ORIG_AWS_PROFILE']
  end
end

before("init", "plan", "apply",
  execute: EnvOverride
)

Logic and Ruby code can be cleaned up. I might need to restore the changes in an after hook, but happy to see this working. Hooks for the win!

cheers!

1 Like