Pass GCP credentials to TFC

Terraspace
1

Is there a recommended way to configure a Terraspace project to pass credentials to Terraform Cloud workspaces? At the moment, our team authenticates with gcloud auth application-default login on their laptops.

Once we run terraspace up {stack} the workspace gets created, but the plans fail with the following:

Error: Attempted to load application default credentials since neither `credentials` nor `access_token` was set in the provider block.  No credentials loaded. To use your gcloud credentials, run 'gcloud auth application-default login'.  Original error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

I don’t think it would be wise to send a developer’s JSON key to Terraform Cloud, as manual runs could then be applied under someone else’s credentials.

We could create a service account for each user, but then we run into the aforementioned issue as the service account’s key would be set on the workspace using vars.rb as well.

Using a single service account is not viable, as Google does not provide a way to label each JSON key. So if a team member leaves, we can’t remove just their key to revoke access.

I love how Terraspace manages things, but I feel like there is a layer of authentication here that I’m just not understanding, or could be missing.

2

Dug into this. This is a Terraform Cloud and Terraform Google Provider, not a Terraspace thing. To help explain, let’s take terraspace out of the picture for a moment.

RE: At the moment, our team authenticates with gcloud auth application-default login on their laptops.

Can see how terraform apply works on the laptop, but don’t see how it will work on TFC. At least, with my understanding.

Local Laptop vs Remote TFC

When using the gcloud auth application-default login auth approach, the terraform apply is successfull because running:

$ gcloud auth application-default login
# creates ~/.config/gcloud/application_default_credentials.json

It is key to note that this all happens locally on your laptop. Terraform’s Google provider is smart enough to use the application_default_credentials.json file for authentication.

When TFC is used, the terraform apply happens in a remote container that TFC spins up. This means the application_default_credentials.json won’t be on the remote machine. For likely security reasons, TFC only allows some control over the remote machine. It mainly comes in the form of env vars.

Believe this is why Terraform recommends using google service accounts instead of gcloud auth application-default login. And terraform print out warning if GOOGLE_AUTH_SUPPRESS_CREDENTIALS_WARNINGS is not set.

Click to See Output of terraspace up 
$ terraspace up demo
Building .terraspace-cache/us-central1/dev/stacks/demo
Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/. To suppress this message, set the GOOGLE_AUTH_SUPPRESS_CREDENTIALS_WARNINGS environment variable.
Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/. To suppress this message, set the GOOGLE_AUTH_SUPPRESS_CREDENTIALS_WARNINGS environment variable.
Built in .terraspace-cache/us-central1/dev/stacks/demo
Current directory: .terraspace-cache/us-central1/dev/stacks/demo
=> terraform apply -input=false

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # random_pet.this will be created
  + resource "random_pet" "this" {
      + id        = (known after apply)
      + length    = 2
      + separator = "-"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes

random_pet.this: Creating...
random_pet.this: Creation complete after 0s [id=shining-hen]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Time took: 40s
$

Resummarizing some of what you’re saying:

  1. It’s unwise to send a developer’s JSON key to Terraform Cloud, as manual runs could then be applied under someone else’s credentials.
  2. Creating service account for each user could mean runs would be applied under someone else’s credentials again.
  3. Using a single service account is not viable, so you can’t revoke access to that specific person.

So the devil-is-in-the-details. Think they’re all valid points. Let’s take a second look at #3.

Variables set on TFC are shared for the workspace. TFC is not really designed to mimic separate individual laptops. It can be thought of as a “shared/single” deploy box. The workspace itself is designed to use shared env variables like GOOGLE_CREDENTIALS.

TFC attempts to address fine-grain access control with the TFC Teams feature instead. You set up teams, assign those teams to workspaces, and managed access control that way. If someone leaves, you revoke their TFC team access. That’s likely going to be the recommendation from TFC to managed access control.

The GOOGLE_CREDENTIALS itself remains the same. The TFC team may say that the GOOGLE_CREDENTIALS is set as a TFC env variable and never passes through the developer’s laptop. So it’s secure. As you mentioned, Terraspace offers the vars.json ability to conveniently set variables. In this case, it could pass through the developer’s laptop if they have permission. There are trade-offs. You can set the variables manually instead if that’s a concern. It’s up to you. Depending on your security posture, you may also want to rotate the GOOGLE_CREDENTIALS.

Some people like the way TFC works, some people do not. :man_shrugging:t2: Ultimately, it’s a Terraform and TFC thing here not a Terrspace thing.

3

Hey @tung,

Super awesome deep dive into what is going on with TFC, and I think my message may have come across more panic toned which caused confusion. However, what you just described is exactly what we already have using a custom monorepo. This is our current structure:

infrastructure
├── modules
│   ├── helm-charts
│   ├── kubernetes-engine
│   ├── memorystore
│   ├── service-account
│   └── workspace
├── stacks
│   ├── cloudtasks-telematics-dev-us-central1
│   ├── dns
│   ├── folders
│   ├── gcr
│   ├── gke-apps-dev-us-central1
│   ├── gke-vault-dev-us-central1
│   ├── memorystore-telematics-dev-us-central1
│   └── vpc-dev
└── tfc-workspaces
    ├── backend.tf
    ├── main.tf
    ├── providers.tf
    └── variables.tf

The tfc-workspaces project is solely responsible for creating tfe_workspace resources and then attaching environment variables to each workspace we create. A workspace is a directory in the stacks folder. The environment variables are stored within the TFC workspace called tfc-workspaces so we only have to set them in one place, and then they get replicated out to the other workspaces using some module magic under the hood.

This approach has allowed us to create a “Master Service Account” that has a JSON key created. Then we are using Teams + Sentinel on TFC to ensure that someone is only running what they’re allowed to run.

We are looking into Terraspace because, as you can tell by the directory structure in stacks, we are going to end up with duplicated code as we build out stage, and prod, environments:

  • gke-apps-dev-us-central1
  • gke-apps-stage-us-central1
  • gke-apps-prod-us-central1

As you can probably guess, this is where we want Terraspace to come in and save the day and help us DRY our stacks ups. We also have some scaffolding issues for Disaster Recovery where workspaces try to build before other workspaces are ready, and built. This seems to also be solve-able with Terraspace and it’s dependency graphing via lookups.

The problem we are facing right now, is how do we replace our tfc-workspaces project and its scaffolding of other workspaces to set the, for example, GOOGLE_CREDENTIALS envvar on the workspace?

We don’t want to pass out our master service account’s JSON key to our developers to store locally on their laptop.

Where this is a problem is if they’re working on a new stack, and run terraspace up stack, GOOGLE_CREDENTIALS will not be available on the remote TFC workspace when it runs. The developer will not have this value either.

I’m wondering if we had a credentials workspace, that has a sensitive output called google_credentials, if we can pass that into the new stack’s Google provider’s credentials config via remote terraform_remote_state data source. Just a random thought while typing this out… I don’t know if it is viable, or if you might have a better idea of how to go about this.

Side note: Would be nice if TFC offered Organization-level Env Vars that are injected to every workspace.

4

I think I just realized our expected workflow is not compatible with Terraspace. We are planning on mirroring GKE clusters to different regions, which under our current project would look like this:

  • gke-apps-prod-us-central1
  • gke-apps-prod-us-west1

I would then expect to have the following stacks in Terraspace:

app/stacks
├── gke-apps-us-central1
└── gke-apps-us-west1

When built, I would expect to see:

❯ tree -L 2 .terraspace-cache
.terraspace-cache
├── us-central1
    └── prod
└── us-west1
    └── prod

Since the provider is defined in config/terraform/provider.tf, we can’t have individual providers defined down in the actual stack; which makes the above not work.

After reading this part of the documentation, with Terraspace we need to have two Terraspace projects, or have two providers defined that are swapped out based on an envvar at build time.

This wont work for us, in the way we want to with automation for DR, as we have some stuff above the clusters that need to come online after both regions are built and ready to go.

What would work is having multiple Terraspace projects for each region, and then a Terraspace project for the global pieces that need each region in place. Then for DR we just bring up things in chunks:

  1. us-central1
  2. us-west1
  3. global
5

RE: I think my message may have come across more panic toned which caused confusion.

Naw, didn’t interpret a panic tone. We’re all just figuring things out. On the flip-side, hopefully, I didn’t come off too strongly worded. The internet has its ways of not relaying the tone of messages properly, sometimes :man_facepalming:t2: Terraspace can be a fit, and sometimes it may not be. That’s ok. Am uncertain here if it’s a fit. Think Terraspace can definitely help here, but is not a cure-all.

RE: Local Laptop vs Remote TFC

Wondering which TFC workflow you’re using:

  1. The CLI-driven Workflow
  2. The VCS-driven Workflow

Unsure which workflow is being used. If it’s the CLI-drven workflow, then think gcloud auth application-default login should work. At least, in the test above was using it. It’s on the “Click to See Output of terraspace up”. It’s the VCS-workflow where the remote machine won’t have the application_default_credentials.json and terraform apply won’t work. :face_with_monocle:

RE: The problem we are facing right now, is how do we replace our tfc-workspaces project and its scaffolding of other workspaces to set the, for example, GOOGLE_CREDENTIALS envvar on the workspace?

It’s quite interesting and novel how you guys are approaching this. So it sounds like the tfc-workspaces is a “master” terraform project that is:

  1. responsible for generating google credentials as well as
  2. scaffolding the other terraform projects.

If am understanding this right, at a high-level, you’ll probably be better off keeping some form of “master” or “orchestration” script to tie things together.

Interestingly, since Terraspace actually has access to a full programming language, Ruby, at your fingertips, you could tap into it and have terraspace to take over the full master script, but it’ll likely grow complex and get messy. So probably wouldn’t take that approach.

Would probably organize things into separate terraspace projects for each team and have each team responsible for that. There will be some shared projects and teams, and there are trade-offs. Here are also some thoughts on this: Customized layering support?

RE: I’m wondering if we had a credentials workspace, that has a sensitive output called google_credentials, if we can pass that into the new stack’s Google provider’s credentials config via remote terraform_remote_state data source. Just a random thought while typing this out… I don’t know if it is viable, or if you might have a better idea of how to go about this.

Here’s where am wondering if you are using the CLI-driven workflow or VCS-driven workflow. Sounds like you are maybe also using VCS-driven for some things and the CLI-driven for other things. In this case, the terraform_remote_state may work. I usually don’t use terraform_remote_state because it feels like it couples things too much. But can’t think of anything better at the moment. :face_with_monocle:

RE: Side note: Would be nice if TFC offered Organization-level Env Vars that are injected to every workspace.

Yes, that would definitely be nice. Got some longer-term goals for this, but it’s a few months out, so it won’t be of immediate help to you.

RE: Since the provider is defined in config/terraform/provider.tf, we can’t have individual providers defined down in the actual stack; which makes the above not work.

The devil-is-in-the-details. Also, those docs probably need to be more clear. https://terraspace.cloud/docs/misc/multiple-providers/ The docs are talking about different cloud providers like AWS and Google with the same code, it’s possible with Terraspace it just feels a little too messy. Going multiple regions with the same cloud is straightforward and clean already.

RE: What would work is having multiple Terraspace projects for each region, and then a Terraspace project for the global pieces that need each region in place. Then for DR we just bring up things in chunks: us-central1, us-west1, global

Think you’re close to nailing it. Though, think it’s more accurate to say you need to “call” terraspace multiple times, instead of having multiple projects. Something like:

TS_ENV=global terraspace all up
TS_ENV=dev GOOGLE_REGION=us-central1 terraspace all up
TS_ENV=dev GOOGLE_REGION=us-west1 terraspace all up

Same code. Different permutation handled by different invocations of terraspace all up. Hence, this is where would consider a “master” orchestration script. Leverage Terraspace for higher-level preassembled infrastructure. But still have an even-higher level orchestration to handle these more specific workflows.

This somewhat reminds me of Handling multiple providers/accounts/roles/regions Taking terraspace out of the picture for a moment and focusing only on terraform itself. This person would like to invoke terraform apply only once and have it switch to 5 different AWS_PROFILES in one-go. Terraform itself won’t be able to do that in one-go. A master orchestration/wrapper script is probably a better approach.

6

We are using the VCS-Driven workflow for everything, but there have been talks about being okay dropping down to CLI-Driven workflows. From how you’re explaining this, it may sound like TFC respects the developer’s application-default credentials on CLI-Driven workflows. This is interesting, as I figured TFC being a remote build box would not pass that from the local machine to the remote machine. I will test this out and report back.

If the above works, then then requirement of setting GOOGLE_CREDENTIALS on each workspace in TFC is no longer required. This would be due to the fact that each developer now becomes the service account, and then we can manage access to what they have access to do via IAM.

In a DR scenario, there would have to be “owner” level users who would bring up their respective stacks.

I’m curious about the TS_ENV=global configuration though. Assume we want to bring up a GKE cluster when TS_ENV=dev, but not when TS_ENV=global. Do we just wrap every file in the GKE stack with conditions based on the envvar to prevent those TF resources from being defined? Or is there a solution to where we can say “ignore this stack when TF_ENV=global”?

Edit: I just stumbled upon the documentation for Multiple Provider Configurations within Terraform itself. This might help solve a major chunk of the multi-region stuff.

7

RE: This is interesting, as I figured TFC being a remote build box would not pass that from the local machine to the remote machine. I will test this out and report back.

It’s good that you double-checked. I tested incorrectly. Think was reusing the TFC project which already had the GOOGLE_CREDENTIALS set from previous runs. :man_facepalming:t2: Using the CLI-workflow doesn’t magically somehow pass the application_default_credentials.json. For TFC, you pretty much have to set GOOGLE_CREDENTIALS.

RE: TS_ENV=global vs TS_ENV=dev and ignore stacks

One way to handle this is to set the config.all.ignore_stacks setting:

config/env/dev.rb

Terraspace.configure do |config|
  config.all.ignore_stacks = [“stack1”, “stack2”]
end

config/env/global.rb

Terraspace.configure do |config|
  config.all.ignore_stacks = [“stack3”, “stack4”]
end

Docs: https://terraspace.cloud/docs/config/reference/

Note, there are ways to remove the duplication for dev.rb and prod.rb with ruby. Just explaining the concept for now.

This post discusses it Prevent modules to be deployed in one environment

RE: I just stumbled upon the documentation for Multiple Provider Configurations within Terraform itself. This might help solve a major chunk of the multi-region stuff.

Yup, it’s possible to use provider aliases for multi-regions. In that case, the GOOGLE_REGION env var won’t matter because the regions are more “hardcoded”. There are trade-offs worth noting. While it’s pretty cool that both regions can be deployed with a single terraform apply, they’re coupled. So if you deploy 1 region, you must deploy both. If one region is down, the terraform apply would fail. You could use a variable to control the regions, but the code gets more complicated.

Terraspace allows you to deploy to each region independently. Generally, prefer the ability to be able to deploy to each region independently. And to have a higher-level script to deploy to both at the same time. So you get the best of both worlds. Ultimately, it’s up to you on which approach you think is a better fit for your needs. Often, the quickest way to reach a decision is to build a minimal project and try it out.

8

Released a few things that are relevant:

This allows further Terraspace customizations. Particularly for mono-repo setups. Think will add some more thoughts about the tradeoffs between a mono-repo vs micro-repos setup later.

Also see again this other forum thread, it’s got some relevant info: Customized layering support?

1 Like