Intermittent output helper

Terraspace
, ,
#1

Where to begin…

I have tagged this topic up with multiple tags because I feel all of them are relevant.

Has anyone else experienced the output() helper function in the tfvars return the mock value intermittently when the value from the terraform state should be available?

I am deeply concerned about this problem we are experiencing on one of our projects, we use this output() helper function to fetch a KMS key in another stack for some ec2, databases, etc

Example tfvars:
kms_key_id = <%= output(‘kms.key_id’, mock: ‘arn:aws:kms:::key/mock-key2’) %>

However, intermittently we see terraspace fail to get the real value from state and rather then fail, just default to the mock value. When using terraspace all up this leads to terraform seeing the kms key as changed and forces a replace, deleting the resources and failing because the mock isn’t valid.

I think this is dangerous because the mock value should never be used in a deploy/up scenario, maybe only for planning when a stack depends on another stacks output that hasn’t been created yet.

I am not really sure why this is happening.

I think there is few ways this can be solved and would like to gather feedback.

  • Terraspace ouput() helper function has configurable number of retries to attempt to get the value from state.
  • Terraspace has an option to fail a deploy if it detects a mock in the compiled terraform code.

Currently this problem makes things very very difficult for us because we can’t trust terraspace all up because it may destroy some resources we really don’t want it too.

As mentioned before, I would be really interested to hear back from terraspace veterans about this problem.

#2

This is a bummer. Yup. Will have to dig into this one.

At least, from the sounds of it, it’s reproducible. Just need create an example project that a KMS key and use the output helper. If you have an example, that’ll be appreciate. If not, no sweat.

RE: * I think there is few ways this can be solved and would like to gather feedback. Terraspace output() helper function has configurable number of retries to attempt to get the value from state.

Sure.

RE: Terraspace has an option to fail a deploy if it detects a mock in the compiled terraform code.

Yup. It should fail hard.

So generally speaking, think:

Hoping to get to it as part of this major version :crossed_fingers: Unsure if will get to it though :face_with_monocle::hourglass_flowing_sand: Happy to take a look PRs. Of course no sweat either way :+1:

1 Like
#3

Hi @tung, sadly my ruby knowledge leaves a lot to be desired, else I would.

I have import information to add. I managed to capture the problem (though its intermittent).Here is what I found.

The temporary state files that terraspace generated in /tmp/terraspace do contain the values.

(I had to change the directory because of CICD)

[Album] imgur.com

So this means terraform is getting the state correctly, however, it appears terraspace is not always loading it correctly. This is inconsistent across the deployment when other stacks are using the same outputs.

[Album] Imgur: The magic of the Internet

You can see that the output helper is setup correctly:

(See next comment: ext1)

I believe there is some race condition occurring when terraspace is pulling the state vs deploying a stack.

#4

@tung not sure why this was ignored/flagged.

#5

I have created an official bug report for this.

#6

It’s the auto-moderation system. Guess it detects similar text and mis-flags :man_facepalming:t2: For the most part it works well, this time not so much. Unflagged it.

As for the report. Thanks spending the time and providing the details. Sure it’ll help when have time to dig into this. Also, glad you were able to figure out a workaround for now.