Terraspace execution timeout

jalam · February 18, 2022, 3:55am

Hi Tung and BoltOps Community,

I am trying to deploy an RDS instance, which takes a while to provision. When I ran my code using Terraspace (I used terraspace all up in this instance btw), it timed out, and I suspect the timeout message below is from Terraspace*. I.e.

/var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:132:in `eof': execution expired (Timeout::Error)
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:132:in `block in all_eof?'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:132:in `each'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:132:in `find'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:132:in `all_eof?'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:50:in `block in handle_streams'
 	from /usr/lib/ruby/2.7.0/timeout.rb:110:in `timeout'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:48:in `handle_streams'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:37:in `block in popen3'
 	from /usr/lib/ruby/2.7.0/open3.rb:219:in `popen_run'
 	from /usr/lib/ruby/2.7.0/open3.rb:101:in `popen3'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:36:in `popen3'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:26:in `shell'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/shell.rb:17:in `run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/terraform/runner.rb:51:in `block in terraform'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/hooks/builder.rb:25:in `run_hooks'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/hooks/concern.rb:6:in `run_hooks'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/terraform/runner.rb:48:in `terraform'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/terraform/runner.rb:15:in `block in run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/terraform/runner.rb:89:in `time_took'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/terraform/runner.rb:14:in `run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/cli/commander.rb:11:in `run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/cli/up.rb:11:in `run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:127:in `run_terraspace'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:62:in `deploy_stack'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:46:in `block (3 levels) in deploy_batch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:45:in `fork'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:45:in `block (2 levels) in deploy_batch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:43:in `each'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:43:in `block in deploy_batch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:42:in `each'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:42:in `each_slice'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:42:in `deploy_batch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:35:in `block in deploy_batches'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:33:in `each'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:33:in `each_with_index'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:33:in `deploy_batches'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:16:in `block in run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:185:in `time_took'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/all/runner.rb:12:in `run'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/cli/all.rb:62:in `up'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/command.rb:61:in `dispatch'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/command.rb:61:in `dispatch'
 	from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/lib/terraspace/cli/concern.rb:65:in `start'
 	from /var/lib/gems/2.7.0/gems/terraspace-1.1.3/exe/terraspace:14:in `<top (required)>'
 	from /usr/local/bin/terraspace:23:in `load'
 	from /usr/local/bin/terraspace:23:in `<main>'

Is there a timeout for Terraspace execution, and is there a way to increase the timeout for the execution?

*Although I suspect the timeout message above is related Terraspace, I think it is due to a terraform bug that might’ve cause Terraspace to hit the timeout in the first place… Ref: https://www.reddit.com/r/Terraform/comments/ssayvp/aws_provider_40_hidden_timeout_bug/

Many thanks,
James

jalam · February 21, 2022, 5:14am

Hello,

Just confirming that there was indeed a bug with the aws provider in 4.0.0 and is now addressed in 4.2.0. i.e.

Reference: Changelog for 4.2.0 (February 18, 2022)

BUG FIXES:

* provider: Credentials with expiry, such as assuming a role, would not renew. ([#23282](https://github.com/hashicorp/terraform-provider-aws/issues/23282))

However, still think that the message in earlier post is due to hitting a default terraspace timeout (and would be interested if we are able to change/increase this)

Many thanks,
James

tung · February 22, 2022, 7:36pm

The terraspace default timeout waiting for terraform output is 1hr.

github.com

boltops-tools/terraspace/blob/404f179fa7ea44f0ba9fcbf07ce67fdb6ffd5f9b/lib/terraspace/shell.rb#L44


def popen3(env)
  Open3.popen3(env, @command, chdir: @mod.cache_dir) do |stdin, stdout, stderr, wait_thread|
    handle_streams(stdin, stdout, stderr)
    status = wait_thread.value.exitstatus
    exit_status(status)
  end
end
BLOCK_SIZE = Integer(ENV['TS_BUFFER_BLOCK_SIZE'] || 102400)
BUFFER_TIMEOUT = Integer(ENV['TS_BUFFER_TIMEOUT'] || 3600) # 3600s = 1h
def handle_streams(stdin, stdout, stderr)
  # note: t=0 and t=nil means no timeout. See: https://bit.ly/2PURlCX
  t = BUFFER_TIMEOUT.to_i unless BUFFER_TIMEOUT.nil?
  Timeout::timeout(t) do
    files = [stdout, stderr]
    until all_eof?(files) do
      ready = IO.select(files)
      next unless ready
      readable = ready[0]

It can be adjusted with TS_BUFFER_TIMEOUT env var, though that may change

jalam · February 22, 2022, 10:55pm

Hi Tung,

That’s great, thanks for confirming and the info

Many thanks,
James