Terraspace Test: Test for failures

Terraspace
1

Hey,

I have a question about the testing framework. We’ve been researching rspec and managed to set up a nice little testing framework that works for our use-cases. The only thing missing right now is the ability to test for failures. As none of us are ruby developers we mainly work by googling, but in this case we can’t seem to find a working solution.

Basically the idea is to test that specific inputs (a specific tfvars file for example) contains values that trigger validation checks in terraform of some kind. Think validation of a variable for emails that checks if there is an @ in the value, but we specify a value without. The expectation would be that “terraspace.up” fails and that we can test for this and continue.

Unfortunately it seems that I can’t find a way to actually continue after terraspace.up runs into an issue, even if we have other tests afterwards that would run another terraspace.up with working input.

From what I understand, terraspace simply exits with 1 if there was any terraform failure in the plan process. rspec-terraspace doesn’t do anything specific to handle failures either, so from my understanding of ruby we should receive a SystemExit exception in our main_spec.rb file.

I tried to encapsulate it with “begin/rescue”, but that doesn’t seem to help - the failed test may return “success”, but no other test will be evaluated afterwards.

this is a small snippet of what I am trying at the moment:

let(:status) do
    begin
      terraspace.up(@module_name)
      true
    rescue
      false
    end
  end

it "deployment" do
    expect(status).to eq test[:expect_deployment]
  end

This will detect a failure, but will still not continue to other ‘it’ further down the file. If we test for success (and terraspace.up doesn’t fail), other examples are tested as usual.

Is there anything we are missing or is there simply no way for us to test for failure with the system? We feel it’s a very important missing feature if that would be the case.

Any help is much appreciated!

Emanuel

2

Interesting. Using let(:status) should call terraspace.up lazily within the it "deployment" do block so that should work. Also, since you’re wrapping it with a catchall rescue so it should catch the SystemExit exception, it should continue to other it tests.

Wondering what the full test file looks like. Maybe there’s something in the before(:all) that might be screwing it up.

Note: Also, maybe try changing before(:all) to before(:each). Believe, that way the code in there fails it still runs the rest of the it tests. It’s a guess, but that might be what’s happening.

Note: Currently, terraspace raises a SystemExit, ie: exit(1), on this fail runs. Wondering if should make it raise a Terraspace::Error subclass exception instead. So it’s a little easier to target the exception to catch.

3

It looks to me that the exit(1) escapes the rescue in some way. It clearly fails the entire test process even if I catch it. The one test itself works as expected but the process still ends with 1 directly after.

I can try to give you a full example to test with, but it will take a couple of days.