LocalStack & Terraspace

Terraspace
#1

Hi Community,

How do I configure my app.rb file to correctly create and connect to S3 using LocalStack as my testing platform? Thank you.

I have been testing with the following configs:

require 'aws-sdk-s3'
    region = "eu-west-1"
#    Aws.config.update(
#      endpoint:  'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
#      access_key_id: 'fake', # update with localstack credentials
#      secret_access_key: 'fake', # update with localstack credentials
#      region: region,
#      force_path_style: true, # Enable 'force_path_style' => true, if bucket name is non DNS compliant
#    )

#    Aws.config.update(
#        endpoint:  'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
#        credentials: Aws::Credentials.new('fake', 'fake'),
#        region: region,
#        force_path_style: true,
#      )


#    # Create an S3 client with custom endpoint and force path style
s3 = Aws::S3::Client.new(
    endpoint:  'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
    region: region,
    access_key_id: 'fake', # update with localstack credentials
    secret_access_key: 'fake', # update with localstack credentials
    force_path_style: true  # Enable force path style
  )
  # Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
  config.logger.level = :info
  # copy_modules setting introduced 2.2.5 to speed up terraspace build
  # See: https://terraspace.cloud/docs/config/reference
  config.build.copy_modules = true
  #config.build.cache_dir	= ":APP/:ROLE/:ENV/:BUILD_DIR" # without :REGION
  #endpoint: 'http://localhost:4572',
  #credentials: Aws::Credentials.new('fake', 'fake'),
  #region: 'us-east-1',
  #force_path_style: true,
end
#2

I am making progress by doing the following:

aws config

[profile localstack-dev]
endpoint_url = http://*.localgd:4566
region = eu-west-1
output = json

Terraspace

infra → config → provider.tf

provider "aws" {
  region                      = "eu-west-1"
  access_key                  = "fake"
  secret_key                  = "fake"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  s3_force_path_style         = true

  endpoints {
    apigateway      = "http://localhost:4566"
    apigatewayv2    = "http://localhost:4566"
    cloudformation  = "http://localhost:4566"
    cloudwatch      = "http://localhost:4566"
    cognitoidp      = "http://localhost:4566"
    cognitosync     = "http://localhost:4566"
    cognitoidentity = "http://localhost:4566"
    dynamodb        = "http://localhost:4566"
    ec2             = "http://localhost:4566"
    es              = "http://localhost:4566"
    elasticache     = "http://localhost:4566"
    firehose        = "http://localhost:4566"
    glue            = "http://localhost:4566"
    iam             = "http://localhost:4566"
    kinesis         = "http://localhost:4566"
    lambda          = "http://localhost:4566"
    rds             = "http://localhost:4566"
    redshift        = "http://localhost:4566"
    route53         = "http://localhost:4566"
    s3              = "http://*.local.gd:4566"
    secretsmanager  = "http://localhost:4566"
    ses             = "http://localhost:4566"
    sns             = "http://localhost:4566"
    sqs             = "http://localhost:4566"
    ssm             = "http://localhost:4566"
    stepfunctions   = "http://localhost:4566"
    sts             = "http://localhost:4566"
  }

Just got to figure out how to fix the hostname / path style

Aws::Errors::NoSuchEndpointError: Encountered a `SocketError` while attempting to connect to:

  http://*.localgd:4566

This is typically the result of an invalid `:region` option or a
poorly formatted `:endpoint` option.

* Avoid configuring the `:endpoint` option directly. Endpoints are constructed
  from the `:region`. The `:endpoint` option is reserved for certain services
  or for connecting to non-standard test endpoints.

* Not every service is available in every region.

* Never suffix region names with availability zones.
  Use "us-east-1", not "us-east-1a"

Known AWS regions include (not specific to this service):

af-south-1
ap-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-south-2
ap-southeast-1
ap-southeast-2
ap-southeast-3
ap-southeast-4
aws-global
ca-central-1
eu-central-1
eu-central-2
eu-north-1
eu-south-1
eu-south-2
eu-west-1
eu-west-2
eu-west-3
il-central-1
me-central-1
me-south-1
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2
aws-cn-global
cn-north-1
cn-northwest-1
aws-us-gov-global
us-gov-east-1
us-gov-west-1
aws-iso-global
us-iso-east-1
us-iso-west-1
aws-iso-b-global
us-isob-east-1

Error evaluating ERB template around line 6 of: /Users/tvl/Documents/repos/i1/tf-terraspace/infra/config/terraform/backend.tf:
 1 # This file was initially generated by terraspace_plugin_aws 0.6.1
 2 # Backend Config Variables Docs
 3 # https://terraspace.cloud/docs/config/backend/variables/
 4 terraform {
 5   backend "s3" {
 6     bucket         = "<%= expansion('terraform-state-:ACCOUNT-:REGION-:ENV') %>"
 7     key            = "<%= expansion(':PROJECT/:REGION/:APP/:ROLE/:ENV/:EXTRA/:BUILD_DIR/terraform.tfstate') %>"
 8     region         = "<%= expansion(':REGION') %>"
 9     dynamodb_table = "terraform_locks"
10   }
11 }
#3

app.rb

require 'aws-sdk-s3'

Aws.config.update(
  endpoint: 'http://s3.localhost.localstack.cloud.local.gd:4566',
  credentials: Aws::SharedCredentials.new(profile_name: 'localstack-dev', access_key_id: 'fake', secret_access_key: 'fake'),
  s3: {
    http_wire_trace: true,
    force_path_style: true,
    ssl_verify_peer: false,
    validate_params: false # Skip credential validation
  }
  #validate_params: true,
  #skip_requesting_account_id: true,
  #skip_metadata_api_check: true,
  #s3: {force_path_style: 'true'},
  #s3: {http_wire_trace: 'true'},
)

# Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
  config.cloud.cost.enabled = false
  config.logger.level = :debug
  config.test_framework = "rspec"
  # copy_modules setting introduced 2.2.5 to speed up terraspace build
  # See: https://terraspace.cloud/docs/config/reference
  config.build.copy_modules = true
  config.build.cache_dir	= ":REGION/:APP/:ROLE/:ENV/:BUILD_DIR" # without :REGION
end
#4

AWS config

[profile localstack-dev]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json

[profile localstack-nonprod]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json

[profile localstack-prod]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json

[services s3-specific]
s3 =
  endpoint_url = http://s3.localhost.localstack.cloud.local.gd:4566
#5

provider.tf

provider "aws" {
  region                      = "eu-west-1"
  access_key                  = "fake"
  secret_key                  = "fake"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  s3_force_path_style         = true

  endpoints {
    apigateway      = "http://local.gd:4566"
    apigatewayv2    = "http://local.gd:4566"
    cloudformation  = "http://local.gd:4566"
    cloudwatch      = "http://local.gd:4566"
    cognitoidp      = "http://local.gd:4566"
    cognitosync     = "http://local.gd:4566"
    cognitoidentity = "http://local.gd:4566"
    dynamodb        = "http://local.gd:4566"
    ec2             = "http://local.gd:4566"
    es              = "http://local.gd:4566"
    elasticache     = "http://local.gd:4566"
    firehose        = "http://local.gd:4566"
    glue            = "http://local.gd:4566"
    iam             = "http://local.gd:4566"
    kinesis         = "http://local.gd:4566"
    lambda          = "http://local.gd:4566"
    rds             = "http://local.gd:4566"
    redshift        = "http://local.gd:4566"
    route53         = "http://local.gd:4566"
    s3              = "http://local.gd:4566"
    secretsmanager  = "http://local.gd:4566"
    ses             = "http://local.gd:4566"
    sns             = "http://local.gd:4566"
    sqs             = "http://local.gd:4566"
    ssm             = "http://local.gd:4566"
    stepfunctions   = "http://local.gd:4566"
    sts             = "http://local.gd:4566"
  }
}
#6

Stuck here now

Initializing the backend...
Initializing modules...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
│ 	status code: 403, request id: 05b6b1f9-2254-464c-ae97-c508b6d1aebc
#8

My app.rb looks like this now

require 'aws-sdk-s3'

Aws.config.update(
  endpoint: 'https://localhost.localstack.cloud:4566',
  credentials: Aws::SharedCredentials.new(profile_name: 'localstack-dev', access_key_id: 'fake', secret_access_key: 'fake'),
  s3: {
    endpoint: 'https://s3.localhost.localstack.cloud:4566',
    access_key_id: 'fake',
    secret_access_key: 'fake',
    region: 'eu-west-1',
    http_wire_trace: true,
    force_path_style: true,
  }
)

# Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
  config.cloud.cost.enabled = false
  config.logger.level = :debug
  config.test_framework = "rspec"
  # copy_modules setting introduced 2.2.5 to speed up terraspace build
  # See: https://terraspace.cloud/docs/config/reference
  config.build.copy_modules = true
  config.build.cache_dir	= ":REGION/:APP/:ROLE/:ENV/:BUILD_DIR"
end```
#9

And my AWS config looks like this

[profile localstack-nonprod]
endpoint_url = https://localhost.localstack.cloud:4566
services = localstack
region = eu-west-1
output = json
cli_history = enabled
cli_pager = bat

[profile localstack-prod]
endpoint_url = https://localhost.localstack.cloud:4566
services = localstack
region = eu-west-1
output = json
cli_history = enabled
cli_pager = bat

[services localstack]
s3 =
  addressing_style = path
#10

And my bucket is created as you can see from here when I run terraspace up demo

aws s3 ls --profile localstack-dev
2023-10-09 16:40:33 terraform-state-000000000000-eu-west-1-dev
#11

But I am stuck here.

Conn close
Bucket already exist: terraform-state-000000000000-eu-west-1-dev
opening connection to s3.localhost.localstack.cloud:4566...
opened
starting SSL for s3.localhost.localstack.cloud:4566...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256
<- "GET /terraform-state-000000000000-eu-west-1-dev/?tagging HTTP/1.1\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.185.1 ua/2.0 api/s3#1.136.0 os/macos#21 md/x86_64 lang/ruby#3.0.3 md/3.0.3 cfg/retry-mode#legacy\r\nHost: s3.localhost.localstack.cloud:4566\r\nX-Amz-Date: 20231010T163438Z\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=fake/20231010/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c6c4beef839192b954840ada989d74ab0b9862b5b18fc011c53a4abe01d230e7\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 404 \r\n"
-> "Content-Type: application/xml\r\n"
-> "Content-Length: 249\r\n"
-> "x-amz-request-id: ************************************\r\n"
-> "x-amz-id-2: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=\r\n"
-> "Connection: close\r\n"
-> "date: Tue, 10 Oct 2023 16:34:38 GMT\r\n"
-> "server: hypercorn-h11\r\n"
-> "\r\n"
reading 249 bytes...
-> ""
-> "<?xml version='1.0' encoding='utf-8'?>\n<Error><Code>NoSuchTagSet</Code><Message>The TagSet does not exist</Message><RequestId>************************************</RequestId><BucketName>terraform-state-000000000000-eu-west-1-dev</BucketName></Error>"
read 249 bytes
Conn close
Table already exist: terraform_locks
=> terraform init -reconfigure -get -input=false

Initializing the backend...
Initializing modules...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
│ 	status code: 403, request id: 8fbf3fdb-3d5d-4f74-8a67-16726b48253a
│