Hi Community,
How do I configure my app.rb file to correctly create and connect to S3 using LocalStack as my testing platform? Thank you.
I have been testing with the following configs:
require 'aws-sdk-s3'
region = "eu-west-1"
# Aws.config.update(
# endpoint: 'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
# access_key_id: 'fake', # update with localstack credentials
# secret_access_key: 'fake', # update with localstack credentials
# region: region,
# force_path_style: true, # Enable 'force_path_style' => true, if bucket name is non DNS compliant
# )
# Aws.config.update(
# endpoint: 'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
# credentials: Aws::Credentials.new('fake', 'fake'),
# region: region,
# force_path_style: true,
# )
# # Create an S3 client with custom endpoint and force path style
s3 = Aws::S3::Client.new(
endpoint: 'http://s3.localhost.localstack.cloud:4566', # update with localstack endpoint
region: region,
access_key_id: 'fake', # update with localstack credentials
secret_access_key: 'fake', # update with localstack credentials
force_path_style: true # Enable force path style
)
# Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
config.logger.level = :info
# copy_modules setting introduced 2.2.5 to speed up terraspace build
# See: https://terraspace.cloud/docs/config/reference
config.build.copy_modules = true
#config.build.cache_dir = ":APP/:ROLE/:ENV/:BUILD_DIR" # without :REGION
#endpoint: 'http://localhost:4572',
#credentials: Aws::Credentials.new('fake', 'fake'),
#region: 'us-east-1',
#force_path_style: true,
end
I am making progress by doing the following:
aws config
[profile localstack-dev]
endpoint_url = http://*.localgd:4566
region = eu-west-1
output = json
Terraspace
infra → config → provider.tf
provider "aws" {
region = "eu-west-1"
access_key = "fake"
secret_key = "fake"
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
s3_force_path_style = true
endpoints {
apigateway = "http://localhost:4566"
apigatewayv2 = "http://localhost:4566"
cloudformation = "http://localhost:4566"
cloudwatch = "http://localhost:4566"
cognitoidp = "http://localhost:4566"
cognitosync = "http://localhost:4566"
cognitoidentity = "http://localhost:4566"
dynamodb = "http://localhost:4566"
ec2 = "http://localhost:4566"
es = "http://localhost:4566"
elasticache = "http://localhost:4566"
firehose = "http://localhost:4566"
glue = "http://localhost:4566"
iam = "http://localhost:4566"
kinesis = "http://localhost:4566"
lambda = "http://localhost:4566"
rds = "http://localhost:4566"
redshift = "http://localhost:4566"
route53 = "http://localhost:4566"
s3 = "http://*.local.gd:4566"
secretsmanager = "http://localhost:4566"
ses = "http://localhost:4566"
sns = "http://localhost:4566"
sqs = "http://localhost:4566"
ssm = "http://localhost:4566"
stepfunctions = "http://localhost:4566"
sts = "http://localhost:4566"
}
Just got to figure out how to fix the hostname / path style
Aws::Errors::NoSuchEndpointError: Encountered a `SocketError` while attempting to connect to:
http://*.localgd:4566
This is typically the result of an invalid `:region` option or a
poorly formatted `:endpoint` option.
* Avoid configuring the `:endpoint` option directly. Endpoints are constructed
from the `:region`. The `:endpoint` option is reserved for certain services
or for connecting to non-standard test endpoints.
* Not every service is available in every region.
* Never suffix region names with availability zones.
Use "us-east-1", not "us-east-1a"
Known AWS regions include (not specific to this service):
af-south-1
ap-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-south-2
ap-southeast-1
ap-southeast-2
ap-southeast-3
ap-southeast-4
aws-global
ca-central-1
eu-central-1
eu-central-2
eu-north-1
eu-south-1
eu-south-2
eu-west-1
eu-west-2
eu-west-3
il-central-1
me-central-1
me-south-1
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2
aws-cn-global
cn-north-1
cn-northwest-1
aws-us-gov-global
us-gov-east-1
us-gov-west-1
aws-iso-global
us-iso-east-1
us-iso-west-1
aws-iso-b-global
us-isob-east-1
Error evaluating ERB template around line 6 of: /Users/tvl/Documents/repos/i1/tf-terraspace/infra/config/terraform/backend.tf:
1 # This file was initially generated by terraspace_plugin_aws 0.6.1
2 # Backend Config Variables Docs
3 # https://terraspace.cloud/docs/config/backend/variables/
4 terraform {
5 backend "s3" {
6 bucket = "<%= expansion('terraform-state-:ACCOUNT-:REGION-:ENV') %>"
7 key = "<%= expansion(':PROJECT/:REGION/:APP/:ROLE/:ENV/:EXTRA/:BUILD_DIR/terraform.tfstate') %>"
8 region = "<%= expansion(':REGION') %>"
9 dynamodb_table = "terraform_locks"
10 }
11 }
app.rb
require 'aws-sdk-s3'
Aws.config.update(
endpoint: 'http://s3.localhost.localstack.cloud.local.gd:4566',
credentials: Aws::SharedCredentials.new(profile_name: 'localstack-dev', access_key_id: 'fake', secret_access_key: 'fake'),
s3: {
http_wire_trace: true,
force_path_style: true,
ssl_verify_peer: false,
validate_params: false # Skip credential validation
}
#validate_params: true,
#skip_requesting_account_id: true,
#skip_metadata_api_check: true,
#s3: {force_path_style: 'true'},
#s3: {http_wire_trace: 'true'},
)
# Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
config.cloud.cost.enabled = false
config.logger.level = :debug
config.test_framework = "rspec"
# copy_modules setting introduced 2.2.5 to speed up terraspace build
# See: https://terraspace.cloud/docs/config/reference
config.build.copy_modules = true
config.build.cache_dir = ":REGION/:APP/:ROLE/:ENV/:BUILD_DIR" # without :REGION
end
AWS config
[profile localstack-dev]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json
[profile localstack-nonprod]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json
[profile localstack-prod]
endpoint_url = http://local.gd:4566
service = localstack-s3
region = eu-west-1
output = json
[services s3-specific]
s3 =
endpoint_url = http://s3.localhost.localstack.cloud.local.gd:4566
provider.tf
provider "aws" {
region = "eu-west-1"
access_key = "fake"
secret_key = "fake"
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
s3_force_path_style = true
endpoints {
apigateway = "http://local.gd:4566"
apigatewayv2 = "http://local.gd:4566"
cloudformation = "http://local.gd:4566"
cloudwatch = "http://local.gd:4566"
cognitoidp = "http://local.gd:4566"
cognitosync = "http://local.gd:4566"
cognitoidentity = "http://local.gd:4566"
dynamodb = "http://local.gd:4566"
ec2 = "http://local.gd:4566"
es = "http://local.gd:4566"
elasticache = "http://local.gd:4566"
firehose = "http://local.gd:4566"
glue = "http://local.gd:4566"
iam = "http://local.gd:4566"
kinesis = "http://local.gd:4566"
lambda = "http://local.gd:4566"
rds = "http://local.gd:4566"
redshift = "http://local.gd:4566"
route53 = "http://local.gd:4566"
s3 = "http://local.gd:4566"
secretsmanager = "http://local.gd:4566"
ses = "http://local.gd:4566"
sns = "http://local.gd:4566"
sqs = "http://local.gd:4566"
ssm = "http://local.gd:4566"
stepfunctions = "http://local.gd:4566"
sts = "http://local.gd:4566"
}
}
Stuck here now
Initializing the backend...
Initializing modules...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
│ status code: 403, request id: 05b6b1f9-2254-464c-ae97-c508b6d1aebc
My app.rb looks like this now
require 'aws-sdk-s3'
Aws.config.update(
endpoint: 'https://localhost.localstack.cloud:4566',
credentials: Aws::SharedCredentials.new(profile_name: 'localstack-dev', access_key_id: 'fake', secret_access_key: 'fake'),
s3: {
endpoint: 'https://s3.localhost.localstack.cloud:4566',
access_key_id: 'fake',
secret_access_key: 'fake',
region: 'eu-west-1',
http_wire_trace: true,
force_path_style: true,
}
)
# Docs: https://terraspace.cloud/docs/config/reference/
Terraspace.configure do |config|
config.cloud.cost.enabled = false
config.logger.level = :debug
config.test_framework = "rspec"
# copy_modules setting introduced 2.2.5 to speed up terraspace build
# See: https://terraspace.cloud/docs/config/reference
config.build.copy_modules = true
config.build.cache_dir = ":REGION/:APP/:ROLE/:ENV/:BUILD_DIR"
end```
And my AWS config looks like this
[profile localstack-nonprod]
endpoint_url = https://localhost.localstack.cloud:4566
services = localstack
region = eu-west-1
output = json
cli_history = enabled
cli_pager = bat
[profile localstack-prod]
endpoint_url = https://localhost.localstack.cloud:4566
services = localstack
region = eu-west-1
output = json
cli_history = enabled
cli_pager = bat
[services localstack]
s3 =
addressing_style = path
And my bucket is created as you can see from here when I run terraspace up demo
aws s3 ls --profile localstack-dev
2023-10-09 16:40:33 terraform-state-000000000000-eu-west-1-dev
But I am stuck here.
Conn close
Bucket already exist: terraform-state-000000000000-eu-west-1-dev
opening connection to s3.localhost.localstack.cloud:4566...
opened
starting SSL for s3.localhost.localstack.cloud:4566...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256
<- "GET /terraform-state-000000000000-eu-west-1-dev/?tagging HTTP/1.1\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.185.1 ua/2.0 api/s3#1.136.0 os/macos#21 md/x86_64 lang/ruby#3.0.3 md/3.0.3 cfg/retry-mode#legacy\r\nHost: s3.localhost.localstack.cloud:4566\r\nX-Amz-Date: 20231010T163438Z\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=fake/20231010/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c6c4beef839192b954840ada989d74ab0b9862b5b18fc011c53a4abe01d230e7\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 404 \r\n"
-> "Content-Type: application/xml\r\n"
-> "Content-Length: 249\r\n"
-> "x-amz-request-id: ************************************\r\n"
-> "x-amz-id-2: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=\r\n"
-> "Connection: close\r\n"
-> "date: Tue, 10 Oct 2023 16:34:38 GMT\r\n"
-> "server: hypercorn-h11\r\n"
-> "\r\n"
reading 249 bytes...
-> ""
-> "<?xml version='1.0' encoding='utf-8'?>\n<Error><Code>NoSuchTagSet</Code><Message>The TagSet does not exist</Message><RequestId>************************************</RequestId><BucketName>terraform-state-000000000000-eu-west-1-dev</BucketName></Error>"
read 249 bytes
Conn close
Table already exist: terraform_locks
=> terraform init -reconfigure -get -input=false
Initializing the backend...
Initializing modules...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
│ status code: 403, request id: 8fbf3fdb-3d5d-4f74-8a67-16726b48253a
│