Error when trying to deploy to AWS [Urgent Help Needed]

Jets
,
1

I get this error when I try to deploy to AWS; I am 100% sure my AWS credentials are correct.

   Tidying project: removing ignored files to reduce package size.
    => rsync -a --links /tmp/jets/pcp_jets_app/stage/code/vendor/gems/ruby/2.7.0/ /tmp/jets/pcp_jets_app/stage/opt/ruby/gems/2.7.0/
    => Replacing compiled gems with AWS Lambda Linux compiled versions: /tmp/jets/pcp_jets_app/stage/opt
    Checking projects gems for binary serverlessgems...
    Traceback (most recent call last):
    	36: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/bin/ruby_executable_hooks:22:in `<main>'
    	35: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/bin/ruby_executable_hooks:22:in `eval'
    	34: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/bin/jets:23:in `<main>'
    	33: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/bin/jets:23:in `load'
    	32: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/exe/jets:14:in `<top (required)>'
    	31: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/cli.rb:5:in `start'
    	30: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/cli.rb:28:in `start'
    	29: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/base.rb:27:in `perform'
    	28: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/base.rb:38:in `dispatch'
    	27: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
    	26: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
    	25: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
    	24: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/main.rb:27:in `deploy'
    	23: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/deploy.rb:32:in `run'
    	22: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/deploy.rb:61:in `build_code'
    	21: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/commands/build.rb:27:in `build_code'
    	20: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/code_builder.rb:38:in `build'
    	19: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/code_builder.rb:38:in `chdir'
    	18: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/code_builder.rb:41:in `block in build'
    	17: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/code_builder.rb:384:in `package_ruby'
    	16: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/code_builder.rb:395:in `build_lambda_layer'
    	15: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/lambda_layer.rb:10:in `build'
    	14: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/lambda_layer.rb:65:in `replace_compiled_gems'
    	13: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/gem_replacer.rb:20:in `run'
    	12: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/gem_replacer.rb:20:in `each'
    	11: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/jets-3.2.0/lib/jets/builders/gem_replacer.rb:22:in `block in run'
    	10: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/serverlessgems-0.2.0/lib/jets/gems/extract/gem.rb:11:in `run'
    	 9: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/serverlessgems-0.2.0/lib/jets/gems/extract/gem.rb:55:in `download_gem'
    	 8: from /Users/adnanmateen/.rvm/gems/ruby-2.7.6/gems/serverlessgems-0.2.0/lib/jets/gems/extract/base.rb:44:in `download_file'
    	 7: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:50:in `open'
    	 6: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:744:in `open'
    	 5: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:174:in `open_uri'
    	 4: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:233:in `open_loop'
    	 3: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:233:in `catch'
    	 2: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:235:in `block in open_loop'
    	 1: from /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:764:in `buffer_open'
    /Users/adnanmateen/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/open-uri.rb:387:in `open_http': 403 Forbidden (OpenURI::HTTPError)

When I add

Jets.application.configure do
  config.gems.disable = true
end

The deployment works perfectly fine. I have narrowed it down to the issue with serverlessgems
but when I ran jets gems:check it worked perfectly fine. Need urgent help

Thank you in Advance
Adnan

2

Sorry. Been busy with the Holidays. :christmas_tree:

Bummer. Looked into it. So got uptime monitoring on Serverless Gems and no downtime was reported. Serverless Gems API is actually a jets app. It’s lambda and APIGW. So don’t believe it was an Lambda or API issue.

The 403 was came from the generated download url. This is essentially CloudFront. So thinking that CloudFront may have been having issues at the time :unamused: That’s just a guess. It’s the first time seeing this so thinking that it’s rare. Hoping that it’s been long resolved by now. If it happens again, please let me know. Got some ideas for some additional safeguards. IE: generating an direct s3 signed url instead and using that based on some automatic monitoring. That type of stuff can sometimes be tricky to implement also. It typically takes some time to reduce flapping alarms. Also, it’s just one idea.

At least, wanted to respond though. Happy holidays :confetti_ball::tada::champagne::clinking_glasses:

3

Thank you for the response, and no issue.

Could it be because of the region? I am working from Pakistan.

Also, I sometimes get this error as well.

Command Failed cd /tmp/jets/pcp_serverless_app/stage/opt && unzip -qo /tmp/jets/pcp_serverless_app/serverlessgems/downloads/gems/puma-5.1.1.zip > /dev/null 2>&1

Also, details of custom layer creation documentation would be appreciated for jets. I couldn’t create a layer to include the nokogiri gem 1.13.10 properly because of that; my application kept throwing errors about missing the nokogiri gem. I had to degrade my nokogiri gem to 1.10.10.

4

And I am still getting 403 errors.

5

RE: custom layers

AWS has docs covering that.

RE: Could it be because of the region? I am working from Pakistan.

Maybe. Tried connecting to a VPN. My VPN client didn’t have Pakistan, but was able to test from India and that worked. Unsure.

Finally someone else reported it also this morning. It’s seems to be an intermittent issue from the pre-signed CloudFront urls. Maybe it’s worse in Pakistan :man_shrugging:t2: In any case, improved the client download logic to automatically retry download retry for 403 forbidden by tongueroo · Pull Request #7 · boltops-tools/serverlessgems · GitHub Released in serverlessgems 0.2.1 That should fix/mitigate the issue.