Running a forked version of Jets

fabiensebban · February 7, 2019, 3:33pm

Hi!

I am trying to test the pull request 103 (Lambda trigger and DLQ support for SQS extension) because I need it for a project.

To achieve this goal I followed the steps below :

Fork Jets
Add the sqs_queue and sqs_lambda_trigger extensions in the sqs-lambda-trigger branch
In my cloud9 environment :
git clone https://github.com/fabiensebban/jets.git
git checkout sqs-lambda-trigger
cd jets/ && gem build jets.gemspec
gem install jets-1.6.10.gem
jets -v to check if the installation went OK

Here is the error I have :

Traceback (most recent call last):
        10: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in '<main>'
         9: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in 'eval'
         8: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bin/jets:23:in '<main>'
         7: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bin/jets:23:in 'load'
         6: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bundler/gems/jets-2a930de8a4de/exe/jets:13:in '<top (required)>'
         5: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bundler/gems/jets-2a930de8a4de/exe/jets:13:in 'require'
         4: from /home/ec2-user/.rvm/gems/ruby-2.5.3/bundler/gems/jets-2a930de8a4de/lib/jets.rb:65:in '<top (required)>'
         3: from /home/ec2-user/.rvm/rubies/ruby-2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:291:in 'require'
         2: from /home/ec2-user/.rvm/rubies/ruby-2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:257:in 'load_dependency'
         1: from /home/ec2-user/.rvm/rubies/ruby-2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:291:in 'block in require'
/home/ec2-user/.rvm/rubies/ruby-2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:291:in 'require': cannot load such file -- jets-gems (LoadError)

Did I miss something to install my forked gem correctly?

Thank you for your help and for this powerful gem!

tung · February 8, 2019, 7:48pm

RE: Running a forked version of Jets

To run a fork, can use the Gemfile.

Gemfile:

  gem "jets", git: "https://github.com/fabiensebban/jets.git", submodules: true, branch: "sqs-lambda-trigger"

This will run the fork for the project during “runtime”.

During, what I’m calling “compile” time, when you use the jets command standalone and it’s not using bundler so you can use wrapper script to ensure it’s calling a local version.

Setting up the jets project locally.

mkdir ~/environment
cd enviroment
git clone https://github.com/fabiensebban/jets.git
cd jets
git checkout sqs-lambda-trigger
git submodule init
git submodule update
bundle

Create a ~/bin/jets wrapper script that calls the local version:

mkdir ~/bin
cat << EOF > ~/bin/jets
#!/bin/bash
>&2 echo "Using local version at ~/environment/jets"
exec ~/environment/jets/exe/jets "$@"
EOF
chmod a+x ~/bin/jets

export PATH="~/bin:$PATH" # add this to your .profile if you wish

Now you can call jets -h and you’ll see Using local version at ~/environment/jets to remind yourself that you’re running a local version.

Note, when you’re running things locally, you need to remember to update and manage the local copy. IE: run bundle if you a git pull that brings in new dependencies.

RE: (Lambda trigger and DLQ support for SQS extension) because I need it for a project.

Released this SQS Events support Some other person was asking about it on https://www.reddit.com/r/ruby/comments/ant6r2/jets_introduction_series_lambda_api_gateway/efwe3q9/ And took a quick stab at it.

fabiensebban · February 11, 2019, 10:33am

Thank so much for your answer! It works great

Now I can run my local jets gem.

This looks really nice! I will give it a try after doing my own SQS lambda trigger

roque86 · January 24, 2022, 12:30pm

Hi @tung,

I am also trying to use a forked version but facing some issues while trying to deploy.
I need to use a slightly different handler template (ib/jets/builders/templates/handler.rb) and decided to fork the project and make the required modifications (given we are not currently able to pass custom handler templates to jets).
Is that the best approach for now?

Moreover, I am still unable to use the forked version to deploy. The zip files are generated fine and I can see my expected handlers as well but the CloudFormation deploy process gives me an error:
“ApiDeployment20220124073305-NDNHED4XUG8J/3590c750-7d02-11ec-8c10-06c5cf500f92 was not successfully created”
“The following resource(s) failed to create: [BasePathMapping]”.

Is that related to the fact that I am using the forked version?

roque86 · January 25, 2022, 6:08pm

It was actually version 3.1.1 breaking things (regardless of my forked version) but everything was fine with 3.0.3.

tung · January 26, 2022, 11:03pm

Think this is the main clue:

The following resource(s) failed to create: [BasePathMapping]

The main change from jets 3.0 to jets 3.1 was updating zeitwerk. This PR: https://github.com/boltops-tools/jets/pull/614 Wondering if that has something to do with it.

However, BasePathMapping is related to the lambda function for a Custom CloudFormation Resource that handles adding a custom domain. https://rubyonjets.com/docs/routing/custom-domain/

Tried deploying a demo app with a custom domain and that worked though.

Click to Expand and See Debugging Session

$ jets deploy
...
Uploading /tmp/jets/demo/stage/zips/code-ac6ed458.zip (169 KB) to S3
Uploaded to s3://demo-dev-s3bucket-1ao88t1oku1uu/jets/code/code-ac6ed458.zip
Time to upload code to s3: 0s
Checking for modified public assets and uploading to S3.
Time for public assets to s3: 0s
Deploying CloudFormation stack with jets app!
Waiting for stack to complete
10:52:47PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack demo-dev User Initiated
10:52:53PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
10:52:53PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob 
10:52:54PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
10:52:56PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 
10:52:57PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiResources1 
10:52:59PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPublicController 
10:52:59PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack PostsController 
10:53:28PM UPDATE_COMPLETE AWS::CloudFormation::Stack JetsPreheatJob 
10:53:34PM UPDATE_COMPLETE AWS::CloudFormation::Stack JetsPublicController 
10:53:34PM UPDATE_COMPLETE AWS::CloudFormation::Stack PostsController 
10:53:36PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiDeployment20220126225243 
10:53:38PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiDeployment20220126225243 Resource creation Initiated
10:54:39PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiDeployment20220126225243 
10:54:41PM UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack demo-dev 
10:54:43PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiDeployment20220126222116 
10:54:53PM UPDATE_COMPLETE AWS::CloudFormation::Stack JetsPreheatJob 
10:55:06PM DELETE_COMPLETE AWS::CloudFormation::Stack ApiDeployment20220126222116 
10:55:17PM UPDATE_COMPLETE AWS::CloudFormation::Stack PostsController 
10:55:17PM UPDATE_COMPLETE AWS::CloudFormation::Stack JetsPublicController 
10:55:18PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiResources1 
10:55:19PM UPDATE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
10:55:20PM UPDATE_COMPLETE AWS::CloudFormation::Stack demo-dev 
Stack success status: UPDATE_COMPLETE
Time took for stack deployment: 2m 33s.
Prewarming application.
API Gateway Endpoint: https://de34sa2d92.execute-api.us-west-2.amazonaws.com/dev/
Custom Domain: https://custom-domain.example.com
$ grep BasePathMapping /tmp/jets/demo/templates/*.yml
/tmp/jets/demo/templates/demo-dev-api-deployment.yml:  BasePathMapping:
/tmp/jets/demo/templates/demo-dev-api-deployment.yml:    Type: Custom::BasePathMapping
/tmp/jets/demo/templates/demo-dev-api-deployment.yml:  BasePathMapping:
/tmp/jets/demo/templates/demo-dev-api-deployment.yml:    Value: !Ref BasePathMapping
$

Unsure exactly how to re-produce right now. Wondering if you can try deploying again and this time while it’s deploying try to see what the error is in the CloudFormation console. See: https://rubyonjets.com/docs/debugging/cloudformation/

Hopefully, that provides some more info. Thanks.

roque86 · February 1, 2022, 9:24pm

Hi @tung, thanks for the help.

The issue is that I cannot see any error with the resource or event related to BasePathMapping. It is just stuck while creating (until I cancel or it fails as unable to create).
I moved from V3.0.3 to V3.0.23 to get the fix about wrong file sizes on CI environments and once again I was unable to deploy because of ApiDeployment/BasePathMapping

Any other advice to help me get to the root of the problem?

tung · February 1, 2022, 10:41pm

Wondering if you’re checking in the CloudFormation console also. Looking for the nested stack. Just double checking.

roque86 · February 1, 2022, 11:40pm

Yes, here goes the screenshots from events and resources.
It is just stuck at finishing the BasePathMapping

roque86 · February 2, 2022, 1:22am

Given enough time, it just fails with something like:

CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [23dce581-f7d9-483b-9643-86e741a9b14d]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.

tung · February 2, 2022, 2:24am

I see. Believe it or not these screenshots are helpful.

Think what might be happening is that the lambda function erroring so early that it’s not even able to report the error to CloudFormation and hence the error does not surface

Sadly, custom CloudFormation resources are pretty difficult tough to write and debug. You don’t want to know long it took to initially figure out

Here’s how would go about debugging this though.

First, some background is needed:

The source code for the lambda function that jets uses to create the base path mapping is here: https://github.com/boltops-tools/jets/blob/master/lib/jets/internal/app/functions/jets/base_path.rb - It gets compiled down as an ERB template with @stage_name replaced.
That lambda function code calls CfnResponse.new which is a library that handles the mind-numbingly boring request and response cycle of a Custom CloudFormation resource. https://github.com/boltops-tools/cfn_response
The library wraps a big rescue Exception around all code within the block so that if any error happens it will
1. log the error to cloudwatch
2. send the failed status to CloudFormation so it fails quickly. So it won’t result in CloudFormation custom resource hanging and eventually timing out after 2 hours from a code error.

Since the function is erroring with no data at all that means it’s not even making it to the cfn.response do ... end code. It’s erroring super early. To see the error you have to find the corresponding base-path-{timestamp} function’s logs and see what that reports.

So deploy again, and try to find the sp-payments-dev-jets-base-path-XXX function and check out it’s logs.

Note: Jets had to create a function with a timestamp otherwise Custom CloudFormation resources can’t tell the lambda function is different. Like I mentioned, Custom CloudFormation Resources are a bit of pain to create and debug. Pretty much had to do for the BasePathMappings though.

roque86 · February 2, 2022, 11:13am

Thank you @tung, it helped a lot when I managed to find the log (jets-base-path-XXX).

The issue is with gems from git:
“Init error when loading handler handlers/functions/jets/base_path.lambda_handler”
“XXX is not yet checked out. Run bundle install first.”
Where XXX is whatever gem we use from github repository.

I will now try to identify what changed between versions to cause such an error for my case.

roque86 · February 2, 2022, 4:46pm

I managed to avoid the problem by using the github rubygems repository but now I am facing another issue.
We were using the development environment in remote as well, but Jets does not include the development and test groups from the Gemfile (as expected).
However, given that bundler_groups is defined as [:default, Jets.env.to_sym] and in case we are in development remote, Bundler.require(*bundler_groups) with try to check for the gems defined in the development group, but they were not included.

This was not an issue with version 3.0.3. What would be the recommendation for the newer versions in this case?

tung · February 2, 2022, 7:04pm

Looked at the compare between 3.0.3…master and don’t see anything immediately obvious that would change the behavior.

Unsure why it worked before Wondering if it’s an upstream thing with bundler itself. Though, 90% of the time when guess it’s something upstream it’s usually me So it’s likely am missing something

Stepping back and thinking about this more generally. Maybe there should be a configuration setting to allow the bundle install to install also development gems. Current code where the logic is:

github.com

boltops-tools/jets/blob/4e6842598c2fa9c5a7492f4b177345ac2b91b056/lib/jets/builders/ruby_packager.rb#L218


      cache_bundle_config = "#{cache_area}/.bundle/config"
      app_bundle_config = "#{@full_app_root}/.bundle/config"
      FileUtils.mkdir_p(File.dirname(app_bundle_config))
      FileUtils.cp(cache_bundle_config, app_bundle_config)
    end
    # On circleci the "#{Jets.build_root}/.bundle/config" doesnt exist
    # this only happens with ssh debugging, not when the ci.sh script gets ran.
    # But on macosx it exists.
    # Dont know why this is the case.
    def create_bundle_config(frozen: false)
      FileUtils.rm_rf("#{cache_area}/.bundle")
      frozen_line = %Q|BUNDLE_FROZEN: "true"\n| if frozen
      text =<<-EOL
---
#{frozen_line}BUNDLE_PATH: "vendor/gems"
BUNDLE_WITHOUT: "development:test"
EOL
      bundle_config = "#{cache_area}/.bundle/config"
      FileUtils.mkdir_p(File.dirname(bundle_config))
      IO.write(bundle_config, text)

Maybe:

config/application.rb:

Jets.application.configure do
  config.build.bundle_without = "development:test" # by default
end

Would request that docs are updated also: https://rubyonjets.com/docs/app-config/reference/

Will consider PRs. No sweat either way. Will get to this in time