Init error when loading handler - errorType": "Init<Gem::LoadError>

Jets
#1

Hi Gurus,

This morning I had the bad surprise that my Web App in Production environment is not working anymore.

I did not push any new version of my app since one month.

Please find the log below:

Blockquote---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| @timestamp | @message |
|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2020-04-11 08:30:04.417 | Init error when loading handler handlers/controllers/sessions_controller.create |
| 2020-04-11 08:30:04.417 | { |
| 2020-04-11 08:30:04.417 | “errorMessage”: “You have already activated json 2.3.0, but your Gemfile requires json 2.2.0. Prepending bundle exec to your command may solve this.”, |
| 2020-04-11 08:30:04.417 | “errorType”: “InitGem::LoadError”, |
| 2020-04-11 08:30:04.417 | “stackTrace”: [ |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:312:in check_for_activated_spec!'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:31:inblock in setup’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/spec_set.rb:147:in each'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/spec_set.rb:147:ineach’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:26:in map'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:26:insetup’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler.rb:149:in setup'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/setup.rb:20:inblock in <top (required)>’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:136:in with_level'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:88:insilence’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in <top (required)>'", | | 2020-04-11 08:30:04.417 | "/var/task/handlers/controllers/sessions_controller.rb:1:inrequire’”, |
| 2020-04-11 08:30:04.417 | “/var/task/handlers/controllers/sessions_controller.rb:1:in <top (required)>'", | | 2020-04-11 08:30:04.417 | "/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:inrequire’”, |
| 2020-04-11 08:30:04.417 | “/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:in `require’” |
| 2020-04-11 08:30:04.417 | ] |
| 2020-04-11 08:30:04.417 | } |
| 2020-04-11 08:30:04.108 | END RequestId: 607273bb-9f0d-4752-b415-33f2de695147 |
| 2020-04-11 08:30:04.108 | REPORT RequestId: 607273bb-9f0d-4752-b415-33f2de695147 Duration: 389.25 ms Billed Duration: 400 ms Memory Size: 1536 MB Max Memory Used: 21 MB |
| 2020-04-11 08:30:04.108 | Unknown application error occurred Function |
| 2020-04-11 08:30:04.061 | }

Any idea to solve this issue? and why it happens so suddenly whereas I do not push any new code or version?

Cheers,

Fabien.

#2

Hi,

I just get a feedback from aws support :slight_smile:
Hi Fabien,

Thank you for contacting AWS Premium Support. I hope you are keeping well! My name is Wassim and I will be assisting you today.

I understand you noticed that your Lambda function started to fail execution without performing any code change.

Thanks for sharing the error message, that was of great help to initiate the investigation. The error suggest that the ‘json’ library was upgraded to version “ 2.3.0”, but your code was expecting version “2.2.0” ( as specified by the error message, “You have already activated json 2.3.0, but your Gemfile requires json 2.2.0” ). The default Lambda environment will tend to use always the latest libraries to enable the latest set of features and security updates as specify in this documentation[1], hence , I do believe that the Lambda team have upgraded the library on the default Lambda environment which caused this issue.

In order to avoid such behavior, you need to package all of your dependencies in deployment package, this will allow you to have a full control of the environment and avoid being affected by libraries changes. I have this link[2] for your convenience for additional information regarding creating deployment package for Ruby.

As a quick workaround here, I would recommend creating a layer[3] with the json 2.2.0 Gem, Ruby 2.5.0 uses “Amazon Linux" as an Operating System[4], I would recommend you creating the layer on an “Amazon Linux" EC2 instance then upload it to Lambda to avoid any compatibility issues.

Further, I checked on Lambda function using command “puts Gem.loaded_specs[‘json’].version” and I see the environment is indeed using json library 2.3.0.

I hope this was helpful, please don’t hesitate to reach out back to me if you need any additional information, I will be more than happy to assist you further Fabien!

Have a great day !

[1] https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code
[2] https://docs.aws.amazon.com/lambda/latest/dg/ruby-package.html#ruby-package-dependencies
[3] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
[4] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Best regards,

Wassim J.
Amazon Web Services

Maybe it can help other people.

Fabien.

#3

Reading the Jets documentation I found that Jets already use Gem Layer https://rubyonjets.com/docs/extras/gem-layer/

So I should not get this error and I should not have to package a specific layer, shouldn’t I ?

#4

Same thing is happening to me. Unsure how to fix it exactly.

#5

@Fabien I did a fresh deploy and it fixed itself. I too hadn’t deployed in weeks. No idea what caused it or why a fresh bundle/deploy suddenly fixed it, but give that a try.

#6

Hi all,

I get two other answers from AWS support:

1/
Hi Fabien,

Thanks for reaching back out to me and confirming you were indeed including this in the layer, As per the privacy policy agreed with AWS[1] we don’t have any visibility into your internal data, therefore we can’t see the content of the layer/ Lambda code.

I went ahead and created a layer with the json 2.2.0 gem and I confirm the function kept referencing the json 2.3.0 instead. I believe this is an unexpected behavior. I have escalated this to the Lambda developers team to further investigate into this issue as they do have permission on the service more than I do. I assigned your ticket the highest priority.

I will reach out back to you as soon as I have additional information, while I am working on this issue internally with the Lambda team, please don’t hesitate to contact me again if you need any additional information, I will be more than happy to assist you further !

Talk to you soon !

Best regards,

Wassim J.
Amazon Web Services

2/
Hi Fabien,

Much appreciate your patience!

The Lambda developer team confirmed they are enforcing now the use of JSON 2.3.0, this was as part of a security update that was recently introduced to Lambda. As this is used by the runtime, there is no way ,unfortunately, to use JSON 2.2.0, the only workaround for this at the present moment is to update your environment to use JSON 2.3.0. Also, as the environment will be forced to use whatever JSON version used by the runtime, you can rely on that version instead of importing it in your environment. I do appreciate this may be a bit of work to do on your end to get back the function up and running , however this is really recommended for security reasons.

Further, I have sent a feedback to the team to send communication for similar security updates in advance to avoid. Apologies for inconvenience may this caused to you.

Please, let me know if you have any queries or concerns and I will pass them on to the Lambda developers team.

Have a nice day !

Best regards,

Wassim J.
Amazon Web Services

So to sum-up AWS lambda imposes to use specific versions of component and you can’t do anything…

It is really annoying in a production environment!!!

#7

Thank very much for the details.

Wow. Feel like ran into something similar a week ago or so with ruby, lambda, and json, and lono actually. Using lono for lambda functions also. At the time, had to removed the json gem from the Gemfile and that allows ruby to use whatever system-level json gem was installed on the AWS Lambda runtime. So it was more “free” at that point. Guessing AWS is now straight up vendorizing json 2.3.0 and enforcing it.

Bummer about this whole thing. Especially, since it’s was on production. Am also a little sympathetic to the AWS Lambda team. I vaguely remember having ruby and json issues a long time ago. At some point, it seems like ruby team itself decided to include json into stdlib core. That seem to cause issues. Don’t even remember the exact details because it was pretty confusing at the time. Am also sympathetic to the Ruby core team. It seems like a difficult problem.

Thanks for posting again. It’s useful.

#8

This happened to the Jets app that BoltOps is running. Actually run quite a few of them now. Looks like AWS is rolled out the update more generally on 4/16/2020. Thanks to this report knew exactly what to do! Upgrade json 2.3. Bummer again about it!

Also, this PR should migrate the issue somewhat https://github.com/tongueroo/jets/pull/470 since it removes json from Gemfile.lock. Though projects or other gems can still add json, it at least migrates it.

#9

I just encountered this problem and ended up adding
gem "json", "2.3.1" to my project gemfile. This seems to have worked, but I’m not sure if that is the recommended solution. Is there a preferrable way to fix this json issue?

#10

This issue has come up again. Jets is bundling a version of the json gem into the gem layer, due to a transitive dependency. This is fine until AWS changes the version included in the ruby runtime, then deployed Jets lambdas begin failing.

“You have already activated json 2.5.1, but your Gemfile requires json 2.3.1. Prepending bundle exec to your command may solve this.”

The json gem needs to be removed from the Gemfile.lock to avoid this conflict. First thought, however undesirable, is to manually edit the Gemfile.lock. But jets build regenerates the lock file as part of the build process when running bundle install, so editing the file is not an option.

This is a serious issue causing production systems to fail without notice.

Any thoughts on how to deal with this problem?

#11

Jets v2.3.16 and above removes json from jets itself. Details here: https://github.com/boltops-tools/jets/pull/470

So believe, json should not be in Gemfile.lock , at least from Jets and it’s built-in dependencies. :face_with_monocle:

However, if there’s another gem in your Gemfile that includes json as a dependency then json can be included in the Gemfile.lock.

Wondering, maybe take a look at your Gemfile.lock and see what gem is causing the json gem to be included. Then you’ll need to remove that parent gem or replace it with a gem that doesn’t include json. Hope that helps.

#12

Would it help if you added --deployment to the bundle install command ?

env bundle install --deployment --path #{cache_area}/vendor/gems --without development test

It would be best if Jets did not use the shared/system gems.

Perhaps setting BUNDLE_DISABLE_SHARED_GEMS true in the bundle config would stop the conflicts.

Thanks,
Tom

#13

Dont think the --deployment option will help. Jets already writes a .bundle/config that results in that option being used. Also, jets bundles all the gems up to create the Lambda layer. It doesn’t use system gems in that sense. Unsure if BUNDLE_DISABLE_SHARED_GEMS will help. Have seen bundler and different versions sometimes be a bit inconsistent.

Would probably check the Gemfile.lock again and confirm if that has includes the json gem. If so the key would be finding which gem is causing the json gem to be included. Hope that helps.

#14

I know which gem is causing the json gem to be included … and I need that gem so can’t remove it.

I also notice that jets build is downloading the json gem that AWS uses and copying it into the distribution. Not sure why you would do that if it’s part of the ruby runtime. This also results in having two different version of the gem in stage/opt/ruby/gems/2.5.0/gems if my gem file also includes the json gem.

#15

Jets uses bundler, which bundles and includes whatever gems is in the Gemfile.lock. So really only way is to get that gem out of your Gemfile. It’s explained here: https://github.com/boltops-tools/jets/pull/470

One idea/option is to fork that gem, remove the json depedency, and use the fork. IE:

Gemfile:

gem "gemname", git: "https://github.com/yourfork/gemname"

This avoids the json gem from being included in the Gemfile.lock Hope that helps. :peace_symbol:️

#16

I appreciate your help.

I do feel this is a shortcoming of the Jets framework. There is really nothing wrong with gems that spec a version of json. Jets is not sufficiently isolating dependencies. If AWS adds gems to the runtime this issue can grow beyond the json gem. How many gems are incompatible with Jets due to this issue?

#17

It’s a bit more complex. The json gem seems to be a unique case, in that it’s so useful that runtime authors use it as part of their runtime. Whether it’s the core Ruby runtime itself or AWS Lambda.

When dug into this a while back, it seems like the Ruby core team included the json gem into its Ruby runtime itself for a while. Seems like this cause issues in the Ruby world and core team and moved json out. It went back and forth like this for a while. AWS Lambda went through the same process. Then AWS Lambda team sort of had to bundle specific versions of json due to security issues that cropped up with the json gem. As mentioned earlier in this thread, am actually sympathetic to both teams. They’re trying.

Using bundler to ensure deployed gems are controlled, pretty much have to. Bundler calculates the dependency tree, and if the AWS runtime version cannot resolve with the version in Gemfile.lock, bundler ends up in this state.

RE: How many gems are incompatible with Jets due to this issue?

FWIW, have only seen json with this issue. It seems to be quite unique. Though, like you mention, if AWS decides to add and lock additional gems, it could cause issues. AWS would generally try to keep their runtime as light as possible, though. Of course, anything can happen. You can also build and use your own custom runtime, but that’s more effort than it’s worth.:v:️

#18

I appreciate all you’ve said. The workaround to fork the library and remove the dependency is a good short term solution, but my concern is larger than just this one gem.

AWS could at any time, and without notice, introduce other gems to the runtime that may cause production system failure. It is necessary to isolate an application from the lambda runtime to achieve reliability.

This is from the AWS docs regarding Lambda best practices.

https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Blockquote Control the dependencies in your function’s deployment package. The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js and Python runtimes (a full list can be found here: Lambda runtimes. To enable the latest set of features and security updates, Lambda will periodically update these libraries. These updates may introduce subtle changes to the behavior of your Lambda function. To have full control of the dependencies your function uses, package all of your dependencies with your deployment package.

AWS’s own guidance recommends isolating the function dependencies from the runtime.

I’m looking forward to Lambda Container Image support in Jets, as this would also address the issue.

I want to thank you and all who contributed to Jets. Overall experience has been very good.

Thanks,
Tom

#19

Jets already helps to control the dependencies by using bundler. :unamused: Wondering if you see that the issue is a project-specific and AWS lambda issue. :thinking: Let’s remove Jets out of the picture for the moment. :face_with_monocle:

If the project were a pure Lambda function (no Jets) with a Gemfile to control dependencies, per AWS docs. If AWS locks json or any gem, the same problem exists.

It’s because the project (no Jets) wants to control a json version with a Gemfile and AWS wants to control the json version too. This is the crux of the issue. If your project says use json v1 and AWS says use json v2, it’ll conflict and there’s not much Jets can do about that.

#20

No, I guess I don’t see that. The product is not and can not be made reliable because of this issue.

I’ve implemented lambdas in other languages (python) where I initially had failures due to dependencies on the runtime. In python I was able to isolate the dependencies. Ruby lambdas are obviously different, and i’m not clear how to fix the issue, but not ready to accept it can not be fixed.

I’ll keep looking into the problem, and if I have any further thoughts on how to resolve it I’ll share.

:peace_symbol: :v: