Issue with native gems from darwin

jhottenstein · January 8, 2021, 4:03pm

We ran into an issue yesterday when we upgraded nokogiri to 1.11.0.

“Could not find nokogiri-1.11.0 in any of the sources”

Looking at the gem layer, two things stood out to me: the gem directory name included darwin and there was no nokogiri in the extensions directory.

$ find prod-api-services-gems-4798ad2d-0411-4225-b627-95b881850be0 -type directory -name '*nokogiri*'
prod-api-services-gems-4798ad2d-0411-4225-b627-95b881850be0/ruby/gems/2.5.0/gems/nokogiri-1.11.0-x86_64-darwin
prod-api-services-gems-4798ad2d-0411-4225-b627-95b881850be0/ruby/gems/2.5.0/gems/nokogiri-1.11.0-x86_64-darwin/ext/nokogiri
prod-api-services-gems-4798ad2d-0411-4225-b627-95b881850be0/ruby/gems/2.5.0/gems/nokogiri-1.11.0-x86_64-darwin/lib/nokogiri

I tried to force the use of compiled gems (via bundle config set force_ruby_platform true). This fixed the gem directory but added the wrong platform for the extensions:

$ find prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84 -type directory -name '*nokogiri*'
prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84/ruby/gems/2.5.0/extensions/x86_64-darwin-19/2.5.0/nokogiri-1.11.1
prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84/ruby/gems/2.5.0/extensions/x86_64-darwin-19/2.5.0/nokogiri-1.11.1/nokogiri
prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84/ruby/gems/2.5.0/gems/nokogiri-1.11.1
prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84/ruby/gems/2.5.0/gems/nokogiri-1.11.1/ext/nokogiri
prod-api-services-gems-81a44894-60e9-455d-9863-b123823bce84/ruby/gems/2.5.0/gems/nokogiri-1.11.1/lib/nokogiri

I am deploying from os x.

dkobia · January 22, 2021, 10:02pm

I encountered the same issue, probably with nokogiri (1.11.1-x86_64-darwin). It is quite painful to debug too because Custom::BasePathMapping hangs up the cloudformation stack and it takes up to an hour for it to release the stack. The jets-base-path lambda has recurring errors.

{
    "errorMessage": "Could not find nokogiri-1.11.1 in any of the sources",
    "errorType": "Init<Bundler::GemNotFound>",
    "stackTrace": [
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/spec_set.rb:87:in `block in materialize'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/spec_set.rb:81:in `map!'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/spec_set.rb:81:in `materialize'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/definition.rb:175:in `specs'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/definition.rb:245:in `specs_for'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/definition.rb:227:in `requested_specs'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/runtime.rb:91:in `block in definition_method'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/runtime.rb:20:in `setup'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler.rb:148:in `setup'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/setup.rb:20:in `block in <top (required)>'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/ui/shell.rb:136:in `with_level'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/ui/shell.rb:88:in `silence'",
        "/var/runtime/gems/bundler-2.2.5/lib/bundler/setup.rb:20:in `<top (required)>'",
        "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
        "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
        "/var/task/handlers/functions/jets/base_path.rb:1:in `<top (required)>'",
        "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
        "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
    ]
}

dkobia · January 22, 2021, 10:58pm

FYI, I was able to resolve this by building a custom lambda layer with - gem 'nokogiri', '~> 1.11.1'

wafendy · January 28, 2021, 11:57am

Hi, could you elaborate more on how to build a custom lambda layer with gem 'nokogiri', '~> 1.11.1'? Perhaps link on how to go about doing this? Thanks.

tung · January 28, 2021, 6:19pm

Bummer. Had to do with the way gem detection for the nokogiri in these later versions. Fixed

Here’s an example app, note it’s on the jets-v2 branch: https://github.com/tongueroo/jets-nokogiri-demo/blob/jets-v2/app/controllers/nokogiri_controller.rb

Screenshot:

Screen Shot 2021-01-28 at 10.15.21 AM

Note, app has already been deleted.

dkobia · January 29, 2021, 11:00am

@tung you mention that this issue was resolved with https://github.com/boltops-tools/jets/pull/524 in 3.0.1 yet this issue is happening in 3.0.2.

tung · January 29, 2021, 4:28pm

Issue was actually fixed first in Jets v3 a while back. Just tested it again with the demo nokogiri project:

github.com

tongueroo/jets-nokogiri-demo/blob/master/app/controllers/nokogiri_controller.rb

require 'open-uri'

class NokogiriController < ApplicationController
  def index
    # Fetch and parse HTML document
    doc = Nokogiri::HTML(URI.open('https://nokogiri.org/tutorials/installing_nokogiri.html'))

    # Search for nodes by css
    content = nil
    doc.css('nav ul.menu li a', 'article h2').each do |link|
      content = link.content
    end

    render json: {
      content: content,
      nokogiri_version: Nokogiri::VERSION,
      jets_version: Jets::VERSION,
      ruby_version: RUBY_VERSION,
    }
  end

This file has been truncated. show original

Note: The master branch in demo project is using Jets v3. Screenshot:

Screen Shot 2021-01-29 at 8.22.10 AM

Unsure. Maybe double check with the demo project and also do a rm -rf /tmp/jets in case.