1 SQS Queue / Lambda per job

Jets
1

In jets 6, is there a way to have jets create a single lambda/queue per job?

We have jobs that can be in the millions of events per hour, a single job/queue makes this impossible to throttle specific jobs and prevents seperation of concerns (job 1 gets backed up blocking job 2).

2

Interesting, separation of concerns is a good reason. Ok, would like to add the ability for multiple queues. Unsure exactly when will get to it. Am focusing more on Jets recently. :+1:

Currently, Jets only deploys one JetsQueue and lambda function to process the queue events.

Concurrency Cheatsheet

Also, this might be helpful. This shows you how to control at least the concurrency of the jets-queue_event-handle lambda function that handles processing for the JetsQueueEvent.

jets concurrency:info
jets concurrency:get -n jets-queue_event-handle
jets concurrency:set -n jets-queue_event-handle --reserved 200

Example Session

❯ jets concurrency:get -n jets-queue_event-handle
Settings for Function: rails-dev-jets-queue_event-handle
Reserved concurreny: 5
Provisioned concurrency: not set

❯ jets concurrency:set -n jets-queue_event-handle --reserved 200
Will update the concurrency settings for rails-dev-jets-queue_event-handle
Are you sure? (y/N) y
Updating concurrency settings for rails-dev-jets-queue_event-handle
Set reserved concurrency to 200

❯ jets concurrency:get -n jets-queue_event-handle
Settings for Function: rails-dev-jets-queue_event-handle
Reserved concurrency: 200
Provisioned concurrency: not set

❯ jets concurrency:info
Concurrency for rails-dev
+---------------------------+----------+
|         Function          | Reserved |
+---------------------------+----------+
| controller                | 25       |
| jets-prewarm_event-handle | 2        |
| jets-queue_event-handle   | 200      |
| total                     | 227      |
+---------------------------+----------+
Account Limits
  Concurrent Executions: 1000
  Unreserved Concurrent Executions: 530

❯

And to tail and monitoring the queue event processing

❯ jets logs -f -n jets-queue_event-handle

P.S. Also, it seems like you were able to enable the jets job feature and deploy the ActiveSupport :jets_job adapter. :+1: There are some edge cases, which we will be looking at soon. Thanks.

3

Sounds good thanks! Currently converting some apps from jets 5 to rails/jets 6. It looks like we can handle the above situation with the jets events (old jets 5 jobs) until this is available.

4

Jets now supports multiple SQS queues.

Docs: Jets Job Multiple Queues - Jets Serverless Deployment Service

There’s a config.job.additional_queues config available now.

Example

Jets.deploy.configure do
  config.job.enable = true

  config.job.default_queue.lambda.reserved_concurrency = 5

# Add additional queues. Note: default queue is always created
  config.job.additional_queues = %w[urgent low_priority]

  # Adjust settings for each queue
  config.job.queues.low_priority.lambda.reserved_concurrency = 2
  config.job.queues.urgent.lambda.memory_size = 2048
  config.job.queues.urgent.lambda.reserved_concurrency = 10
end

To use, you need to update to the latest jets and jets-rails. At least:

Gemfile

gem "jets", ">= 6.0.2"
gem "jets-rails", ">= 1.1"

To update and confirm

bundle update
bundle info jets
bundle info jets-rails