Hi there,
I have an issue which is more likely related to my bad understanding of SQS queues and Lambda than it is an issue, so I’m creating a question here:
I create a bunch of models in my database (postgres if it’s important) and when I create them I send their ids to SQS queue. I have another job that reads from this queue and it all works perfectly and is tested.
That is, it works good until I hit an outside API limit. Each job calls an API which has a limit of 5 parallel calls, so all my jobs that don’t fit in this (that comes 6th and so on…) receive nil from the API and finish.
I’m wondering if there is a smarter way of designing this system and I would welcome any speculation in that direction, but what would help me instantly would be an advice how to limit pulling from SQS queue.
So to reiterate - I have hundreds SQS messages and each one should call external API which has a limit of 5 parallel calls. How to best implement this limit?
Thanks!