How to Manage Concurrency and Prioritize Recipes Effectively

[tstepnik]

Hi 🙂

I'm working on a project where we integrate multiple systems, including NetSuite, Salesforce, and Snowflake. NetSuite has a limitation on the number of concurrent transactions, which is currently 5. Because of this, there’s a high risk of recipe failures, even when retrying them three times.

To mitigate this, the best solution would be to queue the recipes—is this even possible?

Our current license has a "1 concurrency limit"—what exactly does this mean? I read that it prevents the same recipe from running multiple times simultaneously, but what about different recipes? For example, if I have 20 different recipes, can they run in parallel with this limit, while only identical recipes are restricted? Or does this limit apply to all active recipes, allowing only one recipe to run at a time?

I assume it’s the latter, meaning only one recipe runs at a time. Given this, is there a way to prioritize some recipes over others? For example, let's say a recipe is currently running, and another recipe (X) enters the queue. After a while, a Y recipe is also added. The execution order looks like this:

Currently running recipe
Recipe X
Recipe Y
Is there a way to ensure that recipe Y is always prioritized over recipes like X? For instance, even if there are 15 "X" recipes in the queue, whenever a new "Y" recipe arrives, it moves to the front.

How do you handle such cases? Also, what do you do when a recipe fails after three retries? Do you store the Workato log in one of the systems, or is there a way in Workato to automatically rerun failed recipes?

Thanks in advance for your insights!😊

[mppowe]

Hi @tstepnik , I don't know about your license restrictions in particular, but I believe it may be referring to the concurrency setting on your specific recipes.  If you open a recipe and then go to "Settings" > General and scroll down, you'll see a section called "Max concurrency".  This is the max number of jobs that can run at the same time for this specific recipe.  I believe under the textbox it'll have text that reads something like "You can assign up to # concurrency to this recipe".  Do you see a 1 there?

I'm not aware of a way to handle recipe priority OOTB, other than within a recipe having them always check some Lookup table that defines their priority, then querying for running jobs using the RecipeOps connector but I feel like it would get messy and add task-usage overhead to every recipe.  But I don't know that it's necessary if the throttling you'll experience is just for specific recipes being invoked simultaneously, not ANY recipe.

Lastly, I do believe there's a RecipeOps trigger for failed jobs where you can handle them accordingly.  There's also an Accelerator pack that Workato can share with you that has a more sophisticated way of tracking failures and automatically responding to them.  We explored it but it was a bit much for what we were doing at the time.  For us we just use the Monitor block and child recipes bubble errors up and parent recipes open an incident in ServiceNow for further research/resolution.

Hope this helps