Working around the 90 minute limit

[jessica-lie]

[Dec 3, 2020] Mike Power (CRM Enterprise Architect at American University) posted:

Good afternoon all,

I have a parent recipe that retrieves records from an outside source, then loops through each transaction and calls a child recipe to process that individual transaction.  When all are done, it marks a flag that all transactions are complete.

Currently I make the calls to the child recipe synchronously so that when I get to the last steps in the parent recipe, I know they’re all done.  Recently I started hitting that 90 minute timeout in Workato.

So I’m considering how to re-do things.

My initial thoughts are to call the child recipe asynchronously (though it seems to only allow it up to 5 times in parallel, where I may have a couple thousand transactions to get through), then at the end of the parent recipe update either a Lookup table or maybe there’s something in the pub/sub I can utilize (though I haven’t read up on that yet).  Then maybe each child recipe, when they finish, they update either the Lookup or pub/sub so that a recipe gets triggered to compare the count completed with the total count (recorded by the parent recipe when it finished), and only when they equal does a triggered recipe update the flag.

Clearly this is half-baked and I’m just starting to think it through.  I’d welcome any thoughts from anyone who has already cracked this nut before. 

[jessica-lie]

[Dec 4, 2020] Deven Maru (CSM at Workato) replied:

Mike,

Take a look at the wait featurehttps://docs.workato.com/features/callable-recipes/wait-for-async-action.html. This way you'll avoid manual callable recipe completion synchronization.

[jessica-lie]

[Dec 4, 2020] Mike Power (CRM Enterprise Architect at American University) replied:

Thanks Deven.  I did consider that but if I can only execute 5 of my child workflows at a time (I’m assuming the “Concurrency” setting applies regardless of sync/async), I still worry about the parent recipe waiting for all of the child recipes to finish within the 90 minute allotment.  I had hoped to get away from the 90 minute limit completely.  Do you think there’s another way to go about that?

[jessica-lie]

[Dec 10, 2020] Deven Maru (CSM at Workato) replied:

Posting rest of the conversation here after the above message was posted by Mike.

[DEVEN] Mike, there's a timeout limit input field in the Wait action of Callable recipe. You can set it to more than90mins and try it out. I'm including Gabriel who can provide more information.

[MIKE] Thanks Deven.  Doesn’t that timeout limit mean that the parent recipe which is awaiting the child recipe will wait longer than 90 minutes, but the parent recipe itself will still get timed out after 90 minutes?

[DEVEN] No, the parent recipe will "sleep" at the Wait step waiting for your callable recipes to complete. The parent recipe timeout clock "sleeps" as well. When the wait is over, the timeout clock resumes. So, to answer your question, the parent recipe should not get timed out.

[jessica-lie]

[Dec 11, 2020] Mike Power (CRM Enterprise Architect at American University) replied:

Thanks.  I’ve set the child recipe to be asynchronously called (concurrency = 5) and then I calculate a wait time based on the number of transactions.

Worked ok in Test so I’ll move it to Prod soon, and hopefully all will be well!