topic Re: Retry with exponential backoff in Workato Pros Discussion Board

Retry with exponential backoff

jason1 — Fri, 29 Jan 2021 00:03:15 GMT

Hi, is there a simple way to build in exponential backoff when hit with a rate limit API error? The error handling step allows for retries, but there doesn't seem to be a way to set custom wait intervals in between tries.

Perhaps something that can be implemented with the new Wait step, but curious how others have tackled this.

Thanks!

Re: Retry with exponential backoff

jakob-maynard — Fri, 29 Jan 2021 00:12:40 GMT

We've implemented a sleeper utility, so when we hit rate limit errors, we sychronously call the util with an x minute wait, then when it returns, retry the initial call.

Re: Retry with exponential backoff

nguyen-lu — Fri, 29 Jan 2021 00:14:27 GMT

Thanks Jakob. We are faced with the same challenge as Jason. Could you please elaborate on the sleeper utility?

Re: Retry with exponential backoff

jakob-maynard — Fri, 29 Jan 2021 00:18:15 GMT

Yeah, so it's a very simple recipe that takes in an integer, then uses a Execute Ruby Code step with ruby's sleep function. Workato has it's own Wait function on the Scheduler app, but we've noticed for longer wait times, it tends to stall and won't return for much longer than the initial time.

Re: Retry with exponential backoff

jason1 — Fri, 29 Jan 2021 00:19:21 GMT

Jakob Maynard Thank you. Is this invoked within the on error step with a set number of retries?

Re: Retry with exponential backoff

george4 — Fri, 29 Jan 2021 00:20:13 GMT

It's very simple Jason Jho here is the random wait time in the specific interval - https://share.getcloudapp.com/GGu6XznD

Re: Retry with exponential backoff

george4 — Fri, 29 Jan 2021 00:23:05 GMT

Also, sometimes I use thing like that to pass the wait limit in seconds - https://share.getcloudapp.com/P8ukd7l2

Re: Retry with exponential backoff

jakob-maynard — Fri, 29 Jan 2021 00:24:12 GMT

The way we've implemented it was wrapping the whole in a for loop with a set number of attempts, and if the call passes, skip all the steps in the loop, but if it doesn't and it's the rate limit, start the wait.

Re: Retry with exponential backoff

jdasilva — Fri, 29 Jan 2021 22:15:09 GMT

Here is a related enhancement suggestion that I posted: https://support.workato.com/support/discussions/topics/1000100031

The suggestion to leverage a FOR EACH is a great one. We chose to build in retries to the error handling routine that gets called within the On Error to send notifications. We have configuration in a Lookup table for notification channel as well as these retries. If exponential retries are configured then we write the job to a Lookup table then leverage a recipe that runs every 5 minutes to re-run the job with Recipe Ops. This has worked well for systems with regular maintainence windows of 1-2 hours.

Re: Retry with exponential backoff

harish — Sat, 30 Jan 2021 03:07:32 GMT

Have you tried using try catch blocks? The have retry logic built in to them..

Re: Retry with exponential backoff

jonar-ramos — Mon, 08 Feb 2021 16:38:50 GMT

Hi Jason. This is Jonar, we worked on a couple of implementations last year.

So there are a lot of good answers here. I would just like to add that one thing we felt worked well was to let a callable recipe handle the error handling and retry logic. This callable can work independently and decide on the proper logic according to what kind of error you experienced in the main recipe and some qualifiers you can set into a lookup table, with which you can decide if you should rerun it immediately, wait for a specified amount of time, or stop rerunning when you reach a certain number. You then use the recipe ops action to rerun the job you identified.

Honestly, it will need a bit of design and prep work, and this is mostly done as part of a holistic error handling framework for all your recipes. but we found that once the initial hurdle of setting it up is completed, it is well worth the effort.