Hi Jason. This is Jonar, we worked on a couple of implementations last year.
So there are a lot of good answers here. I would just like to add that one thing we felt worked well was to let a callable recipe handle the error handling and retry logic. This callable can work independently and decide on the proper logic according to what kind of error you experienced in the main recipe and some qualifiers you can set into a lookup table, with which you can decide if you should rerun it immediately, wait for a specified amount of time, or stop rerunning when you reach a certain number. You then use the recipe ops action to rerun the job you identified.
Honestly, it will need a bit of design and prep work, and this is mostly done as part of a holistic error handling framework for all your recipes. but we found that once the initial hurdle of setting it up is completed, it is well worth the effort.
