One strategy we have used for large lists that may require lots of calculations (like analysis on all POs for the entire year) is to do something like this:
Recipe 1 (Controller Recipe) - determines number of pages and kicks off the individual processes (or for an unknown number of pages monitors for when the process is complete)
Recipe 2 (Grunt Work Recipe) - callable recipe that gets each page of data and stores data in a temporary location (potentially doing some calculation/transformation)
Recipe 3 (Clean-up) - once all the pages are processed the controller recipe will start the clean-up recipe to create the final deliverable
We have built recursive-style processes using workato pub/sub and we have daisy-chained multiple processes of these together to do multiple large scale extractions and transformations.
Generating the 1099 report is a good example. This process has multiple large scale extractions that are dependent on each other. We had to get:
1. all vendors that are 1099-able out of NetSuite
2. all accounts in NetSuite that are designated as 1099-able
3. all bills for the vendors
4. all payments for those bills
5. calculate the percentage of lines on the bill that were 1099-able, and what percentage of the payment was applied to each line to get the 1099 amount
6. after all the calculations were complete - do the roll up to calculate the 1099 amount per payment, per bill, and eventually per vendor.
We didn't know the number of vendors, bill, or payments and had to track all of this independently (in addition to dealing with delays in getting data out of netsuite and doing calculations). The process was 100% automated (click "Start" and then 20 minutes later you received the report of all 1099s for vendors).
