โ12-21-2021 03:05 PM
I'm running into a couple cases where I need to process a lot of data out of an API (like all projects in Jira, or all channels in Slack), and these endpoints tend to be paginated.
I've been playing around with the repeat action, but that takes a fixed length list to iterate, and with paginated data you often don't know how many interations you're going to have to do. I guess I could set up a helper list with 50000 iterations, and stop when I'm done, but it feels like there should be a better solution.
I haven't been able to find any good design documentation/tutorials on how to work with paginated data in Workato. Does anyone know where to find that?
Thanks, Michiel
โ12-21-2021 03:09 PM
often the API will tell you total records as well as the page size. You can use this in the create list helper step, as a formulia. total/pagesize. BTW it rounds correctly, UP.
โ12-21-2021 03:29 PM
If the API tells you how many records, or better yet, how many pages then you can do this:
1. Call the API once
2. Create a helper list with size = number of pages (or number of records/page size)
3. Iterate through the helper list with a FOR EACH loop
4. Within the for each, call the API and set the page_num = the sequence number of your for each loop
After this if you can do whatever you want, either process the records in the for each loop or add them to the end of a pre-defined list so you can process them when the for each loop completes.
If the API doesn't tell you how many records it's going to return, then that's truly unfortunate, but manageable. You can define a variable as a high number to start with to mimic the number of pages in step 2 above. But then, you can avoid calling the endpoint an unnecessary number of times by saving the latest number of records returned from the endpoint within your for each loop as a variable, and use an IF condition within the for each to only call the endpoint if the most recent number of returned records was > 0, or if the number of returned records is equal to the max page size if you know that information.
โ12-21-2021 03:44 PM
You still got to make sure it can get processed within the allowed timeframe. In this case I think it is 90 minutes or it will just time-out.
โ12-21-2021 03:33 PM
It would be great if Workato offered a WHILE loop to handle this unknown number of pages, but they don't. I suspect they did that to avoid unintentional, long running jobs