Want to get a sense how you deal with pagination. For me, my approach is to leverage CSV in Box/Dropbox to park the data there + append CSV via callable recipes.
Box: nice that I can hard code the file id, which doesn’t change if the file path changes
Dropbox: have to hard cod file path. If the path changes, then it breaks. (feature request please)
Lookup table: there is no bulk upload, so an additional repeat action is required to put data there (feature request please 😀)
Google Drive: it seems to allow uploading multiple files with the same name (huh....?). I guess I am coming from the Box user experience and shocked to see this.
Google Sheet: not a fan of it but happy to see if anyone uses it. What do u use it for?
Thanks pros for sharing your experience. Have a nice week!
If I wanted to accumulate all of a paginated API’s records in a single recipe using only Workato features, I would begin by using a Workato list (which can have up to 50,000 items). You can actually concatenate two lists together in a single step with a Ruby action as demonstrated in this recipe, https://app.workato.com/recipes/1456290?st=13ab59, so you wouldn’t have to iterate through every record in a page to add them to the accumulated list one at a time.
Another approach would be to use Workato collections (which can have up to 50,000 records), https://docs.workato.com/features/collection.html, which are essentially in memory database tables that last for the duration of a job. Collections require the ETL/ELT Bulk Data Processing platform add-on.
Hmn...whoa... their opened a new world for me. I am going to try the Ruby trick definitely.
By the way, sometimes parking the data elsewhere like Box may not be a bad idea either. This way, other recipes could also use it - if it doesn’t mind a slightly dated data
So, with your proposed method, it would be
1. Master recipe to do initial call, send the list to callable one
2. Callable one: received the next URL, list from master, make another call, Concat the two list and call callable 2
3. Callable 2: do exact the same thing
Not vetted but...I will give it a try 😍