Pagination - Box, Dropbox, Google Sheet, Google Drive, Lookup Tables, etc

[gordonhuworkato]

Hey Pros,

Want to get a sense how you deal with pagination. For me, my approach is to leverage CSV in Box/Dropbox to park the data there + append CSV via callable recipes.

Box: nice that I can hard code the file id, which doesn’t change if the file path changes

Dropbox: have to hard cod file path. If the path changes, then it breaks. (feature request please)

Lookup table: there is no bulk upload, so an additional repeat action is required to put data there (feature request please 😀)

Google Drive: it seems to allow uploading multiple files with the same name (huh....?). I guess I am coming from the Box user experience and shocked to see this.

Google Sheet: not a fan of it but happy to see if anyone uses it. What do u use it for?

Thanks pros for sharing your experience. Have a nice week!

Gordon

[ben-urbanski]

Hello Gordon,

I assume you’re trying to get all records from an API that requires you to make multiple paginated requests. How many total records do you need to get?

Regards,

[gordonhuworkato]

Hello, we have close to 2000 Okta users. Right now I use one master recipe + two callable to call each other.

Gordon 

[ben-urbanski]

Thanks, Gordon.

If I wanted to accumulate all of a paginated API’s records in a single recipe using only Workato features, I would begin by using a Workato list (which can have up to 50,000 items). You can actually concatenate two lists together in a single step with a Ruby action as demonstrated in this recipe, https://app.workato.com/recipes/1456290?st=13ab59, so you wouldn’t have to iterate through every record in a page to add them to the accumulated list one at a time.

Another approach would be to use Workato collections (which can have up to 50,000 records), https://docs.workato.com/features/collection.html, which are essentially in memory database tables that last for the duration of a job. Collections require the ETL/ELT Bulk Data Processing platform add-on.

Regards,

[gordonhuworkato]

Hmn...whoa... their opened a new world for me. I am going to try the Ruby trick definitely.

By the way, sometimes parking the data elsewhere like Box may not be a bad idea either. This way, other recipes could also use it - if it doesn’t mind a slightly dated data

So, with your proposed method, it would be

1. Master recipe to do initial call, send the list to callable one

2. Callable one: received the next URL, list from master, make another call, Concat the two list and call callable 2

3. Callable 2: do exact the same thing

—
Not vetted but...I will give it a try 😍