topic Re: Bring 50k from SF in Workato Pros Discussion Board

Bring 50k from SF

daniel-chupak — Thu, 27 May 2021 22:23:11 GMT

Hi,

Has anyone tried to bring about 50k records from SF using the custom action and pagination method?

The issue is that to collect all the data I use the 'Accumulate to list' action and it seems to take too much time which makes me exceed the 90 minutes limit.

I was wondering if anyone has found another solution?

Re: Bring 50k from SF

ajorde — Thu, 27 May 2021 22:32:48 GMT

If I need to grab more than 2K lines from SF, I use the bulk > csv action (Just make sure that recipe is started so that it wakes up 5 minutes later to finish the bulk action). Also, you should use mapper / common data models functionality rather than accumulate to list to bulk map the data in one action rather than looping through each record.

Re: Bring 50k from SF

mroldanvega — Thu, 27 May 2021 22:59:32 GMT

Bulk is the way to go, I agree with Amy Jorde . This is what we use for these types of use cases. Keep in mind that Bulk doesn't seem to work in Test mode, only when the recipe is active.

Re: Bring 50k from SF

daniel-chupak — Thu, 27 May 2021 23:49:17 GMT

Alright! thank you all. 🙂

Re: Bring 50k from SF

john-colburn — Fri, 28 May 2021 00:13:05 GMT

I ran into a similar problem downloading a large SF report and looping through each line. You need to add an IF statement at the top of your loop to check the count of the index you are looping through and pause for .5 seconds every 4000-5000 rows. This will allow your job to run without timing out after 90 minutes

Re: Bring 50k from SF

ajorde — Fri, 28 May 2021 03:01:43 GMT

Processing each record individually is going to greatly increase your task usage in Workato... I still definitely recommend using a mapper action rather than looping through each record individually. I use these in almost every recipe to avoid loops wherever possible.

https://docs.workato.com/recipes/tasks.html

https://docs.workato.com/connectors/mapper.html#mapper-by-workato

https://docs.workato.com/features/common-data-model.html

Re: Bring 50k from SF

sanjay-rathod1 — Fri, 28 May 2021 13:30:25 GMT

I had also faced the same type of scenario while used Accumulated List within a Repeat action. One of the solutions is to use the "Search record in Bulk using SOQL query" action to get the exact data that you want. This will return CSV as result. Another possible solution is, try to use "mapper action" or "Create List Variable" rather than accumulated list.