topic Processing 100k records from csv file in Workato Pros Discussion Board

Processing 100k records from csv file

Laxman — Wed, 11 Feb 2026 15:43:47 GMT

Hello everyone

Want to process 100k records from csv file and dump them in a database
here i'm getting below error

Error

Job has been shut down by the system. Usually it happens due to high memory consumption. Try reworking the recipe to process less amount of data per step.

in for each the batch size given as 30000

here how the data will be processed to database...?

Re: Processing 100k records from csv file

shivakumara — Wed, 11 Feb 2026 17:51:01 GMT

Hi @Laxman ,

There is no issue with the current recipe design. Although there may not be a documented upper limit for batch size, it is recommended to determine the practical processing threshold through controlled testing.

For example, if testing shows that the system can process up to 5,000 records in a single batch, it would be prudent not to set the batch size exactly at that limit. Instead, configure it slightly lower (e.g., between 4,500 and 4,800 records).

This buffer helps:
Reduce the risk of memory-related failures
Improve job stability under varying data volumes
Handle unexpected payload size variations
Ensure smoother execution in concurrent environments

Operating slightly below the maximum tested threshold provides better reliability and long-term maintainability.

Thanks and Regards,
Shivakumara K A

Re: Processing 100k records from csv file

Laxman — Thu, 12 Feb 2026 05:23:45 GMT

Hi Shiva,

Thanks for your inputs

given the batch size 100 also getting error like job execution process died

is there any best practices to dump the data

Re: Processing 100k records from csv file

rajeshjanapati — Fri, 13 Feb 2026 05:37:05 GMT

Hi @Laxman,

Could you please share the error message once? Also, if the processing of 100 records is failing, we may need to use a child recipe with a concurrency of 5 and the batch size is chosen based on the target application acceptance as @shivakumara mentioned.

Re: Processing 100k records from csv file

Laxman — Fri, 13 Feb 2026 06:38:56 GMT

Hi Rajesh
in step 4 it load the all 100k records at a time which causing memory issue.

Re: Processing 100k records from csv file

rajeshjanapati — Fri, 13 Feb 2026 11:35:41 GMT

Hi @Laxman,

We can able to implement this with do-while action... as it is a CSV File, you can go through a batch-offset approach with GET LINES from CSV File action in Workato FileStorage. Please go through the attached images below, and if you get stuck at a point, please let me know. This solution will work for sure.

Re: Processing 100k records from csv file

Laxman — Fri, 13 Feb 2026 11:54:12 GMT

Hi Rajesh,

now i'm getting 100k rows in output getting db related issues now like bad gateway and Connection reset by peer

Re: Processing 100k records from csv file

rajeshjanapati — Fri, 13 Feb 2026 13:51:02 GMT

@Laxman please follow my previous post and guidance, I have
1. created a variable
2. added DO-WHILE action block
3. inside DO-WHILE block, you need to call GET LINES FROM CSV File action and in the BATCH OFFSET, you need to map the variable you have created in 2nd step.
4. in the while loop condition you need to map the HAS MORE rows pill (as shown in images in previoyus reply)

Hope you got my point?

Re: Processing 100k records from csv file

Laxman — Fri, 13 Feb 2026 14:39:31 GMT

Hi Rajesh,
Thanks for your valuble sujjetions
please share me the recipe if possible that would be much appreciated.

Re: Processing 100k records from csv file

rajeshjanapati — Fri, 13 Feb 2026 16:25:28 GMT

sure, here is the recipe link
recipe link: 100K records handling recipe with csv file from File Storage

Re: Processing 100k records from csv file

Laxman — Fri, 13 Feb 2026 16:54:13 GMT

Hi Rajesh,
Thanks alot for your support actually get lines from csv from workato file storage able to load 50k records at a time we need to take it twice for 100k after that using for each 2 times to dump them in db
let me know is it good practice or not..?

Re: Processing 100k records from csv file

rajeshjanapati — Fri, 13 Feb 2026 17:32:42 GMT

as per my recipe, from 4th step you will get 50K records, after that based on batch size supported by your DB take for each action if doesn't supports 50K records to load into DB or directly map 50K records to DB. hope you got my point?

FOR-EACH action is considered based on batch size supported by DB otherwise not needed.