Processing 100k records from csv file

[Laxman]

Hello everyone

Want to process 100k records from csv file and dump them in a database
here i'm getting below error

Error

Job has been shut down by the system. Usually it happens due to high memory consumption. Try reworking the recipe to process less amount of data per step.

in for each the batch size given as 30000

here how the data will be processed to database...?

[Laxman]

Hi Rajesh,
Thanks alot for your support actually get lines from csv from workato file storage able to load 50k records at a time we need to take it twice for 100k after that using for each 2 times to dump them in db
let me know is it good practice or not..?

[rajeshjanapati]

as per my recipe, from 4th step you will get 50K records, after that based on batch size supported by your DB take for each action if doesn't supports 50K records to load into DB or directly map 50K records to DB. hope you got my point?

FOR-EACH action is considered based on batch size supported by DB otherwise not needed.