Processing 100k records from csv file

[Laxman]

Hello everyone

Want to process 100k records from csv file and dump them in a database
here i'm getting below error

Error

Job has been shut down by the system. Usually it happens due to high memory consumption. Try reworking the recipe to process less amount of data per step.

in for each the batch size given as 30000

here how the data will be processed to database...?

[rajeshjanapati]

Hi @Laxman,

We can able to implement this with do-while action... as it is a CSV File, you can go through a batch-offset approach with GET LINES from CSV File action in Workato FileStorage. Please go through the attached images below, and if you get stuck at a point, please let me know. This solution will work for sure.

[Laxman]

Hi Rajesh,

now i'm getting 100k rows in output getting db related issues now like bad gateway and Connection reset by peer 

[Laxman]

Hi Rajesh,
Thanks for your valuble sujjetions 
please share me the recipe if possible that would be much appreciated.

[rajeshjanapati]

@Laxman please follow my previous post and guidance, I have
1. created a variable
2. added DO-WHILE action block
3. inside DO-WHILE block, you need to call GET LINES FROM CSV File action and in the BATCH OFFSET, you need to map the variable you have created in 2nd step.
4. in the while loop condition you need to map the HAS MORE rows pill (as shown in images in previoyus reply)

Hope you got my point?

[rajeshjanapati]

sure, here is the recipe link
recipe link: 100K records handling recipe with csv file from File Storage