Working w/ Bulk Data in NetSuite

[jessica-lie]

[Nov 25, 2020] Joe Blanchett (Product Manager, Finance Systems at Seat Geek) posted:

We have a few recipes where we need to work with > 200 rows of data in NetSuite. The search records call maxes out at 200 results on the first page, so the only way I've come up with to get all results involves multiple steps that iterate through each page, and accumulating the results into a workato variable list along the way. As you can see below, this method involves 10 steps just to get all the data that's needed. Has anyone come up with a more elegant way to get high volumes of data from NetSuite? 

[jessica-lie]

[Dec 2, 2020] Amy Jorde (Senior Integration Engineer at Malwarebytes) replied:

If you have an OPA, you can also connect to the NetSuite JDBC driver to pull higher volumes of data. That said, this might not be completely practical for a regular integration use case without parking the data in a data warehouse, as it does pull the object ids rather than field values for the related objects (like subsidiary, location, custom list fields), so some work would be required to piece the details back together. 

We send the raw NetSuite data into Snowflake, and then I piece it back together using SQL for use cases with this type of requirements (like sending transaction data to Anaplan, for example).