Hello team,
I've been working on a project with a very large database. I need to fetch all records using an API endpoint within a custom connector SDK. This third-party API has some limitations: it only returns 1000 results per request and uses an offset parameter to fetch the next set of records. For example, with an offset value of 0, it returns records 1-1000; with an offset value of 1, it returns records 1001-2000, and so on.
To fetch all records, I implemented a loop that increments the offset value by 1 with each API call until all records are retrieved. This approach worked for fetching around 3000 records, but the connector started giving an "INTERNAL SERVER ERROR" when trying to fetch more records due to the long execution time.
I contacted chat support, and they suggested using a multistep action. However, this doesn't seem appropriate because it relies on asynchronous processing from the third-party API, which returns a job ID.
Can anyone please suggest what to do in this case?
