I am not sure of your use-case, but for best performance and scalability, store date and other columns that you want to search/lookup your data in a separate column and all the metadata in a JSON or XML object under a data column.Eg: If you currenctl...
The design that we use is to move the logic to a callable recipe and perform one job per page of the API call. When all the records in the page are processed, call the same recipe with the next page URL (you can call callable recipe xyz). Stop the ex...
