We are currently working on a dynamic data migration framework and are developing a custom connector that processes CSV files and sends them to Salesforce without needing a predefined schema.
The Bulk APis of Salesforce has a limit of about 100MB per batch, we plan to chunk the CSV in custom Connector and then submit multiple batches- similar to how the OOB connector does.
Any help or pointers to achieve the same will be much appreciated.
