Ruby Connector file size limit

vascoeagles — Wed, 05 Feb 2025 11:35:19 GMT

Support is not available at this time so trying this method. We have a job that downloads a file from Hubspot, converts to JSON so that it can be imported into Snowflake. The file is now 30.5MB and we are getting an error using the Ruby Connector. The error is "CSV file is too big. Max allowed: 30 MB, got: 30.5 MB"

The way Hubspot works is that you have to export the entire deals file without a filter so making the file smaller is not an option at this point. We will need to consider a longer term architectural fix so I'm looking to get this working for now.

We are going to convert the Ruby connector to Python Code and will see if that will fix the issue.

Does anyone have any other suggestion in case this does not work?

Ruby Connector code:

est = now + 3.hours
formatted_time = est.strftime('%Y-%m-%d %H:%M')

{out:workato.csv.parse(input["csv"], headers: true).map{|x|x.merge(x){|_,v|v.presence}.merge({snapshot_date: formatted_time})}.to_json}

Re: Ruby Connector file size limit

Prudvi — Wed, 05 Feb 2025 12:25:54 GMT

Hi @vascoeagles,
Does the endpoint support compressing the file.
Regards,
Prudvi

Re: Ruby Connector file size limit

gary1 — Wed, 05 Feb 2025 21:43:30 GMT

If the Hubspot file is a CSV, I would dump it into a SQL Collection and then query all records. I regularly use Collections with spreadsheets/CSVs that are hundreds of megabytes in size.

topic Ruby Connector file size limit in Workato Pros Discussion Board

Ruby Connector file size limit

Re: Ruby Connector file size limit

Re: Ruby Connector file size limit