topic S3 file movement in Workato Pros Discussion Board

S3 file movement

ajorde — Thu, 31 Mar 2022 01:19:36 GMT

We're trying to move CSV files in an S3 bucket from one path to another. It seems like the standard S3 connector requires the file to be downloaded, reuploaded, and then original file deleted. Does anyone have thoughts on a better way to perform the file move?

Constraint: We do not have an OPA to be able to run AWS CLI commands.

Alternative considered:

I'm thinking of the possibility to use a Snowflake command to copy into an S3 stage from an S3 stage, but not sure how to make that fully dynamic to not need to specify columns in the snowflake command.

COPY INTO @workato_s3_stage/new_path/test.csv

from (select $1 from @workato_s3_stage/original_path/test.csv)

Anyone have any thoughts on the Snowflake command option or any other possible alternatives?

Thanks!

Re: S3 file movement

alek-oleszkiewi — Thu, 31 Mar 2022 19:20:22 GMT

Instead of connecting directly to the bucket using the S3 connector, you can use AWS Transfer Family service to create an SFTP server on top of your S3 bucket.

Then in Workato, you will be able to use the SFTP connector, which provides an action allowing you to move the file to a different folder.

Re: S3 file movement

kobi-sh — Thu, 31 Mar 2022 22:29:22 GMT

You can map the content of the file from the Download step to the content of the file in the Upload step.

Then, you can Delete the original file.