topic Re: Parquet files in Workato Pros Discussion Board

Parquet files

ajorde — Tue, 01 Mar 2022 04:00:21 GMT

We have a usecase where we need to take data and reformat into parquet files to load to S3. Does anyone have a sample recipe or methodology where you've done this?

Thanks!

Re: Parquet files

deepak23shastri — Tue, 01 Mar 2022 04:08:49 GMT

You could write a simple python code as below:

import pandas as pd
df = pd.read_csv('example.csv')
df.to_parquet('output.parquet')

Re: Parquet files

ajorde — Tue, 01 Mar 2022 04:10:25 GMT

I saw this post online too - but I'm not sure how to import packages into the Workato python action and want to keep this off of the OPA.

Re: Parquet files

chijie — Tue, 01 Mar 2022 04:16:14 GMT

Hi Amy,

It's possible to do this with our JavaScript connector. See sample recipe here: https://app.workato.com/recipes/2226332?st=9a56d4

All you need to do is edit the json schema which defines the schema in the parquet file, then map the input data from there. No need to edit the code (FYI - we're using parquetjs-lite library for this).

That being said, your request is extremely timely as we are gathering feedback from users to support reading/writing Parquet.

Can you share the requirements for creating Parquet files? How large is the content for each file? Is there a need to control encoding/compression/partitions/row group size etc? Do you need to be able to append to an existing parquet file?

If there's anyone else in the community who needs Parquet support as well, please share with us your requirements.

Your feedback will help us prioritize this feature. Thanks!