How to stream parquet directly from SQL to S3 using C#

Viewed 621

I am getting a result from SQL and using C# writing it to a local temp parquet file on disk before uploading to S3.

I am using the parquet.net library as well as all the standard AWS packages for s3 (AWSSDK.S3 I believe)

Please excuse the certain areas where I just left pseudocode but this is the basic structure... I want to be able to stream the rows directly to S3 instead of writing the file locally first.

Thanks in advance.

using (var parquetWriter = new ParquetWriter(parquetSchema, fileStream))
{
    var parquetTable = new Parquet.Data.Rows.Table(parquetSchema);
    foreach(object[] data in dataset)
    {
         //prepare data for parquet schema here

         parquetTable.Add(data);
         if (parquettablesize >= number of rows I want to write at one time)
         {
              parquetWriter.Write(parquetTable);
              parquetTable = new Parquet.Data.Rows.Table(parquetSchema); //cant reuse the same object
         }
    }

    // write whatever is left here
    if (parquetTable.Count > 0)
          parquetWriter.Write(parquetTable);


    using (var fileStream = new FileStream(tempFile, FileMode.Open))
    {
         //upload the file to my S3 location
    }
}
0 Answers
Related