I am getting a result from SQL and using C# writing it to a local temp parquet file on disk before uploading to S3.
I am using the parquet.net library as well as all the standard AWS packages for s3 (AWSSDK.S3 I believe)
Please excuse the certain areas where I just left pseudocode but this is the basic structure... I want to be able to stream the rows directly to S3 instead of writing the file locally first.
Thanks in advance.
using (var parquetWriter = new ParquetWriter(parquetSchema, fileStream))
{
var parquetTable = new Parquet.Data.Rows.Table(parquetSchema);
foreach(object[] data in dataset)
{
//prepare data for parquet schema here
parquetTable.Add(data);
if (parquettablesize >= number of rows I want to write at one time)
{
parquetWriter.Write(parquetTable);
parquetTable = new Parquet.Data.Rows.Table(parquetSchema); //cant reuse the same object
}
}
// write whatever is left here
if (parquetTable.Count > 0)
parquetWriter.Write(parquetTable);
using (var fileStream = new FileStream(tempFile, FileMode.Open))
{
//upload the file to my S3 location
}
}