My requirements are:
- Read parquet file (File size is 270 MB or more)
- Convert the above read data into any user readable format file (eg: json)
- Upload the above readable format file into Azure blob.
I'm able to read the file by using ChoJSONWriter and store it back into StringBuilder. Now I have to convert it to either a byte array or a stream so that I can upload it to Azure in json. But while converting it to a byte array or toString() or a stream, I'm getting an exception of type System.OutOfMemoryException.
Here is my code.
StringBuilder json = new StringBuilder();
using (var r = new ChoParquetReader(FILE_NAME))
{
using(var w = new ChoJSONWriter(json))
w.Write(r);
}
Console.WriteLine("capacity: " + json.Capacity);
Console.WriteLine("Max capac : " + json.MaxCapacity);
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(json.ToString());
here getting memory out of exception issue.
When I check StringBuilder.Capacity then I see its value as 2147483008, and StringBuilder.MaxCapacity value is 2147483647.
Please suggest any open source packages to read huge parquet and as well as to convert read parquet data into any user readable format and upload back to azure blob.