How can I create an Azure dataset in Azure ML studio (through the GUI) from a parquet file created with Azure Spark

user_4553735 0
Viewed 178

I'm trying to load files as a dataset in the GUI of Azure ML Studio. These parquet files have been created through Spark.

In my folder, Spark creates files such as "_SUCCESS" or "_committed_8998000".

Azure ML Studio is not able to read them or ignore them and tells me:

The provided file(s) have invalid byte(s) for the specified file encoding.
{
  "message": " "
}

I selected "Ignore unmatched files path" and yet, it still does not work.

If I remove the "_SUCCESS" and other Spark files, it works.

azure-machine-learning-studio parquet apache-spark azure
2 Answers

Thanks for the feedback. You can use globing in path. e.g. path = '**/*.parquet' to select only the parquet files

user_126002270

This is a problem with Azure ML Studio. It presumably tries to parse and decode the CRC files that are written next to the Parquet file by default.

The solution right now is to get rid of those files and hopefully, Microsoft will eventually fix this issue on their side.

user_6471510
Related