Writing Parquet file using AvroParquetWriter to GCS

Viewed 410

I'm trying to write a parquet file to goole storage bucket and I'm getting the following error:

java.lang.IllegalArgumentException: java.net.MalformedURLException: unknown protocol: default
Configuration conf = new Configuration();
        conf.set(GoogleHadoopFileSystemConfiguration.GCS_PROJECT_ID.getKey(), "my-project");
        conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem");
        conf.set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS");
        conf.setBoolean("google.cloud.auth.service.account.enable", true);
        conf.set("google.cloud.auth.service.account.json.keyfile", "/path/to/file");
        conf.setBoolean(GoogleHadoopFileSystemConfiguration.GCS_GRPC_ENABLE.getKey(), true);
     
URI uri = URI.create("gs://theBucket/" + fileName);
        Path path = new Path(uri);
OutputFile outputFile = HadoopOutputFile.fromPath(path, conf);

try (ParquetWriter<SomeItem> parquetWriter = AvroParquetWriter.<SomeItem>builder(outputFile)
        .withSchema(SomeItemSchema)
        .withDataModel(ReflectData.get())
        .withConf(parquetConfigurations)
        .withCompressionCodec(CompressionCodecName.SNAPPY)
        .withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
        .build()) {

    for (SomeItem item : items) {
        parquetWriter.write(item);
    }
} catch (Exception e) {
    String msg = "could not write companies to parquet file:";
    log.error(msg, e);
    throw new LookAlikeException(msg, e);
}

I'm able to write the parquet file to my local filesystem using the same code with the following configurations:

Path path = new Path("file:///Users/<myUser>/stuff/" + fileName);
OutputFile outputFile = HadoopOutputFile.fromPath(path, new Configuration(true));

anyone has an idea what I'm doing wrong?

0 Answers
Related