I'm trying to write a parquet file to goole storage bucket and I'm getting the following error:
java.lang.IllegalArgumentException: java.net.MalformedURLException: unknown protocol: default
Configuration conf = new Configuration();
conf.set(GoogleHadoopFileSystemConfiguration.GCS_PROJECT_ID.getKey(), "my-project");
conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem");
conf.set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS");
conf.setBoolean("google.cloud.auth.service.account.enable", true);
conf.set("google.cloud.auth.service.account.json.keyfile", "/path/to/file");
conf.setBoolean(GoogleHadoopFileSystemConfiguration.GCS_GRPC_ENABLE.getKey(), true);
URI uri = URI.create("gs://theBucket/" + fileName);
Path path = new Path(uri);
OutputFile outputFile = HadoopOutputFile.fromPath(path, conf);
try (ParquetWriter<SomeItem> parquetWriter = AvroParquetWriter.<SomeItem>builder(outputFile)
.withSchema(SomeItemSchema)
.withDataModel(ReflectData.get())
.withConf(parquetConfigurations)
.withCompressionCodec(CompressionCodecName.SNAPPY)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.build()) {
for (SomeItem item : items) {
parquetWriter.write(item);
}
} catch (Exception e) {
String msg = "could not write companies to parquet file:";
log.error(msg, e);
throw new LookAlikeException(msg, e);
}
I'm able to write the parquet file to my local filesystem using the same code with the following configurations:
Path path = new Path("file:///Users/<myUser>/stuff/" + fileName);
OutputFile outputFile = HadoopOutputFile.fromPath(path, new Configuration(true));
anyone has an idea what I'm doing wrong?