Import Hudi Modules in Pyspark

Viewed 298

How to import hudi modules in pyspark ?

%spark.pyspark
 import org.apache.hudi.DataSourceWriteOptions
 import org.apache.hudi.DataSourceReadOptions
 import org.apache.hudi.config.HoodieWriteConfig

Error : ModuleNotFoundError: No module named 'org'

1 Answers

I'm assuming you want to import these to use Hudi options.
When using pyspark You don't do these imports, these are needed when using scala or java.
In pyspark you specify options as key:value pairs.

Following the Hudi Spark guide this is how you declare options:

hudi_options = {
'hoodie.table.name': tableName,
'hoodie.datasource.write.recordkey.field': 'uuid',
'hoodie.datasource.write.partitionpath.field': 'partitionpath',
'hoodie.datasource.write.table.name': tableName,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'ts',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2
}

df.write.format("hudi").
    options(**hudi_options).
    mode("overwrite").
    save(basePath)
Related