Parse address function in pyspark

user_11367634 0
Viewed 258

I'd like to add a column in my dataframe in PySpark with the address parsed through the libpostal library.

import pyspark.sql.functions as sf
from postal.parser import parse_address
df = spark.read.parquet(path_hdfs)
df = df.select("id", "name" ,"street", "cty")\
       .withColumn("address", parse_address(sf.concat(col("street"),sf.lit(" ") ,col("cty"))))\
       .dropDuplicates()

but I get the error:

TypeError: cannot convert 'Column' object to bytes

how can I change the argument of parse_address in order to be accepted by that function?

pyspark apache-spark geocoding user-defined-functions python
1 Answers

You need to use a UDF to call Python libraries, e.g.

df = df.select("id", "name" ,"street", "cty")\
       .withColumn("address", 
           sf.udf(parse_address, 'array<struct<val:string,key:string>>')
           (sf.concat("street", sf.lit(" "), "cty"))
       )\
       .dropDuplicates()
user_141657300
Related