I'm trying to run the PyPostal python bindings for the C program LibPostal to perform address standardisation on spark (AWS Glue to be precise). I have followed the guide https://github.com/openvenues/pypostal/issues/58 which links to https://github.com/openvenues/libpostal/issues/153 but to no avail.
The steps I have taken.
- Compiled the libpostal into a single tar file
joint.tar.gz - Compiled the libpostal data files into a single tar file
libpostal_datadir.tar.gz - Created an egg of the PyPostal python bindings
postal-1.1.9-py2.7-linux-x86_64.egg - Configured pyspark shell with the following command.
gluepyspark3 \
--conf "spark.yarn.dist.archives=s3://{some_S3_location}/libpostal/joint.tar.gz,s3://{some_S3_location}/libpostal/libpostal_datadir.tar.gz" \
--conf "spark.executorEnv.LD_LIBRARY_PATH=./joint.tar.gz" \
--conf "spark.executorEnv.LIBPOSTAL_DATA_DIR=./libpostal_datadir.tar.gz" \
--conf "spark.driver.extraLibraryPath=./joint.tar.gz" \
--conf "spark.driver.LibraryPath=./joint.tar.gz" \
--conf "spark.yarn.appMasterEnv.LD_LIBRARY_PATH=./joint.tar.gz" \
--conf "spark.yarn.appMasterEnv.LIBPOSTAL_DATA_DIR=./libpostal_datadir.tar.gz" \
--py-files "s3://{some_s3_location}/libpostal/postal-1.1.9-py2.7-linux-x86_64.egg"
- Inside Spark shell
from postal.parser import *
The following error is returned.
Python 3.6.12 (default, Aug 31 2020, 18:56:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/aws/glue/etl/jars/glue-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/02/19 15:12:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/02/19 15:12:19 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/02/19 15:12:32 WARN Client: Same path resource s3://safari-dev-assets/glue/jobs/kw_libpostal/postal-1.1.9-py2.7-linux-x86_64.egg added multiple times to distributed cache.
21/02/19 15:12:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.3
/_/
Using Python version 3.6.12 (default, Aug 31 2020 18:56:18)
SparkSession available as 'spark'.
>>> from postal.parser import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/tmp/spark-57e88948-952a-4a7b-b104-f7832af5f9c8/userFiles-cf4d4719-3866-4d63-8ef6-fbe3b4dd08ad/postal-1.1.9-py2.7-linux-x86_64.egg/postal/parser.py", line 2, in <module>
File "/mnt/tmp/spark-57e88948-952a-4a7b-b104-f7832af5f9c8/userFiles-cf4d4719-3866-4d63-8ef6-fbe3b4dd08ad/postal-1.1.9-py2.7-linux-x86_64.egg/postal/_parser.py", line 7, in <module>
File "/mnt/tmp/spark-57e88948-952a-4a7b-b104-f7832af5f9c8/userFiles-cf4d4719-3866-4d63-8ef6-fbe3b4dd08ad/postal-1.1.9-py2.7-linux-x86_64.egg/postal/_parser.py", line 6, in __bootstrap__
File "/usr/lib64/python3.6/imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: libpostal.so.1: cannot open shared object file: No such file or directory