How to connect to hdfs using pyarrow in python

user_1899010 0
Viewed 4603

I've installed pyarrow and would like to connect to hdfs file in a Hadoop cluster. I'm having the following line and it give me error.

 fs = pa.hdfs.connect(host='...', 50057, user='...', kerb_ticket='/tmp/krb5cc_0')

this is the error message I'm getting

ArrowIOError: Unable to load libhdfs

How should I install libhdfs? What other dependecies/settings I need to do?

hdfs pyarrow hadoop python
1 Answers
pyarrow.hdfs.connect(host='default', port=0, user=None, kerb_ticket=None, driver='libhdfs', extra_conf=None)

you have to be sure that libhdfs.so is in $HADOOP_HOME/lib/native as well as in $ARROW_LIBHDFS_DIR.

For HADOOP:

bash-3.2$ ls $ARROW_LIBHDFS_DIR
examples libhadoop.so.1.0.0 libhdfs.a libnativetask.a
libhadoop.a libhadooppipes.a libhdfs.so libnativetask.so
libhadoop.so libhadooputils.a libhdfs.so.0.0.0 libnativetask.so.1.0.0

The last version as i know is Hadoop 3.2.0

You can load any native shared library using DistributedCache for distributing and symlinking the library files.

This example shows you how to distribute a shared library, mylib.so, and load it from a MapReduce task. see informations

  1. First copy the library to the HDFS: bin/hadoop fs -copyFromLocal mylib.so.1 /libraries/mylib.so.1

  2. The job launching program should contain the following:

    DistributedCache.createSymlink(conf); DistributedCache.addCacheFile("hdfs://host:port/libraries/mylib.so. 1#mylib.so", conf);

  3. The MapReduce task can contain: System.loadLibrary("mylib.so");

Note: If you downloaded or built the native hadoop library, you don’t need to use DistibutedCache to make the library available to your MapReduce tasks.

user_100332860
Related