pyarrow: non-legacy filesystem cannot connect, even though legacy can

user_7064628 0
Viewed 343

When trying to use the non-legacy dataset from pyarrow package (version 0.17.1), i get the OSError: HDFS connection failed error when connecting to hdfs. However, when i use the legacy one, i can establish the connection without any issues.

i.e. the bellow works:

filesystem = pa.hdfs.connect(host = 'host_name', kerb_ticket = path_to_kerb_ticket_cache, port = 0)

but this does not:

filesystem = pa.fs.HadoopFileSystem(host = 'host_name', kerb_ticket = path_to_kerb_ticket_cache, port = 0)

Moreover, if i first create connection using the legacy way and then try to connect with the non-legacy way, everything works:

filesystem = pa.hdfs.connect(host = 'host_name', kerb_ticket = path_to_kerb_ticket_cache, port = 0)
filesystem = pa.fs.HadoopFileSystem(host = 'host_name', kerb_ticket = path_to_kerb_ticket_cache, port = 0)

Any ideas how to properly use pa.fs.HadoopFileSystem?

pyarrow python
1 Answers

When you use the legacy connect function and enter the host as default it takes the default host from core-site.xml (defaultFS). The new HadoopFileSystem call does not do this. Manually entering the host into HadoopFileSystem fixed it for me.

e.g. entering the host in this part

fs.HadoopFileSystem(host='hdfs://your-hdfs-host' ...)

instead of entering default

fs.HadoopFileSystem(host='default' ...)

user_11789710
Related