Hadoop FTPFileSystem fails to list files and throws SocketTimeOutException

Viewed 29

I am using Apache Hadoop FTPFileSystem version 3.2.0 to list and read files from an FTP Server.

Here is my test code:

public static void main(String[] args) throws IOException {
    String host = "some-host";
    int port = 21;
    Configuration conf = new Configuration(false);
    conf.set("fs.ftp.host", host);
    conf.setInt("fs.ftp.host.port", port);
    conf.set("fs.ftp.user." + host, "username");
    conf.set("fs.ftp.password." + host, "password");
    conf.set("fs.ftp.data.connection.mode", "PASSIVE_LOCAL_DATA_CONNECTION_MODE");
    conf.set("fs.ftp.impl", "org.apache.hadoop.fs.ftp.FTPFileSystem");
    
    String fsURL = String.format("ftp://%s:%s", host, String.valueOf(port));
    conf.set("fs.default.name", fsURL);
    FileSystem fs =  FileSystem.newInstance(conf);
    Path somePath = new Path("actual/path");
    fs.getFileStatus(somePath).isDirectory(); // returns true
    fs.listStatus(somePath); // keeps spinning then throws SocketTimeOutException
}

After some debugging the deadlock or the delay happens at this method org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPFileEntryParser, String) at this method execution: engine.readServerList(socket.getInputStream(), getControlEncoding()); as below:

private FTPListParseEngine initiateListParsing(
        FTPFileEntryParser parser, String pathname)
throws IOException
{
    Socket socket = _openDataConnection_(FTPCmd.LIST, getListArguments(pathname));

    FTPListParseEngine engine = new FTPListParseEngine(parser, __configuration);
    if (socket == null)
    {
        return engine;
    }

    try {
        engine.readServerList(socket.getInputStream(), getControlEncoding());
    }
    finally {
        Util.closeQuietly(socket);
    }

    completePendingCommand();
    return engine;
}

The method call keeps blocked until it finally throws a socketTimeoutException, even-though using FileZilla with same credentials and properties I can list & read files smoothly and in a much faster time.

The credentials I am using and properties are correct as the initial connection and fs.getFileStatus(somePath).isDirectory(); call works and return correct value.

Is there a property I can add to make things faster or is it a bug in apache hadoop FTPFileSystem version 3.2.0?

0 Answers
Related