Code Workbooks - File not found using hadoop_path

user_1483132 0
Viewed 67

I have a python transform in code workbooks that is running this code:

import pandas as pd

def contents(dataset_with_files):
    fs = dataset_with_files.filesystem()
    filenames = [f.path for f in fs.ls()]
    fp = fs.hadoop_path + "/" + filenames[0]
    with open(fp, 'r') as f:
        t = f.read()
    rows = {"text": [t]}
    return pd.DataFrame(rows)

But I am getting the error FileNotFoundError: [Errno 2] No such file or directory:

My understanding is that this is the correct way to access a file in the hdfs, is this a repository versus code workbooks limitation?

foundry-code-repositories palantir-foundry
1 Answers

This documentation helped me figure it out: https://www.palantir.com/docs/foundry/code-workbook/transforms-unstructured/

It was actually a pretty small change. If you are using the filesystem() you only need the relative path.

import pandas as pd

def contents_old(pycel_test):
    fs = pycel_test.filesystem()
    filenames = [f.path for f in fs.ls()]
    with fs.open(filenames[0], 'r') as f:
        value = ...
    rows = {"values": [value]}
    return pd.DataFrame(rows)

There is also this option, but I found it 10x slower.

from pyspark.sql import Row

def contents(dataset_with_files):
    fs = dataset_with_files.filesystem() # This is the FileSystem object.
    MyRow = Row("column")
    def process_file(file_status):
        with fs.open(file_status.path, 'r') as f:
            ...
    rdd = fs.files().rdd
    rdd = rdd.flatMap(process_file)
    df = rdd.toDF()
    return df
user_14831320
Related