Copying files from a Foundry dataset to another Foundry dataset

user_17881764 0
Viewed 163

I have two Foundry datasets that contain raw files (lets say xml or csv files). I would like to merge these two within a transform to create a new dataset with a collection from both.

(This explicit example was due to a API schema being updated, and required to merge the existing data with the new version).

ex A: csv1, csv2, csv3, csv4, csv5 (source) B: csv1, csv2, csv3 (target)

palantir-foundry
1 Answers

Because Foundry datasets store raw files, a simple Python transform using shutil.copyfileobj should do the trick. This is further documented under Palantir docs: transforms/python-raw-file-access#writing-files

for file_status in in_source.filesystem().ls(glob='*.csv'):
    with in_source.filesystem().open(file_status.path, 'rb') as in_f:
        with out.filesystem().open(file_status.path, 'wb') as out_f:
            shutil.copyfileobj(in_f, out_f)
user_178817640
Related