Can we do image processing with Palantir Foundry?

Although most examples are given using tabular data, in reality a lot of use case are using foundry for both unstructured and semi-structured data processing. You should think of a dataset as a container of files with an API to access and process the files. using the file level API you can get access to the files in the dataset and process them as you like. If these files are images you can extract information from the file and use it as you like. a common use case is to have PDFs as files in a dataset and to extract information from the PDF and store it as tabular information so you can do both structured and unstructured search over it.

here is an example of file access to extract PDF:

import com.palantir.transforms.lang.java.api.Compute;
import com.palantir.transforms.lang.java.api.FoundryInput;
import com.palantir.transforms.lang.java.api.FoundryOutput;
import com.palantir.transforms.lang.java.api.Input;
import com.palantir.transforms.lang.java.api.Output;
import com.palantir.util.syntacticpath.Paths;
import com.google.common.collect.AbstractIterator;
import com.palantir.spark.binarystream.data.PortableFile;
import java.io.IOException;
import java.io.InputStream;
import java.util.Iterator;
import java.util.UUID;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.Row;
import org.apache.pdfbox.pdmodel.PDDocument; 
import org.apache.pdfbox.text.PDFTextStripper;


public final class ExtractPDFText {

    private static String pdf_source_files_rid = "SOME RID";
    private static String dataProxyPath = "/foundry-data-proxy/api/dataproxy/datasets/";
    private static String datasetViewPath = "/views/master/";

    @Compute 
    public void compute(
        @Input("/Base/project_name/treasury_pdf_docs") FoundryInput pdfFiles, 
        @Output("/Base/project_name/clean/pdf_text_extracted") FoundryOutput output) throws IOException {

        Dataset<PortableFile> filesDataset = pdfFiles.asFiles().getFileSystem().filesAsDataset(); 

        Dataset<String> mappedDataset = filesDataset.flatMap((FlatMapFunction<PortableFile, String>) portableFile -> 
            portableFile.convertToIterator(inputStream -> {

                String pdfFileName = portableFile.getLogicalPath().getFileName().toString();
                return new PDFIterator(inputStream, pdfFileName);
            }), Encoders.STRING());

        Dataset<Row> dataset = filesDataset
                .sparkSession()
                .read()
                .option("inferSchema", "false")
                .json(mappedDataset);

        output.getDataFrameWriter(dataset).write();
    }

    private static final class PDFIterator extends AbstractIterator<String> {
        private InputStream inputStream;
        private String pdfFileName;
        private boolean done;

        PDFIterator(InputStream inputStream, String pdfFileName) throws IOException {
            this.inputStream = inputStream;
            this.pdfFileName = pdfFileName;
            this.done = false;
        }

        @Override
        protected String computeNext() {
            if (done) {
                return endOfData();
            }

            try {
                String objectId = pdfFileName;
                String appUrl = dataProxyPath.concat(pdf_source_files_rid).concat(datasetViewPath).concat(pdfFileName);
                PDDocument document = PDDocument.load(inputStream);

                PDFTextStripper pdfStripper = new PDFTextStripper();

                String text = pdfStripper.getText(document);
                String strippedText = text.replace("\"", "'").replace("\\", "").replace("“", "'").replace("”", "'").replace("\n", "").replace("\r", "");

                done = true;
                return "{\"id\": \"" + String.valueOf(UUID.randomUUID()) + "\", \"file_name\": \"" + pdfFileName + "\", \"app_url\": \"" + appUrl + "\", \"object_id\": \"" + objectId + "\", \"text\": \"" + strippedText + "\"}\n";
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }
    }
} 

Indeed you can do image analysis on Foundry as you have access to files and can use arbitrary libraries (for example pillow or skimage for python). This can be done at scale as well as it can be parallelised.

A simple python snippet to stitch two pictures together should get you started:

from transforms.api import transform, Input, Output
from PIL import Image


@transform(
    output=Output("/processed/stitched_images"),
    raw=Input("/raw/images"),
    image_meta=Input("/processed/image_meta")
)
def my_compute_function(raw, image_meta, output, ctx):

    image_meta = image_meta.dataframe()

    def stitch_images(clone):
        left = clone["left_file_name"]
        right = clone["right_file_name"]
        image_name = clone["image_name"]
        with raw.filesystem().open(left, mode="rb") as left_file:
            with raw.filesystem().open(right, mode="rb") as right_file:
                with output.filesystem().open(image_name, 'wb') as out_file:

                    left_image = Image.open(left_file)
                    right_image = Image.open(right_file)

                    (width, height) = left_image.size

                    result_width = width * 2
                    result_height = height
                    result = Image.new('RGB', (result_width, result_height))

                    result.paste(im=left_image, box=(0, 0))
                    result.paste(im=right_image, box=(height, 0))

                    result.save(out_file, format='jpeg', quality=90)

    image_meta.rdd.foreach(stitch_images)

The image_meta dataset is just a dataset that has 2 file names per row. To extract filenames from a dataset of raw files you can use something like:

@transform(
    output=Output("/processed/image_meta"),
    raw=Input("/raw/images"),
)
def my_compute_function(raw, output, ctx):

    file_names = [(file_status.path, 1) for file_status in raw.filesystem().ls(glob="*.jpg")]

    # create and write spark dataframe based on array

As others have mentioned Palantir-Foundry’s focus is on tabular data and doesn’t currently provide GPU or other tensor processing unit access. So doing anything intense like an FFT transform or Deep Learning would be ill advised at best if not downright impossible at worst.

That being said you can upload image files into dataset nodes for read/write access. You could also store their binary information as a blob type into a Dataframe in order to store files in a given record field. Given that there are a multitude of Python image processing adjacent and matrix math libraries available on the platform and given that it’s also possible to upload library packages to the platform manually through the Code Repo app, it is conceivable that someone could make use of simple manipulations on a somewhat large scale as long as it wasn’t overly complex or memory intensive.

Note that Foundry seems to have no GPU support at the moment, so if you are thinking about running deep learning based image processing, this will be quite slow on CPUs.