I have a PySpark transform in Palantir Foundry that's outputting to a csv file for export into other systems.
Currently, using the write_dataframe method the name of the file looks like this:
spark/part-00002-cfba77d5-c6ce-4b2a-ac9a-59173c7ede5a-c000.snappy.csv
is it possible to specify a filename, such as "my_export.csv" ?
It's likely easier to accomplish this using an export task rather than via a transform. Some documentation on export tasks is available here, but it is described in more detail in the in-platform docs.
If you're using a file system or SFTP export task, there is an option to rewrite paths in the task config. For example,
rewritePaths:
".*": "my_export.csv"
would rename all files to my_export.csv. I wouldn't recommend doing exactly that, as you'll have a collision if there are multiple files, but you can also capture part of the existing file name and use it to make the renamed files unique:
rewritePaths:
"^spark/(.*)": "my_export-$1.csv"