AWS S3 Inventory will output a file called symlinks.txt
symlinks.txt is an implementation of SymlinkTextInputFormat
There's documentation on AWS and Databricks on how to read these files into Athena, with mentions about the fact it should be possible with Spark but I'm unable to confirm this is working .
- What would be a proper way to read symlink format in spark.
- What should be the format of the file for spark to be able to see the files?