I have a requirement where I need the output file name to be in format - <Test_YYYYMMDD_NNNN>.txt. Output file will have multiple partitions.
NNNN represent partition sequence.
Example-
1.Test_20220306_0001.txt 2.Test_20220307_0002.txt
Please let me know if this is achievable. Currently, I am getting the file in format Test_20220306-000001.txt which is the default spark schema.
Looks like the default Spark partitioning schema will not be ignored and it will automatically add the -00001/-00002/.. partition value to each partition file name when using file name option as pattern in sink settings.
If you know the number of partitions value, you can use per partition in File name option in sink settings and provide each partitioned file name dynamically.
filename parameter contains concat('Test_',replace(toString(currentDate()),'-',''))
Output files:
