Is lineSep option really works with Spark 2.4 version.
lineSep (default covers all \r, \r\n and \n): defines the line separator that should be used for parsing. Maximum length is 1 character.
I am writing a Dataframe to GCS bucket location but its always writing it with line separator as '\n' only.
df
.select("COLLECTTIME","SITE","NETWORK")
.coalesce(1)
.limit(10)
.write
.option("header", false)
.option("compression", "gzip")
.option("lineSep","\r\n")
.csv(tmpOutput)
I was looking for CRLF at the end of each line.
I tried below as well but not working
df2.withColumn(df2.columns.last,concat(col(df2.columns.last),lit("\r")))
.write
.option("header", false)
.option("compression", "gzip")
.csv(tmpOutput)
I tried below as well but no luck.
import org.apache.spark.sql.functions._
df2.withColumn(df2.columns.last,regexp_replace(col(df2.columns.last),"[\\r]","[\\r\\n]"))
.write
.option("header", false)
.option("compression", "gzip")
.csv(tmpOutput)
Now I am thinking to read file again from GCS once its written and read it line by line and append '\r' at the end of each record. isn't something short and simple available with Spark 2.4. I just need to have 'CRLF' at the end of each record.
read and update is also not possible since objects stored on gcs buckets are immutable. I cannot keep files on buffer since they little bigger in size as well