lineSep option with Spark 2.4

Viewed 1896

Is lineSep option really works with Spark 2.4 version.

lineSep (default covers all \r, \r\n and \n): defines the line separator that should be used for parsing. Maximum length is 1 character.

I am writing a Dataframe to GCS bucket location but its always writing it with line separator as '\n' only.

df
  .select("COLLECTTIME","SITE","NETWORK")
  .coalesce(1)
  .limit(10)
  .write
  .option("header", false)
  .option("compression", "gzip")
  .option("lineSep","\r\n")
  .csv(tmpOutput)

I was looking for CRLF at the end of each line.

I tried below as well but not working

df2.withColumn(df2.columns.last,concat(col(df2.columns.last),lit("\r")))
  .write
  .option("header", false)
  .option("compression", "gzip")
  .csv(tmpOutput)

I tried below as well but no luck.

import org.apache.spark.sql.functions._
df2.withColumn(df2.columns.last,regexp_replace(col(df2.columns.last),"[\\r]","[\\r\\n]"))
  .write
  .option("header", false)
  .option("compression", "gzip")
  .csv(tmpOutput)

Now I am thinking to read file again from GCS once its written and read it line by line and append '\r' at the end of each record. isn't something short and simple available with Spark 2.4. I just need to have 'CRLF' at the end of each record.

read and update is also not possible since objects stored on gcs buckets are immutable. I cannot keep files on buffer since they little bigger in size as well

1 Answers

I am very sorry, but AFAIK, I am afraid that Spark allows the different separators you cited in your question:

lineSep (default covers all \r, \r\n and \n): defines the line separator that should be used for parsing. Maximum length is 1 character.

only for reading, but not for writing; in the later case, either \n is hardcoded or, since Spark versions 2.4 and 3.0, you can choose a custom line separator but limited to a single character.

Please, consider read this Github issue, it provides the whole background about the problem. This other one could be helpful as well.

Related