I'm using pySpark to write some CSVs. The problem I'm having can be seen in the screenshot below: When my string fields have commas, it looks like that (this can be seen perfectly in excel) the CSV/the Data is taking the comma in the string as a lineseparator, and thanks to that it's messing up with the structure of my data. I think this has something to do with quotes or escape, but I'm not sure how to fix it. Would appreciate your help, this seems pretty sensitive but I've spent many hours already. These commas in my string columns should be escaped so they won't count as lineseparators, as it's happening right now... I've seen other similar questions on stackoverflow but the solution wasn't clear.
Here's my code that writes the csvs:
def write_to_csv(df, path, compression):
print("log: starting to write the csvs, this might take some time")
df.write.csv(path = path ,sep=",", header=True, lineSep="\n", escape='"', nullValue=None, compression=compression)
print("log: finished writing csvs")