How to retain the timestamp format when extracting from a dataframe into a string in PySpark

Question

I have the following dataframe in PySpark (Databricks). How can I extract the exact literal timestamp as displayed in the dataframe into a string?. At the moment, using the code below, I get: "2022-02-25 06:32:29" instead of "2022-02-25T06:32:29.000+0000". some conversion is happening that strips of the milliseonds part including the 'T'. I want to retain the literal string as it appears on the dataframe?

code:

table = [x["ts"] for x in ts.rdd.collect()]
for row in table:
    print(row)

code:

table = [x["ts"] for x in ts.rdd.collect()]
for row in table:
    print(row)

Alex_Ott_18627 · Answer

timestamps & dates under the hood aren't the strings, but have types like int & long, and it's up to the client program to convert these data into a visual representation. You can use the date_format function to explicitly convert timestamp or date into a string if you want to have uniform data representation...

How to retain the timestamp format when extracting from a dataframe into a string in PySpark

1 Answers