Dataframe turns empty after writing to mount

Viewed 186

I have a dataframe called updatesDF as follows,

+-------+---------+--------------+-------------------+----------------+-------------------+----------------+
|PartyID|      TIN|SourceSystemID|      ODSInsertDate|     ODSInsertBy|      ODSUpdateDate|     ODSUpdateBy|
+-------+---------+--------------+-------------------+----------------+-------------------+----------------+
| 11111|222222222|             1|2021-07-20 01:56:25|sneha|2021-07-20 01:56:25|sneha|
+-------+---------+--------------+-------------------+----------------+-------------------+----------------

so updatesDF.show() gives me the above output. Now I need to write this dataframe to a mount path,

updatesDF.write.format('delta').mode('append').save('/mnt/Sneha/Updates/')

So as soon as I write into this location, the updatesDF turns blank like this

+-------+---+--------------+-------------+-----------+-------------+-----------+
|PartyID|TIN|SourceSystemID|ODSInsertDate|ODSInsertBy|ODSUpdateDate|ODSUpdateBy|
+-------+---+--------------+-------------+-----------+-------------+-----------+
+-------+---+--------------+-------------+-----------+-------------+-----------+

There are no other steps in between and I also tried taking backup of this DF...both backup and original DF turns empty after the append step. Please help

1 Answers

It's a weird situation. Looks like a memory saturation.

Can you try updatesDF.persist() or .cache() just after creation.

Related