Currently, I am migrating an old Databricks environment to a new environment. Due to a lot of dependencies, the objective is to run the old and new environment in parallel until all tests have passed. Therefore I am trying to copy the Databricks tables from the old environment to the new. To attempt a successful copy of the tables I tried to export the Databricks tables from the old subscription to an account storage, and from the account storage import them in the new Databricks environment.
To copy them so far I tried:
df.write.format("delta").save(<mount_path>), spark.read.format("delta").load(<mount_path>), df.write.format("delta").option("path", <location>).saveAsTable(<table_name>)DeltaTable.forPath(spark, <location>)dbutils.fs.cp(<from>, <to>, <recurse>)sh cp --preserve=all -R <from> <to>
As I expected the first one could not work due to the conversion to a dataframe which make you lose the information about the history. The second approach does not have a save method to save the DeltaTable to Databricks. For the third and fourth option both preserve the version history number but not the timestamps, those are updated with the timestamp of when the command was executed.
Therefore I hope that anybody can help me in finding a solution to copy DeltaTables in databricks from one subscription to another while retaining the entire history. Thus the version number as well as the timestamps.
All help and feedback are appreciated.