We had working code as below.
print(f"{file_name} Before insert count", datetime.datetime.now(), scan_df_new.count())
scan_df_new.show()
scan_20220908120005_10 Before insert count 2022-09-14 11:37:15.853588 3
+-------------------+----------+-------------------+--------------------+----------+
| tran_id|t_store_id| scan_datetime| customer_id|updated_by|
+-------------------+----------+-------------------+--------------------+----------+
|1230000000000000004| 4395|2022-09-08 03:00:01|20220816a51cee4264f1|Databricks|
|1230000000000000005| 4394|2022-09-08 02:58:00|20220816a51cee4264f1|Databricks|
|1230000000000000006| 4393|2022-09-08 03:00:04|20220816a51cee4264f1|Databricks|
+-------------------+----------+-------------------+--------------------+----------+
The data frame after the write operation is used for further business logic processing. This was working earlier. But recently we are observing a strange behavior, where in the data in the data frame is getting lost . When wee check the contents, or even the dataframe count its shows empty.
scan_df_new.write.format("jdbc").option("url", jdbcUrl).option("dbtable", scan_table).mode("append").save()
print(f"{file_name} After insert count", datetime.datetime.now(), scan_df_new.count())
scan_df_new.show()
None
scan_20220908120005_10 After insert count 2022-09-14 11:37:18.372147 0
+-------+----------+-------------+-----------+----------+
|tran_id|t_store_id|scan_datetime|customer_id|updated_by|
+-------+----------+-------------+-----------+----------+
Anything recently changed in the Databricks, which is impacting this?