Please help me to fix this issue. I am writing data to event hub from data-bricks(pyspark) streaming query as below:
def foreach_batch_function(df, epoch_id):
df.orderBy("_commit_timestamp")
df.select(to_json(struct("*")).alias("body")).
write.format("eventhubs").options(**ehConf).save()
w_hubble_account_df = hubble_account_df.writeStream\
.option("checkpointLocation", hubble_account_checkpoint_location)\
.foreachBatch(foreach_batch_function).start()
w_hubble_account_df.awaitTermination()
This runs for few minutes and writes few thousands records to event hub successfully and then throws the TimeoutException. This is suppose to run continuously until stopped explicitly. What could be the possible root cause?