Spark structured streaming job: stream-static join is not updated

I'm using Spark Structured streaming to read events from an Azure Eventhub, enrich the events with some data, by doing a stream-static join, and output the enriched events to some other source.

The issue is that, after a few hours, the streaming job stops detecting updated in the static table. So note that updates in the static table do get caught and updated at first but after an unknown amount of time this does not seem to be the case anymore. The static dataframe is a Delta table stored in dbfs that can get updated every 30 minutes.

Does anyone know why it works as expected at first (i.e. changes in the static delta table are processed) but after some time it stops detecting changes in the static table?

Please see my simplified code

# Read eventhub
metricbeat_df = spark \
    .readStream \
    .format("eventhubs") \
    .options(**eh_conf) \
    .load()

# Join event stream with facts
joined_df = join_with_facts(metricbeat_df)

trigger = {"processingTime": "30 seconds"}
for_each_stream = joined_df \
    .writeStream \
    .trigger(**trigger) \
    .foreach(CustomParser()) # This just outputs each event to the desired output
for_each_stream.start()

Where join_with_facts looks like this:

def join_with_facts(dataframe):
    facts_df = spark.table("my_db.facts_table")
    joined_facts = dataframe \
        .join(facts_df, "ID", how='inner')
    return joined_facts