Configuring timezones on Databricks Spark cluster, conflict between Spark configuration and system settings

I am having some trouble configuring the right timezone on our Databricks spark cluster. We want to configure both the timezone in Spark context as well as the system wide timezone (both are in UTC by default). We want all timezones to be Europe/Amsterdam or UTC+02:00 (with support for daylight savings time).

All code snippets are run in a Python Databricks notebook.

Default UNIX system timezone is UTC:

%sh
timedatectl

Output:

                      Local time: Wed 2021-09-22 11:24:39 UTC
                  Universal time: Wed 2021-09-22 11:24:39 UTC
                        RTC time: n/a
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

Default Spark timezone is UTC:

spark.conf.get("spark.sql.session.timeZone")
Out[2]: 'Etc/UTC'

Now when I try to convert a date before and a date after DST is in effect, I get the following results:

for tz in ['Europe/Amsterdam', 'UTC+02:00', 'UTC']:
  spark.conf.set("spark.sql.session.timeZone", tz)
  print(f'Current SPARK timezone: {spark.conf.get("spark.sql.session.timeZone")}')
  for date in ['2021-09-28 10:30:00', '2021-11-11 10:30:00']:  
    df = spark.createDataFrame([(date,)], ['t'])
    dt=[row[0] for row in df.select(to_timestamp(df.t).alias('dt')).collect()][0]
    print(dt)

Output:
Current SPARK timezone: Europe/Amsterdam
2021-09-28 08:30:00
2021-11-28 09:30:00
Current SPARK timezone: UTC+02:00
2021-09-28 08:30:00
2021-11-28 08:30:00
Current SPARK timezone: UTC
2021-09-28 10:30:00
2021-11-28 10:30:00

Question 1: Why does to_timestamp change my dates in some timezone settings... When I convert a string to timetamp I would not expect the data to be changed...

Question 2: Why do UTC+02:00 and Europe/Amsterdam provide different results (here UTC+02:00 does not seem to be DST aware)?

Now when I change the UNIX system timezone setting to Europe/Amsterdam with:

%sh
timedatectl set-timezone Europe/Amsterdam
timedatectl

Output:
                      Local time: Wed 2021-09-22 13:35:49 CEST
                  Universal time: Wed 2021-09-22 11:35:49 UTC
                        RTC time: n/a
                       Time zone: Europe/Amsterdam (CEST, +0200)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

And run the Spark code to convert two timestamp again, I get different results as before:

Output:
Current SPARK timezone: Europe/Amsterdam
2021-09-28 10:30:00
2021-11-28 10:30:00
Current SPARK timezone: UTC+02:00
2021-09-28 10:30:00
2021-11-28 09:30:00
Current SPARK timezone: UTC
2021-09-28 12:30:00
2021-11-28 11:30:00

Question 3: Why are these results now different? Does spark also check system timezone setting besides its own Spark configuration? Why is UTC+02:00 now suddenly DST aware and not Europe/Amsterdam??

So In general I am a bit lost on how to correctly set UNIX and Spark timezones on our cluster so that our logging in Python shows correct timestamps and so that Spark correctly converts timestamp strings to real timestamps AND that our cluster is DST aware. Any help is much appreciated.