I have a spark structured streaming job that is running for around last 3 weeks. When I open the Executors tab on spark web UI, it shows
memory used - 36.1GB
total available memory for storage - 3.2GB
For this application executor memory is set to 6GB.
Please refer to the below screenshot of Spark UI
I am not sure how should I interpret this information.