I guess this is 2 related questions:
Local Disk Encryption
In the documentation it says
Spark does not support encrypted data on local disk, such as intermediate data written to a local disk by an executor process when the data does not fit in memory.
Does this mean by default on-disk encryption is not supported but it can be enabled? Because later on it says:
- (Optional) If you want to enable optional on-disk block encryption, which applies to both shuffle and RDD blocks on disk, complete the following steps:
Data In Transit
Once again it says
When encryption is enabled, Spark encrypts all data that is moved across nodes in a cluster on behalf of a job
But it also says
Additionally, wire encryption is not supported for shuffle files, cached data, and other application files
So what kind of data is encrypted as it travels between the nodes?