spark operator cores and memory setting difference between pyspark and spark

I do use spark operator to deploy spark jobs into k8s.

I find it strange as this works for spark+scala:

  driver:
    cores: 3
    memory: "9G"
    serviceAccount: spark

  executor:
    cores: 3
    instances: 3
    memory: "9G"

but this does not work for pyspark:

  driver:
    cores: 3
    memory: "9G"
    serviceAccount: spark

  executor:
    cores: 3
    instances: 3
    memory: "9G"

So I reduced number of cores for executor from 3 to 1:

  driver:
    cores: 3
    memory: "9G"
    serviceAccount: spark

  executor:
    cores: 1
    instances: 3
    memory: "9G"

Still, I can not get resources.

So I reduced the memory of the executor:

  driver:
    cores: 3
    memory: "4G"
    serviceAccount: spark

  executor:
    cores: 1
    instances: 3
    memory: "4G"

Now, I am able to deploy it.

1. I am using D16 machines so I have 16G, 4vcpu.
2. 3 cores + 9G + 4 pods (3 executors + 1 driver) works when deployed as spark+scala app.
3. It does not work when deployed as pyspark app.

Why?

I have been able to run max:

  driver:
    cores: 3
    memory: "7G"
    serviceAccount: spark

  executor:
    cores: 3
    instances: 3
    memory: "7G"