When I am working with Foundry, there are some options to configure my job profile: num_executors, driver_memory, executor_memory, etc. I am wondering which is the equivalence of these profiles to the worker types in AWS. If I use AWS Glue Studio, I can select G 1x (4vCPU, 16 GB RAM) or G 2x (8vCPU, 32 GB RAM). Additionally, I can select the max. number of workers.
Can someone help me to understand this?
You can go to the code repositories settings and load spark profiles to be available to your transforms.
From there you can import the spark profiles you want, which include similar options. Then you can import into your transform using profiles https://www.palantir.com/docs/foundry/code-repositories/spark-profiles/
from transforms.api import configure
@configure(profile=['EXECUTOR_MEMORY_LARGE', 'EXECUTOR_CORES_MEDIUM', 'NUM_EXECUTORS_64'])
@transform(
# your input dataset(s)
my_input=Input("/path/to/input/dataset"),
# your output dataset
my_ouput=Output("/path/to/output/dataset"),
)
# your data transformation code
def my_compute_function(my_input):
return my_input
Scroll through the spark profiles list and you should find other settings available to you. If any shows up locked, then you'll have to request your platform administrator to unlock it for you. It's often because they are expensive. i.e.: NUM_EXECUTORS_512 or EXECUTOR_CORES_EXTRA_LARGE