Apache Spark: get the executor ID in the map reduce pipeline

user_10461625 0
Viewed 819

Small question regarding how to get the Spark executor ID in a Apache Spark job please.

I have a very straightforward piece of code:

final Dataset<Row> rowDataSet = sparkSession.read()[...].load();
final Dataset<String> stringDataSet = rowDataSet
        .map(
                (MapFunction<Row, String>) row -> 
                        doSomeTransformationFromRowToStringUsingSparkExecutorID(row, SparkEnv.executorId()), Encoders.STRING()
        );
stringDataSet.show();

And the question is regarding the doSomeTransformationFromRowToStringUsingSparkExecutorID method.

This method needs the spark executor ID the row is being processed, in order to do some transformation.

As I need the Spark executor ID, I went to use SparkEnv.executorId() found in the official documentation.

Unfortunately, above is not working, as Non-static method 'executorId()' cannot be referenced from a static context

Is SparkEnv.executorId() even the good way to get the executor ID in this scenario?

If yes, how to workaround this Non-static method 'executorId()' issue please?

If not, what is the best alternative to get the executor ID please?

Thank you

apache-spark java
1 Answers

According to the documentation SparkEnv class has static method get for getting SparkEnv instance. And on this instance you can call executorId() method (which is not static):

import org.apache.spark.SparkEnv

SparkEnv sparkEnv = SparkEnv.get();
String executorId = sparkEnv.executorId();

final Dataset<Row> rowDataSet = sparkSession.read()[...].load();
final Dataset<String> stringDataSet = rowDataSet
        .map(
                (MapFunction<Row, String>) row ->
                        doSomeTransformationFromRowToStringUsingSparkExecutorID(row, SparkEnv.get().executorId()), Encoders.STRING()
        );
stringDataSet.show();
user_67706140
Related