If the number of partitions is 200, is there a way to get the partition id that is from 0,1,2 to 199? The use case is
final JavaPairRDD<Integer, Integer> baseEntity = ....repartition(200);
baseEntity.mapPartitionsToPair(entities -> {
***get partition id here***
return Iterators.transform(entities, entity ->{
***use partition id here***
}
});
From https://spark.apache.org/docs/latest/api/java/org/apache/spark/TaskContext.html#partitionId--, I found two potential methods, TaskContext. partitionId() and TaskContext.get().partitionID(), but I don't understand the difference. Can someone please help?