I have a requirement to load some custom jars on executors depending upon the tasks they will be performing and these jars will not be available on classpath (We have our valid reasons for this).
Current solution: We are able to solve this to some extent by providing wrapper around spark functions like map/flatMap , We are achieving this by intercepting Function class "call" method and creating classloader with our new jars. Jars that we need to load are shipped via SparkFiles.
Challenge: We are stuck where we need to support all other spark API methods that doesn't support or have any functional interface for example reading and writing methods.
Idea: One thought was to use AOP and intercept org.apache.spark.scheduler.ResultTask.runTask(..)) on the executor side but couldn't get this working.
Any help would be appreciated.