How to run arbitrary code on spark executors

Viewed 63

How to run any arbitrary code - which is not related to RDDs - on all of the executors.

lets say I want to save the date into a file on all drivers and executors periodically

Files.write(Paths.get("file.txt"), ZoneddateTime.now().toString().getBytes(StandardCharsets.UTF_8))

This will do it on the driver only.

edit:
I know that I can send files to the executors at spark-submit time, but I want to create/update files runtime.

0 Answers
Related