I have spent way too much time trying to find how to properly set the classpaths and their precedence in submitting a mapreduce job through Java API of AWS EMR. Documentation seems to not provide any details, not have I been successful in finding any proper examples through days of Googling. Can someone help with setting the class paths properly?
The basic issue is that my code uses various java libraries (for example guava version >= 17), and the default versions of libraries available on the EMR clusters created are very old versions, and I want to supply the new versions of these jars to my jobs: Suppose I have all the needed jars in the location "/home/hadoop/lib/". How to set these as the preferred versions (or replace the ones in default paths)?
I have tried various ways, but with no success: Setting the "HADOOP_CLASSPATH" in "hadoop-env" classification and "mapreduce.application.classpath" in "mapred-site" classification through a JSON config at cluster creation.
Additionally, is there a list available somewhere of various valid environment variables that can be set?