Yarn ApplicationMaster Log location

Viewed 635

I enabled Yarn Log aggregation using Spark on Cloudera, but when the spark job failed, I run

yarn logs -applicaitonId <id>

or

yarn logs -applicationId <id> -am ALL

All I got are working logs in stderr

But in Yarn failure diagnose line, I managed to see

Diagnostics:    User class threw exception: java.lang.reflect.InvocationTargetException

The tricky part is I can't find this exception anywhere, including the /var/log/hadoop-yarn log folder location on each machine as well.

I need this exception details so that I could understand which Jar is missing or what exactly is going wrong with the spark environment settings.

Any ideas how I can find this log location? I found some similar questions with below logs:

17/08/16 02:28:06 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.reflect.InvocationTargetException

java.lang.reflect.InvocationTargetException

            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

            at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

Hence I was trying to locate the ApplicationMaster log location and maybe that's different from what we usually do on container logs.

0 Answers
Related