I call kinit keytab right before spark-submit in my shell driver script. The thing is, its working by itself, but when I call the shell driver scrip it through Oozie, I got this error:
Stdoutput py4j.protocol.Py4JJavaError: An error occurred while calling
o49.saveAsTextFile.
Stdoutput : org.apache.hadoop.ipc.RemoteException(java.io.IOException):
Delegation Token can be issued only with kerberos or web authentication
The issue is probably here
file.coalesce(1,True).saveAsTextFile(FQDNofHadoop+output)
EDIT: In my script is: kinit -k -t /home/me/me.keytab me@DOMAIN.HAD
EDIT: Working solution:
I used
spark-submit --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \
and execution of pyspark script in oozie worked with no error (even writing into hive table and so on). Log4j logger didn't work (with standalone scripts it does), but at least print() (stdout into yarn logs) yes ...
Thx