How to stop hadoop spamming log with "WARN HadoopFileSystemOwner: found no group information for myname (auth:SIMPLE)"

Question

I am running a spark app on yarn in AWS EMR and I am getting spammed by MBs of this error: 19/09/30 17:09:29 WARN HadoopFileSystemOwner: found no group information for myname (auth:SIMPLE), using myname as primary group

Following advice from Cloudera here, I tried id myname on my namenode and I see myname under uid, gid and groups.

hdfs groups myname on namenode also shows:
myname : myname

How can I fix or stop this spam of my log?

user_9241362 · Answer

change/add above property in core-site.xml

 
  hadoop.http.staticuser.user 
  yourname

user_2202107 · Answer

The way 'hdfs groups' works is by sending an RPC request with the username to one of the NameNodes. When using the default ShellBasedUnixGroups plugin, the NameNode that received the request will run a 'id -gn username' command as a forked process on its own host and collect the output.

Source: https://community.cloudera.com/t5/Support-Questions/User-not-returning-any-groups-for-hdfs-groups-lt-ID-gt/m-p/71759

How to stop hadoop spamming log with "WARN HadoopFileSystemOwner: found no group information for myname (auth:SIMPLE)"

2 Answers