I am currently making a Helm chart that will instantiate a full HDFS cluster with Kerberos and AD / LDAP support optional. In order to achieve the Kerberos / LDAP part, I've made it so that I have some test containers that produce the Kerberos and LDAP parts; specifically, a Kerberos KDC server and an OpenLDAP server. Once the test containers demonstrably work, it would be easy to swap out the network addresses of my test containers to some user defined addresses that point to real AD / Kerberos servers, thus making a real production environment. I've also made a test program in Go that also runs in its own container, which uses the gokrb5 and hdfs libraries to first authenticate to my Kerberos principal test@HDFS.COM to be granted a ticket, then uses that ticket to perform a root directory crawl through the hdfs library.
There seems to be an issue where for whatever reason it may be, HDFS does not see test@HDFS.COM the Kerberos principal and test the OpenLDAP user as being the same person. The Namenode logs for when I try and perform the root directory crawl from the Go container look like this:
2021-08-16 16:27:31,811 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for test@HDFS.COM (auth:KERBEROS)
2021-08-16 16:27:32,005 INFO org.apache.hadoop.ipc.Server: Connection from 172.17.0.4:44682 for protocol org.apache.hadoop.hdfs.protocol.ClientProtocol is unauthorized for user test (auth:PROXY) via test@HDFS.COM (auth:KERBEROS)
2021-08-16 16:27:32,011 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 172.17.0.4 threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: test@HDFS.COM is not allowed to impersonate test]
I have seen in articles like this by HortonWorks / Cloudera that this issue could be resolved by configuring the hadoop.security.auth_to_local setting in core-site.xml to this:
<property>
<name>hadoop.security.auth_to_local</name>
<value>
DEFAULT
</value>
</property>
Where the DEFAULT parameter simply turns a Kerberos principal like test@HDFS.COM into just test, which should string match the OpenLDAP user test. I have done exactly this, which still resulted in the same logs shown above. I know there is also a sort of "hack" that exists where you can override this "impersonation" error on a per-user basis like this in hdfs-site.xml:
<property>
<name>hadoop.proxyuser.test.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.test.groups</name>
<value>*</value>
</property>
This does work for strictly test purposes (and I was able to perform my directory crawl); however, this cannot be a permanent solution, since I would have to manually add every AD user that exists in hdfs-site.xml and it's obvious how this "solution" does not scale at all.
I will say that I am absolutely not an expert in either Kerberos or OpenLDAP, so I will list exactly what I've configured in my OpenLDAP container here if that ends up being the issue.
I am using this docker image to provide an OpenLDAP server. This is an output from ldapsearch of how my current test user/group looks:
# extended LDIF
#
# LDAPv3
# base <dc=domain,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# domain.com
dn: dc=domain,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: DOMAIN
dc: domain
# users, domain.com
dn: ou=users,dc=domain,dc=com
objectClass: organizationalUnit
ou: users
# groups, domain.com
dn: ou=groups,dc=domain,dc=com
objectClass: organizationalUnit
ou: groups
# test, groups, domain.com
dn: cn=test,ou=groups,dc=domain,dc=com
cn: test
objectClass: top
objectClass: posixGroup
gidNumber: 1000
# test, users, domain.com
dn: uid=test,ou=users,dc=domain,dc=com
cn: test
givenName: test
sn: test
uid: test
uidNumber: 1000
gidNumber: 1000
homeDirectory: /home/test
objectClass: top
objectClass: posixAccount
objectClass: shadowAccount
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
loginShell: /bin/bash
userPassword:: e1NTSEF9WGRRb2FJTFRnSm1rWlJzRHc1TUtwV2ppb0k0UUNiRkg=
# search result
search: 2
result: 0 Success
# numResponses: 6
# numEntries: 5
There may also be issues with my LDAP search filters. Here are mine:
userFilter: (uid=*{0}*)
groupFilter: (objectClass=posixGroup)