How to configure hostname resolution to use a custom DNS server in Java?

with the following Interface and allowing access to java.net.* it is possible to use own DNS provider with JDK8 and JDK9. The new Provider is installed via "INameService.install(new MyNameService());"

public interface INameService extends InvocationHandler {
    public static void install(final INameService dns) throws IllegalArgumentException, IllegalAccessException, NoSuchFieldException, SecurityException, ClassNotFoundException {
        final Class<?> inetAddressClass = InetAddress.class;
        Object neu;
        Field nameServiceField;
        try {
            final Class<?> iface = Class.forName("java.net.InetAddress$NameService");
            nameServiceField = inetAddressClass.getDeclaredField("nameService");
            neu = Proxy.newProxyInstance(iface.getClassLoader(), new Class<?>[] { iface }, dns);
        } catch(final ClassNotFoundException|NoSuchFieldException e) {
            nameServiceField = inetAddressClass.getDeclaredField("nameServices");
            final Class<?> iface = Class.forName("sun.net.spi.nameservice.NameService");
            neu = Arrays.asList(Proxy.newProxyInstance(iface.getClassLoader(), new Class<?>[] { iface }, dns));
        }
        nameServiceField.setAccessible(true);
        nameServiceField.set(inetAddressClass, neu);
    }

    /**
     * Lookup a host mapping by name. Retrieve the IP addresses associated with a host
     *
     * @param host the specified hostname
     * @return array of IP addresses for the requested host
     * @throws UnknownHostException  if no IP address for the {@code host} could be found
     */
    InetAddress[] lookupAllHostAddr(final String host) throws UnknownHostException;

    /**
     * Lookup the host corresponding to the IP address provided
     *
     * @param addr byte array representing an IP address
     * @return {@code String} representing the host name mapping
     * @throws UnknownHostException
     *             if no host found for the specified IP address
     */
    String getHostByAddr(final byte[] addr) throws UnknownHostException;

    @Override default public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable {
        switch(method.getName()) {
        case "lookupAllHostAddr": return lookupAllHostAddr((String)args[0]);
        case "getHostByAddr"    : return getHostByAddr    ((byte[])args[0]);
        default                 :
            final StringBuilder o = new StringBuilder();
            o.append(method.getReturnType().getCanonicalName()+" "+method.getName()+"(");
            final Class<?>[] ps = method.getParameterTypes();
            for(int i=0;i<ps.length;++i) {
                if(i>0) o.append(", ");
                o.append(ps[i].getCanonicalName()).append(" p").append(i);
            }
            o.append(")");
            throw new UnsupportedOperationException(o.toString());
        }
    }
}

Java provided a new system parameter jdk.net.hosts.file for adding customized DNS records, as specified in the source code

 * The HostsFileNameService provides host address mapping
 * by reading the entries in a hosts file, which is specified by
 * {@code jdk.net.hosts.file} system property
 *
 * <p>The file format is that which corresponds with the /etc/hosts file
 * IP Address host alias list.
 *
 * <p>When the file lookup is enabled it replaces the default NameService
 * implementation
 *
 * @since 9
 */
private static final class HostsFileNameService implements NameService {

Example: we can start a Java application with JVM Option

• -Djdk.net.hosts.file=/path/to/alternative/hosts_file

Where hosts_file content could be:

127.0.0.1    myserver-a.local
10.0.5.10    myserver-b.local
192.168.0.1  myserver-c.local

It will do the DNS trick based on the hosts_file

As Pace indicated, the ability to configure name resolution was removed from the JDK. However, in Java 18, it was replaced with the SPI java.net.spi.InetAddressResolverProvider.

Between Java 8 and Java 18, as the JDK Inet4AddressImpl.c eventually ends up calling getnameinfo, you can still intercept that call with LD_PRELOAD shadowing.

There are libraries which make this very straightforward, such as CWrap.