How to get foreground window/process in Java (Using JNA) on MacOS?

There are actually two questions here, foreground window and foreground process. I'll try to answer both.

For the foreground process an easy way using JNA is to map Application Services API. Note that these functions were introduced in 10.9 and are now deprecated, but still work as of 10.15. The newer version is in the AppKit Library, see below.

Create this class, mapping the two functions you'll need:

public interface ApplicationServices extends Library {
    ApplicationServices INSTANCE = Native.load("ApplicationServices", ApplicationServices.class);

    int GetFrontProcess(LongByReference processSerialNumber);
    int GetProcessPID(LongByReference processSerialNumber, IntByReference pid);
}

The "foreground" process can be obtained with GetFrontProcess(). That returns something called a ProcessSerialNumber, a unique 64-bit value used throughout the Application Services API. To translate it for your userspace use, you probably want the Process ID, and GetProcessPID() does that translation for you.

LongByReference psn = new LongByReference();
IntByReference pid = new IntByReference();
ApplicationServices.INSTANCE.GetFrontProcess(psn);
ApplicationServices.INSTANCE.GetProcessPID(psn, pid);
System.out.println("Front process pid: " + pid.getValue());

While the above works, it is deprecated. A new application should use the AppKit Library:

public interface AppKit extends Library {
    AppKit INSTANCE = Native.load("AppKit", AppKit.class);
}

There are multiple other StackOverflow questions regarding the topmost application using this library, such as this one. Mapping all the imports and objects needed is far more work than I have time to do in an answer here, but you might find it useful. It's probably easier to figure out how to use the Rococoa framework (which uses JNA under the hood but has already mapped all of AppKit via JNAerator) to access this API. Some javadocs are here.

There are also solutions using AppleScript that you can execute from Java via command line using Runtime.exec() and capturing output.

With regard to foreground window on the screen, it's a bit more complicated. In my answer to your earlier question on iterating all windows on macOS, I answered how to get a list of all the windows using CoreGraphics via JNA, including a CFDictionary containing more information.

One of those dictionary keys is kCGWindowLayer which will return a CFNumber representing the window layer number. The docs state this is 32-bit, so intValue() is appropriate. The number is the "drawing order" so a higher number will overwrite a lower number. So you can iterate over all the retrieved windows and find the maximum number. This will be the "foreground" layer.

There are some caveats:

• There are actually only 20 layers available. Many things share a layer.
• Layer 1000 is the screensaver. You can ignore layers 1000 and higher.
• Layer 24 is the Dock, usually on top, with Layer 25 (the icons on the dock) at a higher level.
• Layer 0 appears to be the rest of the desktop.
• Which window is "on top" depends on where on the screen you look. Over the dock, the dock will be in the foreground (or the application icon). On the rest of the screen, you need to check the pixel you're evaluating vs. the screen rectangle obtained from the CoreGraphics window. (Use the kCGWindowBounds key which returns a CGRect (a structure with 4 doubles, X, Y, width, height).

You will need to filter to onscreen windows. If you already fetched the list you could use the kCGWindowIsOnscreen key to determine whether the window is visible. It returns a CFBoolean. Since that key is optional you will need to test for null. However, if you are starting from nothing, it would be better to use the kCGWindowListOptionOnScreenOnly Window Option Constant when you initially call CGWindowListCopyWindowInfo().

In addition to iterating all windows, the CGWindowListCopyWindowInfo() function takes a CGWindowID parameter relativeToWindow and you can add (with bitwise or) kCGWindowListOptionOnScreenAboveWindow to the options.

Finally, you might find that limiting to windows associated with the current session may be useful, and you should map CGWindowListCreate() using similar syntax to the CopyInfo() variant. It returns an array of window numbers that you could limit your dictionary search to, or pass that array as an argument to CGWindowListCreateDescriptionFromArray().

As mentioned in my previous answer, you "own" every object you create using Create or Copy functions, and are responsible for releasing them when you are done with them, to avoid memory leaks.

   AppleScriptEngine appleEngine = new apple.applescript.AppleScriptEngine();
    ArrayList<String> processNames = null;
    try {
        String processName = null;
        processNames = (ArrayList<String>) appleEngine
                .eval("tell application \"System Events\" to get name of application processes whose frontmost is true and visible is true");
        if (processNames.size() > 0) {
            processName = processNames.get(0);// the front most process name
        }
        return processName;
    } catch (ScriptException e) {
        log.debug("no app running");
    }