I've been trying to make a binary semaphore that will be able to safely block execution of a method running on the event dispatch thread (EDT) without actually blocking the thread from handling more events. This may initially seem impossible, but Java has some built-in functionality related to this, but I can't quite get it to work.

Use Case

Currently, if you show a modal swing dialog from the EDT, it will appear to block the EDT (because your method that displayed the modal dialog will not continue onto the next line until the dialog is closed), but really there's some under-the-hood magic that makes the EDT enter a new event loop which will continue to dispatch events until the modal dialog is closed.

My team currently has applications that are very slowly migrating from swing to JavaFX (a somewhat tricky transition) and I wanted to be able to display modal JavaFX dialogs from the AWT event dispatch thread in the same way that swing modal dialogs can be shown. It seemed like having some sort of EDT-safe semaphore would meet this use case and likely come in handy for other uses down the road.

Approach

java.awt.EventQueue.createSecondaryLoop() is a method that creates a SecondaryLoop object, which you can then use to kick off a new event handling loop. When you call SecondaryLoop.enter(), the call will block while it processes a new event loop (note that the call blocks, but the thread is not blocked because it is continuing in an event processing loop). The new event loop will continue until you call SecondaryLoop.exit() (that's not entirely true, see my related SO question).

So I've created a semaphore where a blocking call to acquire results in waiting on a latch for a normal thread, or entering a secondary loop for the EDT. Each blocking call to acquire also adds an unblocking operation to be called when the semaphore is freed (for a normal thread, it just decrements the latch, for the EDT, it exits the secondary loop).

Here is my code:

import java.awt.EventQueue;
import java.awt.SecondaryLoop;
import java.awt.Toolkit;
import java.util.Stack;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Semaphore;

@SuppressWarnings("serial")
public class EventDispatchThreadSafeBinarySemaphore extends Semaphore{

    /** Operations used to unblock threads when a semaphore is released.
     * Must be a stack because secondary loops have to be exited in the
     * reverse of the order in which they were entered in order to unblock
     * the execution of the method that entered the loop.
     */
    private Stack<Runnable> releaseOperations = new Stack<>();

    private boolean semaphoreAlreadyAcquired = false;


    public EventDispatchThreadSafeBinarySemaphore() {
        super(0);
    }

    @Override
    public boolean isFair() {
        return false;
    }

    @Override
    public void acquire() throws InterruptedException {

        Runnable blockingOperation = () -> {};

        synchronized(this) {
            if(semaphoreAlreadyAcquired) {

                //We didn't acquire the semaphore, need to set up an operation to execute
                //while we're waiting on the semaphore and an operation for another thread
                //to execute in order to unblock us when the semaphore becomes available

                if(EventQueue.isDispatchThread()) {

                    //For the EDT, we don't want to actually block, rather we'll enter a new loop that will continue
                    //processing AWT events.
                    SecondaryLoop temporaryAwtLoop = Toolkit.getDefaultToolkit().getSystemEventQueue().createSecondaryLoop();

                    releaseOperations.add(() -> temporaryAwtLoop.exit());

                    blockingOperation = () -> {

                        if(!temporaryAwtLoop.enter()) {
                            //I don't think we'll run into this, but I'm leaving this here for now for debug purposes
                            System.err.println("Failed to enter event loop");
                        }
                    };
                }
                else {

                    //Non-dispatch thread is a little simpler, we'll just wait on a latch
                    CountDownLatch blockedLatch = new CountDownLatch(1);
                    releaseOperations.add(() -> blockedLatch.countDown());
                    blockingOperation = () -> {
                        try {
                            blockedLatch.await();
                        } catch (InterruptedException e) {
                            //I'll worry about handling this better once I have the basics figured out
                            e.printStackTrace();
                        }
                    };
                }
            }
            else {
                semaphoreAlreadyAcquired = true;
            }
        }

        //This part must be executed outside of the synchronized block so that we don't block
        //the EDT if it tries to acquire the semaphore while this statement is blocked
        blockingOperation.run();

    }

    @Override
    public void release() {
        synchronized(this) {
            if(releaseOperations.size() > 0) {
                //Release the last blocked thread
                releaseOperations.pop().run();
            }
            else {
                semaphoreAlreadyAcquired = false;
            }
        }
    }

}

And here is my relevant JUnit test code (I apologize for the large size, this is the smallest minimum verifiable example I've been able to come up with so far):

public class TestEventDispatchThreadSafeBinarySemaphore {

    private static EventDispatchThreadSafeBinarySemaphore semaphore;
        //See https://stackoverflow.com/questions/58192008/secondaryloop-enter-not-blocking-until-exit-is-called-on-the-edt
        //for why we need this timer
        private static Timer timer = new Timer(500, null);
        @BeforeClass
    public static void setupClass() {
        timer.start();
    }

    @Before
    public void setup() {
        semaphore = new EventDispatchThreadSafeBinarySemaphore();
    }
        @AfterClass
    public static void cleanupClass() {
        timer.stop();
    }

        //This test passes just fine
        @Test(timeout = 1000)
    public void testBlockingAcquireReleaseOnEDT() throws InterruptedException {

        semaphore.acquire();

        CountDownLatch edtCodeStarted = new CountDownLatch(1);
        CountDownLatch edtCodeFinished = new CountDownLatch(1);

        SwingUtilities.invokeLater(() -> {
            //One countdown to indicate that this has begun running
            edtCodeStarted.countDown();
            try {
                semaphore.acquire();
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            //This countdown indicates that it has finished running
            edtCodeFinished.countDown();

        });

        //Ensure that the code on the EDT has started
        edtCodeStarted.await();

        assertEquals("Code on original AWT event thread should still be blocked", 1, edtCodeFinished.getCount());

        //Ensure that things can still run on the EDT
        CountDownLatch edtActiveCheckingLatch = new CountDownLatch(1);
        SwingUtilities.invokeLater(() -> edtActiveCheckingLatch.countDown());

        //If we get past this line, then we know that the EDT is live even though the 
        //code in the invokeLater call is blocked
        edtActiveCheckingLatch.await();

        assertEquals("Code on original AWT event thread should still be blocked", 1, edtCodeFinished.getCount());

        semaphore.release();

        //If we get past this line, then the code on the EDT got past the semaphore
        edtCodeFinished.await();
    }

        //This test fails intermittently, but so far only after the previous test was run first
    @Test(timeout = 10000)
    public void testConcurrentAcquiresOnEDT() throws InterruptedException {

        int numThreads =100;

        CountDownLatch doneLatch = new CountDownLatch(numThreads);

        try {
            semaphore.acquire();

            //Queue up a bunch of threads to acquire and release the semaphore
            //as soon as it becomes available
            IntStream.range(0, numThreads)
                    .parallel()
                    .forEach((threadNumber) -> 
                        SwingUtilities.invokeLater(() -> {
                            try {
                                semaphore.acquire();
                            } catch (InterruptedException e) {
                                e.printStackTrace();
                            }
                            finally {
                                semaphore.release();
                                //Count down the latch to indicate that the thread terminated
                                doneLatch.countDown();
                            }
                        })
                    );

            semaphore.release();

            doneLatch.await();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }
}

The Problem

testConcurrentAcquiresOnEDT will sometimes pass and sometimes fail. I believe that I know why. I dug into the Java source code and in WaitDispatchSupport (the concrete implementation of SecondaryLoop), the loop basically continues dispatching events until a flag called keepBlockingEDT is cleared. It will check this between events. When I call exit, it will clear that flag and send an event to wakeup the event queue in case it was waiting for more events. However, it will not cause the enter() method to immediately exit (and I don't think there's anyway it possibly could).

So here's how the deadlock results:

• The main thread acquires the semaphore
• The EDT thread tries to acquire the semaphore, but it is already acquired, so it:
  • Creates a new secondary loop
  • Creates a Runnable that will exit the new secondary loop and pushes it to the releaseOperations stack
  • Enters the secondary loop, causing execution to block (note that this last step is by necessity outside of the synchronized block
• The main thread releases the semaphore, which causes the following to happen:
  • The releaseOperations stack is popped and it calls exit on the secondary loop
  • The exit call, sets the keepBlockingEDT flag for that secondary loop to be set to false
• Back in the EDT, it just got done checking the keepBlockingEDT flag (right before it was set to false) and it is fetching the next event.
• It turns out that the next event is another runnable that blocks on the semaphore, so it tries to acquire it
• This creates another SecondaryLoop on top of the original SecondaryLoop and enters it
• At this point, the original SecondaryLoop has already had it's keepBlockingEDT flag cleared and it would be able to stop blocking, except that it is currently blocked running the second SecondaryLoop. The second SecondaryLoop won't ever have exit called on it because no one actually has the semaphore acquired right now, therefore we block forever.

I've been working on this for a few days and every idea I come up with is a dead end.

I believe that I have a possible partial solution, which is to simply not allow more than one thread to be blocked on the semaphore at a time (if another thread tries to acquire it, I'll just throw an IllegalStateException). I could still have multiple secondary loops going if they each use their own semaphore, but each semaphore would create at most 1 secondary loop. I think this would work and it will meet my most likely use case just fine (because mostly I just want to show a single JavaFX modal dialog from the event thread). I just wanted to know if anyone else had other ideas because I feel like I got close to making something pretty cool, but it just doesn't quite work.

Let me know if you have any ideas. And "I'm pretty sure this is impossible and here's why..." is an acceptable answer as well.