This is admittedly a pretty gross solution, but I will leave it here if it helps.

Chaining Awt, Swing, and JavaFX into a Monstrosity

As far as I am aware, this is not possible with merely JavaFX alone. However, we can make use of a few compatibilities to achieve what you want.

• JavaFX has compatibility with Java Swing: See the documentation for examples.
• Java Swing has compatibility with Awt: A few Java Swing classes extend java.awt.Window for compatibility purposes, which is accepted by the method you linked - setWindowProgressState​(Window,Taskbar.State state). Instances of the following javax.swing classes can be passed to the method you linked: javax.swing.JDialog, javax.swing.JFrame, javax.swing.JWindow, javax.swing.Popup

By chaining interoperability with the various Java UI frameworks, you can set the windows Taskbar state.

Here is a demonstration:

import javafx.embed.swing.JFXPanel;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.layout.StackPane;
import javafx.scene.paint.Color;

import javax.swing.*;
import java.awt.*;

public class Test {
    public static void main(String[] args) {
        // initialize a Swing application
        JFrame frame = new JFrame("JavaFX with Swing abuse");

        // create a Swing compatible JavaFX container
        JFXPanel fxPanel = new JFXPanel();
        // initialize JavaFX container
        StackPane root = new StackPane();
        Scene scene = new Scene(root, 600, 400, Color.ALICEBLUE);
        // add elements
        Button button = new Button();
        button.setText("Say 'Hello World'");
        button.setOnAction(event -> System.out.println("Hello World!"));
        root.getChildren().add(button);
        fxPanel.setScene(scene);

        // configure Swing JFrame
        frame.add(fxPanel);
        frame.setSize(600, 400);
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setVisible(true);

        // use Awt compatibility with Swing
        Taskbar taskbar = Taskbar.getTaskbar();
        taskbar.setWindowProgressState(frame, Taskbar.State.ERROR);
        taskbar.setWindowProgressValue(frame, 50);
    }
}

On Windows 10, this is correctly displaying the Taskbar progress state:

Limitations

I cannot say that I recommend this, but regardless, it is possible. There might be a few limitations, however:

• This initializes almost every Java UI framework's subsystem, which could have significant processing and memory overheads. Mixing the various Java UI frameworks is generally ill-advised for this reason.
• Using Java Swing to render your JavaFX application might have serious drawbacks relating to performance or even compatibility with the rest of your application.

JavaFX does not use AWT to create the window, as opposed to Swing.

You could embed a JavaFX Scene into a Swing application, like this:

// On EDT:
JFrame frame = new JFrame();
JFXPanel jfxPanel = new JFXPanel();
frame.setContentPane(jfxPanel);

// On JavaFX Application Thread:
jfxPanel.setScene(new Scene(...));

Then, JFrame extends java.awt.Frame which is a java.awt.Window.