Cannot successfully init controller by ControllerFactory that returns a kotlin singleton object in javafx

I have a javafx project and I want to access the main controller by kotlin singleton reference.

But the injection cannot succeed.

A minimal example on the below:

App.kt

class App: Application() {
    override fun start(primaryStage: Stage) {

        val loader = FXMLLoader(javaClass.getResource("Window.fxml")).also { it.setControllerFactory { Controller } }
        val root = loader.load<Parent>()
        val scene = Scene(root, 400.0, 600.0)

        primaryStage.scene = scene
        primaryStage.show()
    }

    fun main(vararg args: String) {
        launch(*args)
    }
}

Controller.kt

object Controller : Initializable {

    @FXML private lateinit var button: Button
    @FXML fun buttonAction() { println("Clicked") }

    override fun initialize(location: URL?, resources: ResourceBundle?) {
        button.text = "Click Me"
    }
}

Window.fxml

<?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.control.*?>
<?import javafx.scene.layout.*?>

<AnchorPane xmlns="http://javafx.com/javafx"
            xmlns:fx="http://javafx.com/fxml"
            fx:controller="Controller"
            prefHeight="400.0" prefWidth="600.0">
    <Button fx:id="button" onAction="#buttonAction"/>
</AnchorPane>

Then I got the Exception:

Exception in Application start method
Exception in thread "main" java.lang.RuntimeException: Exception in Application start method
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:900)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(LauncherImpl.java:195)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javafx.fxml.LoadException: 
/D:/WorkPlace/Kotlin/LabelPlusFX/target/test-classes/window.fxml

    at javafx.fxml/javafx.fxml.FXMLLoader.constructLoadException(FXMLLoader.java:2625)
    at javafx.fxml/javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2603)
    at javafx.fxml/javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2466)
    at javafx.fxml/javafx.fxml.FXMLLoader.load(FXMLLoader.java:2435)
    at DemoApp.start(DemoApp.kt:16)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
    at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    ... 1 more
Caused by: kotlin.UninitializedPropertyAccessException: lateinit property button has not been initialized
    at Controller.initialize(Controller.kt:20)
    at javafx.fxml/javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2573)
    ... 12 more

Why cannot fxml do injection successfully?