GWT FocusPanel mousedownHandler is not calling immediately in empty space in IE 11

I have a GWT FocusPanel, and this FocusPanel has it's mouse down and up handler, I noticed the mouse down handler is not calling immediately when the mouse is down, it always get delayed called in my IE 11 browser, but it works well in Firefox.

UPDATE: the mouse down handler is not calling immediately in white space only in focus panel in IE browser, it works fine in non-empty space.

My GWT layout structure is:

--Vertical Panel

     --Simple Panel

        --Flex table

          -- Focus Panel(this is where the mousedown handler event located)

Here is my code in GWT:

                simplicity = new FocusPanel(
                        Gelfling.Search_reconstructionEssence);
                simplicity.addMouseUpHandler(new MouseUpHandler() {
                    public void onMouseUp(MouseUpEvent event) {
                        callMouseUpEssenceTop(event);
                    }
                });

                simplicity.addMouseDownHandler(new MouseDownHandler() {

                    @Override
                    public void onMouseDown(MouseDownEvent event) {
                        callMouseDownEssenceTop(event);

                    }
                });

   public static native void callMouseUpEssenceTop(MouseUpEvent event)
    /*-{
        top.onMouseUpEssenceTop(event);
    }-*/;

    public static native void callMouseDownEssenceTop(MouseDownEvent event)
    /*-{
        top.onMouseDownEssenceTop(event);
    }-*/;

Here is the JS code:

function onMouseDownEssenceTop(e) {
console.log("onMouseDownEssenceTop");

//following is my business logic


}

Any idea? Thanks.