ToAscii/ToUnicode in a keyboard hook destroys dead keys

Question

It seems that if you call ToAscii() or ToUnicode() while in a global WH_KEYBOARD_LL hook, and a dead-key is pressed, it will be 'destroyed'.

For example, say you've configured your input language in Windows as Spanish, and you want to type an accented letter á in a program. Normally, you'd press the single-quote key (the dead key), then the letter "a", and then on the screen an accented á would be displayed, as expected.

But this doesn't work if you call ToAscii() or ToUnicode() in a low-level keyboard hook function. It seems that the dead key is destroyed, and so no accented letter á shows up on screen. Removing a call to the above functions resolves the issue... but unfortunately, I need to be able to call those functions.

I Googled for a while, and while a lot of people seemed to have this issue, no good solution was provided.

Any help would be much appreciated!

EDIT: I'm calling ToAscii() to convert the virtual-key code and scan code received in my LowLevelKeyboardProc hook function into the resulting character that will be displayed on screen for the user.

I tried MapVirtualKey(kbHookData->vkCode, 2), but this isn't as "complete" a function as ToAscii(); for example, if you press Shift + 2, you'll get '2', not '@' (or whatever Shift + 2 will produce for the user's keyboard layout/language).

ToAscii() is perfect... until a dead-key is pressed.

EDIT2: Here's the hook function, with irrelevant info removed:

LRESULT CALLBACK keyboard_LL_hook_func(int code, WPARAM wParam, LPARAM lParam) {

    LPKBDLLHOOKSTRUCT kbHookData = (LPKBDLLHOOKSTRUCT)lParam;
    BYTE keyboard_state[256];

    if (code < 0) {
        return CallNextHookEx(keyHook, code, wParam, lParam);
    }

    WORD wCharacter = 0;

    GetKeyboardState(&keyboard_state);
    int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
                     keyboard_state, &wCharacter, 0);

    /* If ta == -1, a dead-key was pressed. The dead-key will be "destroyed"
     * and you'll no longer be able to create any accented characters. Remove
     * the call to ToAscii() above, and you can then create accented characters. */

    return CallNextHookEx(keyHook, code, wParam, lParam);
}

user_99834 · Answer

stop using ToAscii() and use ToUncode()
remember that ToUnicode may return you nothing on dead keys - this is why they are called dead keys.
Any key will have a scancode or a virtual key code but not necessary a character.

You shouldn't combine the buttons with characters - assuming that any key/button has a text representation (Unicode) is wrong.

So:

for input text use the characters reported by Windows
for checking button pressed (ex. games) use scancodes or virtual keys (probably virtual keys are better).
for keyboard shortcuts use virtual key codes.

user_8287405 · Answer

I encountered this issue while creating a key logger in C# and none of the above answers worked for me.

After a deep blog searching, I stumbled across this keyboard listener which handles dead keys perfectly.

user_5467445 · Answer

Here is a full code which covers dead keys and shortcut keys using ALT + NUMPAD, basically a full implementation of a TextField input handling:

    [DllImport("user32.dll")]
    public static extern int ToUnicode(uint virtualKeyCode, uint scanCode, byte[] keyboardState, [Out, MarshalAs(UnmanagedType.LPWStr, SizeConst = 64)] StringBuilder receivingBuffer, int bufferSize, uint flags);

    private StringBuilder _pressCharBuffer = new StringBuilder(256);
    private byte[] _pressCharKeyboardState = new byte[256];

    public bool PreFilterMessage(ref Message m)
    {
        var handled = false;

        if (m.Msg == 0x0100 || m.Msg == 0x0102)
        {

            bool isShiftPressed = (ModifierKeys & Keys.Shift) != 0;
            bool isControlPressed = (ModifierKeys & Keys.Control) != 0;
            bool isAltPressed = (ModifierKeys & Keys.Alt) != 0;
            bool isAltGrPressed = (ModifierKeys & Keys.RMenu) != 0;

            for (int i = 0; i < 256; i++)
                _pressCharKeyboardState[i] = 0;

            if (isShiftPressed)
                _pressCharKeyboardState[(int)Keys.ShiftKey] = 0xff;

            if (isAltGrPressed)
            {
                _pressCharKeyboardState[(int)Keys.ControlKey] = 0xff;
                _pressCharKeyboardState[(int)Keys.Menu] = 0xff;
            }

            if (Control.IsKeyLocked(Keys.CapsLock))
                _pressCharKeyboardState[(int)Keys.CapsLock] = 0xff;

            Char chr = (Char)0;

            int ret = ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);

            if (ret == 0)
                chr = Char.ConvertFromUtf32(m.WParam.ToInt32())[0];
            if (ret == -1)
                ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);
            else if (_pressCharBuffer.Length > 0)
                chr = _pressCharBuffer[0];

            if (m.Msg == 0x0102 && Char.IsWhiteSpace(chr))
                chr = (Char)0;


            if (ret >= 0 && chr > 0)
            {

            //DO YOUR STUFF using either "chr" as special key (UP, DOWN, etc..) 
            //either _pressCharBuffer.ToString()(can contain more than one character if dead key was pressed before)
            //and don't forget to set the "handled" to true, so nobody else can use the message afterwards

            }
        }

        return handled;
    }

user_3716009 · Answer

This works for me

byte[] keyState = new byte[256];

//Remove this if using
//GetKeyboardState(keyState); 

//Add only the Keys you want
keysDown[(int)Keys.ShiftKey] = 0x80; // SHIFT down
keysDown[(int)Keys.Menu] = 0x80; // ALT down
keysDown[(int)Keys.ControlKey] = 0x80; // CONTROL down
  
//ToAscii should work fine         
if (ToAscii(myKeyboardStruct.VirtualKeyCode, myKeyboardStruct.ScanCode, keyState, inBuffer, myKeyboardStruct.Flags) == 1)
{
    //do something
}

user_1795050 · Answer

It is known that ToUnicode() and its older counterpart ToAscii() can change keyboard state of the current thread and thus mess with dead keys and ALT+NUMPAD keystrokes:

As ToUnicodeEx translates the virtual-key code, it also changes the state of the kernel-mode keyboard buffer. This state-change affects dead keys, ligatures, alt+numpad key entry, and so on. It might also cause undesired side-effects if used in conjunction with TranslateMessage (which also changes the state of the kernel-mode keyboard buffer).

To avoid that you can do your ToUnicode() call in a separate thread (it will have a separate keyboard state) or use a special flag in wFlags param that is only documented in ToUnicode() docs:

If bit 2 is set, keyboard state is not changed (Windows 10, version 1607 and newer)

I think it should work with ToAscii() too but better not use this old ANSI codepage-dependant method. Use ToUnicode() API instead that can even return ligatures and UTF-16 surrogate pairs - if keyboard layout have them. Some do.

See Asynchronous input vs synchronous input, a quick introduction for the reason behind this.

ToAscii/ToUnicode in a keyboard hook destroys dead keys

9 Answers