Custom Carbon key event handler fails after mouse events

Unclear if you have an NSTextField as your menu view, but if you use one then it's easy to setup a delegate for that text field that can get the current contents of the field as the user types (this takes care of them moving backwards with the arrow keys and then deleting characters, using the delete key, etc). Your delegate implements the appropriate delegate method and gets called each time the text changes:

extension CustomMenuItemViewController: NSTextFieldDelegate {
    func controlTextDidChange( _ obj: Notification) {
        if let postingObject = obj.object as? NSTextField {
            let text = postingObject.stringValue
            print("the text is now: \(text)")
        }
    }
}

Just to confirm this works as expected, I created the ViewController class for the custom menu item views (label + edit field) in a xib file and then dynamically built a simple test menu with a single menu item that has the custom view controller's view and added it to the menubar inside my app delegate:

func installCustomMenuItem() {
    let menuBarItem = NSMenuItem(title: "Test", action: nil, keyEquivalent: "")
    let menu = NSMenu(title: "TestMenu" )
    let subMenuBarItem = NSMenuItem(title: "Custom View", action: nil, keyEquivalent: "")

    subMenuBarItem.view = menuItemVC.view
    menu.addItem(subMenuBarItem)
    menuBarItem.submenu = menu
    NSApp.mainMenu?.addItem(menuBarItem)
}

Looks like this after I typed "hello":

And you can from the console that my handler got called for every character typed:

the text is now: H 
the text is now: He 
the text is now: Hel 
the text is now: Hell
the text is now: Hello

Your situation is probably a little different, but it seems like this approach is very clean and might work for you. If it won't for some reason, add a clarifying comment and we'll see if we can't make it work for you.

Addition:

It occurred to me that you might wish to not use NSTextField and so I was curious if it was as easy to do this with a custom view and it's relatively easy.

Make a subclass of NSView:

class CustomMenuView: NSView {

    override var acceptsFirstResponder: Bool {
        return true
    }

    override func draw(_ dirtyRect: NSRect) {
        super.draw(dirtyRect)

        // Drawing code here.
    }

    override func keyDown(with event: NSEvent) {
        print("key down with character: \(String(describing: event.characters)) " )
    }
}

Set the class of your root view in the custom view controller to be of this type of class and then all done as before - view controller loaded in applicationDidFinishLaunching and menus built and view controller's view (which is now a CustomMenuView) is set as the menuBarItem.view.

That's it. You now get your keyDown method called for every key down when the menu is dropped down.

key down with character: Optional("H") 
key down with character: Optional("e") 
key down with character: Optional("l") 
key down with character: Optional("l") 
key down with character: Optional("o") 
key down with character: Optional(" ")
key down with character: Optional("T") 
key down with character: Optional("h") 
key down with character: Optional("i") 
key down with character: Optional("s") 
key down with character: Optional(" ") 
key down with character: Optional("i") 
key down with character: Optional("s") 
key down with character: Optional(" ") 
key down with character: Optional("c") 
key down with character: Optional("o") 
key down with character: Optional("o") 
key down with character: Optional("l") 

:)

Now your custom view (and subviews if you like) can do their own drawing and so on.

Addition with requested sample without the ViewController:

// Simple swift playground test
// The pop-up menu will show up onscreen in the playground at a fixed location.   
// Click in the popup and then all key commands will be logged.
// The ViewController in my example above may be taking care of putting the custom view in the responder chain, or the fact that it's in a menubar and being invoked via a MenuItem might be.
// I'd suggest trying it in the actual environment rather than in a playground. In my test app you click the menu name in the menubar to drop down the menu and it is added to the responder chain and works as expected without having to click in the menu first to get the events flowing.
// There is no reason you need to be hooking events either with carbon events or the newer format.  If you're in the responder chain of and implement the necessary, method then you'll get the key events you're looking for.

import AppKit

class CustomMenuView: NSView {

    override var acceptsFirstResponder: Bool {
        return true
    }

    override func draw(_ dirtyRect: NSRect) {
        super.draw(dirtyRect)

        // Drawing code here.
    }

    override func keyDown(with event: NSEvent) {
        print("key down with character: \(String(describing: event.characters)) " )
    }
}


func installCustomMenuItem() -> NSMenu {
//  let menuBarItem = NSMenuItem(title: "Test", action: nil, keyEquivalent: "")
    let resultMenu = NSMenu(title: "TestMenu" )
    let subMenuBarItem = NSMenuItem(title: "Custom View", action: nil, keyEquivalent: "")

    subMenuBarItem.view = CustomMenuView(frame: NSRect(x: 0, y: 0, width: 40, height: 44))
    resultMenu.addItem(subMenuBarItem)
//  menuBarItem.submenu = menu

    return resultMenu
}

var menu = installCustomMenuItem()

menu.popUp(positioning: nil, at: NSPoint(x:600,y:400), in: nil)

I didn't manage to figure out why this issue was happening or how to fix it, but I understood that it's possible to work around this issue by intercepting all the keys and simulating their behavior manually.

For example, this is how I now handle down arrow key which is supposed to select next item in menu list:

class Menu: NSMenu {
  func selectNext() {
    var indexToHighlight = 1
    if let item = highlightedItem {
      indexToHighlight = index(of: item) + 1
    }

    if let itemToHighlight = self.item(at: indexToHighlight) {
      let highlightItemSelector = NSSelectorFromString("highlightItem:")
      perform(highlightItemSelector, with: itemToHighlight)

      if itemToHighlight.isSeparatorItem || !itemToHighlight.isEnabled || itemToHighlight.isHidden {
        selectNext()
      }
    }
  }
}

This way, when I receive a key down event with down arrow key - I can just call the function and return true to prevent the event from reaching default NSMenu handler. Similarly, up arrow key can be done.

In case of a return key, I ended up with the following code:

class Menu: NSMenu {
  func select() {
    if let item = highlightedItem {
      performActionForItem(at: index(of: item))
      cancelTracking()
    }
  }
}

The full commit implementing this is https://github.com/p0deje/Maccy/commit/158610d1d.