UWP apps can register for a custom scheme by using Extension Category="windows.protocol" (MSDN). I'm aware of custom protocol handlers by registering a URI scheme (MSDN) but those always launch a new instance of the app. What I want is the behavior of a UWP app: if there is a currently running instance, it gets activated.
I also looked into URL Monikers and Asynchronous Pluggable Protocols (MSDN) but those only work when invoking the protocol in-process, such as when using a Web view. A UWP app gets activated even if some other process, such as a Web browser, invokes the protocol.
This is for a C++ Win32 desktop application; .NET is not an option.
Also, as I said in my comment to Anders's proposal, I don't want to have to create a separate component to activate my running app when some other process, such as a browser, invokes my protocol. UWP apps have no such requirement and I want the same for my app.