I'm pretty new to Windows programming.
I've been assigned the task of hiding any specific window(s), given a pid, from being recorded using SetWindowDisplayAffinity(). With the help of DLL injection, it works as expected (as far as I've tested), but when I used it on Visual Studio itself, it did work, but that glowing frame around it persisted.
I Google'd and someone pointed out that it's using the extended window style WS_EX_LAYERED, and that I can look at its properties using Spy++. Here's what it looks like:
I was afraid the function that calls SetWindowDisplayAffinity() from my injected DLL did it wrong somehow (the starter code given to me had a do..while loop calling FindWindowEx() which I felt, for some reason, didn't come by the window handle of the glowing frame). So, I wrote my own code to retrieve all current thread IDs, and call EnumThreadWindows() for each one of them, which consequently called my callback function, which then called SetWindowDisplayAffinity().
The code worked as before, but didn't fix the issue. So, I've set an array which I used to store all window handles that were passed to SetWindowDisplayAffinity(), and with the help of Spy++ and MessageBox() (my debugging techniques are archaic, I know), I made sure the main window handle AND the glowing frame handle both were indeed in there.
Here's an image for reference:
I doubted that WS_EX_LAYERED prevented SetWindowDisplayAffinity() from doing its job, but I'm still clueless.
Edit:
I've logged the last error in case SetWindowDisplayAffinity() did fail, for some reason. It turns out it did, and the error was not enough memory resources are available to process this command with error code 0x8, and as I expected the error only showed on the 4 drop shadow border window handles (logged the handles and compared them to Spy++'s info).
Edit:
I've found this post, I guess it sums everything up? Is there any hope left?
Edit:
I've currently setteled for this non-practical workaround:
SetWindowLongPtr(hwnd, GWL_STYLE, GetWindowLongPtr(hwnd, GWL_STYLE) & ~WS_VISIBLE);