Does the VMX mode have the capability to detect previously non-trappable sensitive instructions?

No, not in all cases. Instead VMX mode changes the behavior of some of these instructions so that they are not sensitive.

Here are a couple examples of how sensitive instructions behave in VMX non-root mode:

• SMSW: it doesn't trap when in VMX mode, but it substitutes values specified by the VMM for the actual values of certain bits.
• POPF: the behavior of the instruction doesn't change, but control of interrupts can be overridden by controls in the VMCS, to prevent guest changes to IF from affecting the VMM or other guests.

This reference has a list of sensitive x86 instructions on page 28: https://www.cs.cmu.edu/~410-s14/lectures/L30_Virtualization.pdf. Volume 3, chapter 25 of the Intel SDM has a detailed description of the change in behavior of every instruction.

I'd say yes. According to pages 13 through 17 of this Virtualization and Virtual Machines presentation, hardware virtualization seems to be an attempt to solve that problem.

The pdf @prl posted also says on slide 45:

if a guest kernel runs a sensitive instruction, hardware does a 'VM exit' back to the VMM, indicates which instruction trapped