I'm creating an interaction system for a VR game and I'm struggling with two hand interactions. I'm using the Quaternion.LookRotation() method to generate the rotation of the grabbed object based on the positions and rotations of the hands. The forward part is pretty simple:
Vector3 fwd = (primaryHand.position - secondaryHand.position).normalized;
The "up" part is what I have difficulty with. Initially I tried using the average up direction of both hands:
Vector3 avgUp = Quaternion.Slerp(primaryHand.rotation, secondaryHand.rotation, 0.5f) * Vector3.up;
There is an issue with this approach: the hand's up vector might get aligned with the fwd vector, which causes the object to flip over when it goes over it. Here is a simple illustration of the problem:
The light green arrows represent the up direction of the hands, while the dark green is the calculated direction used as an argument for the LookRotation() method.
The obvious solution seems to be to pick a different fixed vector instead of "up", one which won't be so easily aligned with the fwd vector. In the example it could be a vector aligned with the fingers. But keep in mind that there are no restrictions on initial hand rotation so no matter which vector you choose the hands can always happen to be rotated so that the vectors align. And even if you pick the an optimal vector dynamically (one that is perpendicular to fwd), it's still at best 90 degrees from aligning with fwd.
To solve this I tried restricting the direction to the values which don't cause problems but this caused another issues (I had difficulties with determining which values are ok and which should be discarded). I feel like I'm doing something wrong here, is there any better solution to this problem?
