I am trying to build something like this: Overlaying a 3D model (say, a virtual helicopter) in HoloLens 2 (HL2) per a physically moving point, whose spatial coordinate is known. So when the physical point moves, the HL2 users sees the virtual helicopter moves according to the physical point.
The physical point can be the center of a flying drone (e.g., CrazyFlie), or the tip of a robotic arm, etc. The point’s spatial coordinates are established and captured by some kind of motion capture system, so, in a fixed coordinate.
To realize the above scenario, the tricky thing is that HL2 is a moving origin -- HL2’s (0, 0, 0) is always moving when the user’s head moves -- so it makes the coordinate transformation more challenging than I originally thought.
Anyway, my plan is to:
(1) first, do some kind of “handshaking”, e.g., put a static QR code sticker on a known fixed position (known by the MoCap), let HL2 detects it. The very moment HL2 sees the sticker, it knows the relationship between HL2 coordinate and the MoCap coordinate (translational and angular differences);
(2) Then, as the HL2 moves, it’s in a way is like HL2 is fixed, and the MoCap coordinate is moving (where in reality it’s the opposite). So the MoCap’s origin can be calculated for every instant;
(3) And since the target moving point (e.g., the drone)’s MoCap coordinate is known, it can be transformed to HL2, allowing HL2 to render a helicopter according to the drone’s spatial movement.
This is just a conceptual plan. Does this plan sound good? If so, could you share how to do the handshaking and coordinate transformation? Thank you!
ps.1. I’ve tried using Azure Spatial Anchor for the said scenario, but so far I would say I don’t have high hopes for going forward with it more.
ps.2. The whole scenario need not rotation (e.g., the drone’s rotation doesn’t need to be considered). Hopefully it will make things easier.
ps.3. I am doing this with Unity and MRTK.
Thank you!