ARCore object tracking without a plane

Viewed 3823

I am very new to this ARCore and I have been looking at the HelloAR Java Android Studio project provided in the SDK.

Everthing works OK and is pretty cool, however, I want to place/drop an object when I touch the screen even when no planes have been detected. Let me explain a little better...

As I understand ARCore, it will detect horizontal planes and ONLY on those horizontal planes I can place 3D objects to be motion tracked.

Is there any way (perhaps using PointCloud information) to be able to place an object in the scene even if there are no horizontal planes detected? Sort of like these examples? https://experiments.withgoogle.com/ar/flight-paths https://experiments.withgoogle.com/ar/arcore-drawing

I know they are using Unity and openFrameworks, but could that be done in Java?

Also, I have looked at How to put an object in the air? and how to check ray intersection with object in ARCore

but I don't think I'm understanding the concept of Ancor (I managed to drop the object on the scene, but it either disappears immediately or it is just a regular OpenGL object with no knowledge about the real world.

What I want to understand is: - How and is it possible to create a custom/user defined plane, that is, a plane that is NOT automatically detected by ARCore? - How can I create an Ancor (the sample does it in the PlaneAttachment class, I think) that is NOT linked to any plane OR that is linked to some PointCloud point? - How do I draw the object and place it at the Ancor previously created?

I think this is too much to ask but looking at the API documentation has not helped me at all

Thank you!

Edit:

Here is the code that I added to HelloArActivity.java (Everything is the same as the original file except for the lines after // ***** and before ...

@Override
public void onDrawFrame(GL10 gl) {
...
        MotionEvent tap = mQueuedSingleTaps.poll();

        // I added this to use screenPointToWorldRay function in the second link I posted... I am probably using this wrong
        float[] worldXY = new float[6];
...
        if (tap != null && frame.getTrackingState() == TrackingState.TRACKING) {

            // ***** I added this to use screenPointToWorldRay function
            worldXY = screenPointToWorldRay(tap.getX(), tap.getY(), frame);
...

        }
...

        // Visualize anchors created by touch.
        float scaleFactor = 1.0f;
        for (PlaneAttachment planeAttachment : mTouches) {
...
        }

        // ***** This places the object momentarily in the scene (it disappears immediately)  
        frame.getPose().compose(Pose.makeTranslation(worldXY[3], worldXY[4], worldXY[5])).toMatrix(mAnchorMatrix, 0);

        // ***** This places the object in the middle of the scene but since it is not attached to anything, there is no tracking, it is always in the middle of the screen (pretty much expected behaviour)
        // frame.getPose().compose(Pose.makeTranslation(0, 0, -1.0f)).toMatrix(mAnchorMatrix, 0);

        // *****I duplicated this code which gets executed ONLY when touching a detected plane/surface.
        mVirtualObject.updateModelMatrix(mAnchorMatrix, scaleFactor);
        mVirtualObjectShadow.updateModelMatrix(mAnchorMatrix, scaleFactor);
        mVirtualObject.draw(viewmtx, projmtx, lightIntensity);
        mVirtualObjectShadow.draw(viewmtx, projmtx, lightIntensity);
...
}
1 Answers

You would first have to perform a hit test via Frame.hitTest and iterate over the HitResult objects until you hit a Point type Trackable. You could then retrieve a pose for that hit result via HitResult.getHitPose, or attach an anchor to that point and get the pose from that via ArAnchor.getPose (best approach).

However, if you want to do this yourself from an arbitraty point retrieved with ArPointCloud.getPoints, it will take a little more work. In this approach, the question effectively reduces down to "How can I derive a pose / coordinate basis from a point?".

When working from a plane it is relatively easy to derive a pose as you can use the plane normal as the up (y) vector for your model and can pick x and y vectors to configure where you want the model to "face" about that plane. (Where each vector is perpendicular to the other vectors)

enter image description here

When trying to derive a basis from a point, you have to pick all three vectors (x, y and z) relative to the origin point you have. You can derive the up vector by transforming the vector (0,1,0) through the camera view matrix (assuming you want the top of the model to face the top of your screen) using ArCamera.getViewMatrix. Then you can pick the x and z vectors as any two mutually perpendicular vectors that orient the model in your desired direction.

Related