Applying Rotation Vector to OpenGL camera or "view matrix"

I'm building an augmented reality application where I want objects on the screen to move with respect to the orientation of the phone. For example, if the phone is held still - the object stays at the center of the screen. If the phone is rotated left about its vertical axis in portrait mode, the object moves to the right. This is the sensor I'm accessing in Android

@Override
public void onSensorChanged(SensorEvent event) {
    if (event.sensor.getType() == Sensor.TYPE_ROTATION_VECTOR) {
        SensorManager.getRotationMatrixFromVector(mRotationMatrix, event.values);
        mGLAssetSurfaceView.getRenderer().setRotationMatrix(mRotationMatrix);
    }
}

This is how I'm applying it to my OpenGL renderer:

@Override
public void onDrawFrame(GL10 gl) {
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

    Matrix.setLookAtM(mViewMatrix, 0, 0, 0, -3, 0f, 0f, 0f, 0f, 1.0f, 0.0f);

    Matrix.multiplyMM(mViewMatrix, 0, mViewMatrix, 0, mRotationMatrix, 0);

    Matrix.multiplyMM(mMVPMatrix, 0, mProjectionMatrix, 0, mViewMatrix, 0);

    // Draw shape
    mSquare.draw(mMVPMatrix);
}

As you can see, I'm computing the product

MVRP

and passing that into my vertex shader. The result of this is the square in the middle of the screen rotating about its own axis, not with respect to the phone.