How to move objects in Unity VR Toolkit (Raycast) corretly?

i'm really stuck with some kind of a more basic problem. i want to recreate a feature in oculus home for my own apartment project, where i can place and especially move / turn furniture with my vr controller (raycast around).

at this moment, i can move the furniture while snapping the transform.position of the object to the raycastHit.position. while implementing an offset, i can hit the object where ever i want and move it. The problem:

1. The object ignores other surfaces like walls and floors while moving. (the rigedbody ignores colliders).
2. I can't increase the distance of the raycast, because it is snaped to the objects position. therefore i can only decrease the distance while movinge the pointer towards me, but not away from me.

This video shows the behaviour. Youtube Video Capture from scene

This is the code so far. On the right RayController a script to pass the not needed currently needed position and the hitPoint.position to the selected object

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.Interaction.Toolkit;

public class ControllerPosition : MonoBehaviour
{
    public InputActionProperty positionProperty;
    [SerializeField] XRRayInteractor rayController;

    public Vector3 Position { get; private set; } = Vector3.zero;
    public Vector3 rayHitPosition { get; private set; } = Vector3.zero;


    private void Awake()
    {
        rayController = GetComponent<XRRayInteractor>();
    }

    private void Update()
    {
        if (rayController.TryGetHitInfo(out var hitPosition, out var hitNormal, out _, out _))
        {
            rayHitPosition = hitPosition;
        }

        Position = positionProperty.action.ReadValue<Vector3>();
    }
}

This is the code for the dragable object

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.Interaction.Toolkit;

public class DragObject : XRGrabInteractable
{
    [SerializeField] XRRayInteractor ray;

    private Vector3 mOffset;

    private ControllerPosition controllerPosition = null;

    protected override void Awake()
    {
        base.Awake();
    }

    protected override void OnSelectEntered(SelectEnterEventArgs args)
    {
        base.OnSelectEntered(args);
        controllerPosition = args.interactor.GetComponent<ControllerPosition>();

        mOffset = gameObject.transform.position - controllerPosition.rayHitPosition;
    }

    protected override void OnSelectExited(SelectExitEventArgs args)
    {
        base.OnSelectExited(args);
        controllerPosition = null;
    }

    public override void ProcessInteractable(XRInteractionUpdateOrder.UpdatePhase updatePhase)
    {
        base.ProcessInteractable(updatePhase);

        if (isSelected)
        {
            if (updatePhase == XRInteractionUpdateOrder.UpdatePhase.Dynamic)
            {
                UpdateObjectPosition();
            }
        }
    }

    private void UpdateObjectPosition()
    {
        Vector3 position = controllerPosition ? controllerPosition.Position : Vector3.zero;
        
        transform.position = controllerPosition.rayHitPosition + mOffset;

    }
}

Any advices?

best regards