How to move an object in Unity

Hello guys I try to make a city building game the idea is very simple actually Instantiate an Building and move them with mouse after that hit the place button and place. The issue is if the collider of a another building completely encloses the collider of the building that I am built I can't move new building. I can probably explain better with pictures. Issue 1 The first picture my new building you can see collider limits and the second one my old building that I already placed. I Understand the problem but I can not solve it. And here it is my object drag code

private void OnMouseDrag()
    {
        Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
        RaycastHit hitInfo;
        if (Physics.Raycast(ray, out hitInfo, 1000f,(1 << LayerMask.NameToLayer("Ground"))))
        {
            Debug.Log(hitInfo.transform.name);
            transform.position = SnapToGrid(hitInfo.point);
        }
    }

    private Vector3Int SnapToGrid(Vector3 pos)
    {
        Vector3 tempPos = pos;
        Vector3Int snappedPos;
        snappedPos = new Vector3Int(Mathf.RoundToInt(tempPos.x), Mathf.RoundToInt(tempPos.y), Mathf.RoundToInt(tempPos.z));
        return snappedPos;
    }

Thanks for the help in advance