React useEffect in functional component is being called after hot reload even when dependencies dont change

I'm trying to use vanilla ThreeJS instead of react-three-fiber.

I have a functional component that displays the 3D scene, and I want to call a function that creates my scene only once.

This is the component

import * as Three from 'three';
import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls';
import { FC, MutableRefObject, useCallback, useEffect, useRef } from 'react';
import {  Camera, PerspectiveCamera, Scene, WebGLRenderer } from 'three';
import {  GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader';

export const Background : FC = function(props) {
    const renderer: WebGLRenderer = useRef<WebGLRenderer>(new WebGLRenderer()).current;
    
    const scene: Scene = useRef<Scene>(new Scene()).current;
    
    const camera: Camera = useRef<Camera>(new PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000)).current;
    const orbitControlsRef: MutableRefObject<OrbitControls | undefined> = useRef<OrbitControls | undefined>(undefined);


    const onContainerReferenced = useCallback(
        (container: HTMLDivElement) => {
            console.log(`CustomLog: Container element referenced`);
            if(container == null) return;

            if(orbitControlsRef.current === undefined) orbitControlsRef.current = new OrbitControls(camera, container);
            const orbitControls: OrbitControls = orbitControlsRef.current;

            (camera as any).aspect = container.clientWidth / container.clientHeight;

            renderer.setSize(container.clientWidth, container.clientHeight);
            container.appendChild(renderer.domElement);
            
            orbitControls.domElement = container;
            orbitControls.update();

            function animate() {
                requestAnimationFrame(animate);
                renderer.render(scene, camera);
            }
            animate();
        },
        []
    );

    useEffect(
        () => {
            console.log("CustomLog: Setting up scene");
            camera.position.z = 5.0;
            
            const gltfLoader = new GLTFLoader();
            gltfLoader.load(
                "/Shoe.glb", 
                (shoeGLB) => {
                    scene.add(shoeGLB.scene);

                    // for(let i: number = 0; i < shoeGLB.scene.children.length; i++) {
                    //     (shoeGLB.scene.children[i] as Mesh).position.set(camera.position.x + 5.0, camera.position.y + 5.0 , camera.position.z + 5.0);
                    // }
                }
            );
        },
        [scene, camera]
    );    


    return (
        <div ref={onContainerReferenced} style={{width: "100%", height: "100%"}} />
    );
};

But even though the dependency array does not change the component calls the function after every hot reload.

This is undesirable because the objects in the scene are created every time I hot reload, leading to duplicates.

The dependency array values was verified by breakpoints and the uuid of both the objects were the same as previous renders, so they dont change.

It does not look like a remount either, otherwise I would have lost all the references to variables like scene, camera, etc.

I even tried replacing the dependency array with constants, like this:

useEffect(
  () => {
    // Setup scene
  }, 
  [1, 2]
)

since they are constant it means dependency array has not changed right? But it still calls the function in useEffect after hot reload.

If anybody can fi