Why useState seems to skip steps

TBD. (I'll update this answer when I know for sure)

The error from useState

That error fires if a setState is called after a component has unmounted. In your code on Github, you trigger the sorting, which is an async process, but you don't have anything in place to cancel that process if you unmount the component midway through the process. This is why that error happens. The component is no longer rendered and you're still trying to update its state.

Using Generator Functions instead

A much better way to do what you're doing would be to use Generator functions instead of Async function. Generator functions are almost exactly the same as async function, except, you use the work yield instead of await. That way, the function goes and "pauses" on every yield and you're able to resume it manually externally.

This will make you're visualizer better too as you won't need a static DELAY and you'll be able to let the user click through one step at a time.

Best of all, your code will look mostly the same. And if you still want to automatically advance through the visualization, you'll be able to put a setInterval in the React component itself which will be easier to clean up on unmount.

Here's a basic example of how Generator functions would work:

function* mergeSort(arr, setArr, mainArray) {
  let n = arr.length;
  if (n === 1) {
    return arr;
  }

  let leftArray = arr.slice(0, Math.floor(n / 2));
  let rightArray = arr.slice(Math.floor(n / 2), n);

  leftArray = yield* mergeSort(leftArray, setArr, mainArray);
  rightArray = yield* mergeSort(rightArray, setArr, mainArray);

  let mergeResultArray = yield* merge(leftArray, rightArray);
  let l = mergeResultArray.length;
  let tempArr = [...mergeResultArray];
  let copyArr = [...mainArray];
  let subArr = copyArr.slice(l, mainArray.length);
  tempArr.push(...subArr);
  // Cheaper way to copy.
  setArr(tempArr.slice());

  return mergeResultArray;
}
function* merge(array1, array2) {
  let array3 = [];

  while (array1.length && array2.length) {
    yield;
    if (array1[0].val > array2[0].val) {
      array3.push(array2[0]);
      array2.splice(0, 1);
    } else {
      array3.push(array1[0]);
      array1.splice(0, 1);
    }
  }

  while (array1.length) {
    yield;
    array3.push(array1[0]);
    array1.splice(0, 1);
  }

  while (array2.length) {
    yield;
    array3.push(array2[0]);
    array2.splice(0, 1);
  }
  return array3;
}

// Usage within React:

  let [array, setArray] = useState([]);

  let iterRef = useRef();

  const generateNewArray = () => {
    setArray(generateRandomArr(5, 650, ARRAY_SIZE));
  };

  const mergeSortStep = () => {
    // This will do one step on every click.
    iterRef.current.next();
  };

  useEffect(() => {
    setArray(generateRandomArr(5, 650, ARRAY_SIZE));
    iterRef.current = mergeSort([...array], setArray, [...array]);
  }, []);

Edited on Nov 12.

I created a sandbox. A lot of changes, even typescript added (I don't like programming without the support of types.). Hopefully, it's helpful.

In my understanding, the best approach to make complicated calculations in React Components(also hooks) is never do that in React Components. There are strict rules to write codes in React, such as ticker of re-rendering, rules of hooks, etc. Mixing complicated calculations with React leads to unexpected behaviors and is hard to debug. Instead, move it out of React, subscribe to the results after the component is mounted, and unsubscribe before unmounting. That's all my understanding about avoiding errors in React that letting React only do the views.

@naman-goel has already provided you with a great answer but I wanted to add that you don't have to necessarily do the visualization while doing the sorting, you can also do the sort and store the steps, then later on do the visualization separately. That way you can also store the state for the visualization without doing the sort again. If you really wanted to go the extra mile you can also then use canvas to draw the visualization instead of rerendering a component on every step so it doesn't look like constant flickering.

React will also batch update every 10ms or something, so if your state update is happening 10 times in that 10ms period it also won't show all your visualization, it will just jump to the next ~10 steps.