State is unchanged from within a function returned by a hook

Answer1: The problem is every time the requestAnimationFrame run and square.update() you are actual create new box. You need to clear your frame before drawing new one

import { useContext, useEffect } from "react";
import { Renderer } from "./ctx";

export function Square({ x, y }) {
  let getContext = useContext(Renderer);
  useEffect(() => {
    let ctx = getContext();
    // clear before drawing
    ctx.clearRect(0, 0, window.innerWidth, window.innerHeight);
    ctx.fillStyle = "#000";
    ctx.fillRect(x, y, 10, 10);
  }, [x, y]);
  return null;
}

Check the codesandbox for visualisation

This is a very good series I found online. Check it out

Answer2: requestAnimationFrame create a loop to call square.update(), that's why's you can see the dimensions changed

Answer3: You are updating state inside Square component by calling set**

The problem is that the update function returned from useSquare is a closure and the value of x and y inside it never change, thus the condition of x <= 0 || x >= WIDTH and y <= 0 || y >= HEIGHT are never true. Here is an example of closure:

let x = 1;

function update(y) {
  return function() {
    console.log(y); // always 1
  };
}
setInterval(update(x), 1000);
x = 2;

Closure related bugs are hard to be identified. While you can learn more about closures in various resources including MDN, you can use the exhaustive-deps linting rule in eslint-plugin-react-hooks package to make sure your dependencies array specified in hooks are specified correctly and prevent potential bugs. React docs (Such rule is enabled by default in CodeSandbox.)

But before that, let's simplify the task to find a better way to tackle the problem. Let's say there is a state x that will keep incrementing by 3 until it hits 20. Then it will keep decrementing by the same value until it hits 0.

In this case, we use +1/-1 to represent the direction of increment/decrement. Since this direction changes shouldn't trigger a re-render, we don't have to store it as a state variable. Instead, we store it as ref since we want to access its latest value when computing the new x. To detect if x hits the limits, we set up a useEffect hook. The full implementation would be:

import React, { useState, useRef, useEffect } from "react";
const MAX = 20;
const RATE = 3;

function App() {
  let [x, setX] = useState(0);
  const direction = useRef(1);
  useEffect(() => {
    const id = setInterval(() => {
      setX(val => val + direction.current * RATE);
    }, 500);
    return () => clearInterval(id);
  }, []);
  useEffect(() => {
    if (x >= MAX) direction.current = -1;
    if (x <= 0) direction.current = 1;
  }, [x]);
  return <div>{x}</div>;
}

Back to your case, there are few places that should be updated and the main part is inside useSquare. Like the above example, we no longer store vx and vy as state. Instead, initialize them with useRef. Two useEffect hooks are also set up to change their values according to x and y values.

// useSquare.js    
import { useState, useRef, useEffect, useCallback } from "react";
import { WIDHT, HEIGHT } from "./settings";

export function useSquare(init_x, init_y) {
  let [x, set_x] = useState(init_x);
  let [y, set_y] = useState(init_y);
  let vx = useRef(10);
  let vy = useRef(-5);

  let update = useCallback(() => {
    set_x(prev_x => prev_x + vx.current);
    set_y(prev_y => prev_y + vy.current);
  }, []);

  useEffect(() => {
    if (x >= WIDHT) vx.current = -10;
    if (x <= 0) vx.current = 10;
  }, [x]);

  useEffect(() => {
    if (y <= 0) vy.current = 5;
    if (y >= HEIGHT) vy.current = -5;
  }, [y]);

  useEffect(() => {
    let id;
    function cb() {
      update();
      id = requestAnimationFrame(cb);
    }
    cb();
    return () => cancelAnimationFrame(id);
  }, [update]);

  return { x, y };
}

And the app.js becomes:

function App() {
  let canvas = useRef();
  let square = useSquare(WIDHT / 2, HEIGHT / 2);
  return (
    <>
      <canvas ref={canvas} width={WIDHT} height={HEIGHT} />
      <Renderer.Provider value={() => canvas.current.getContext("2d")}>
        <Square {...square} />
      </Renderer.Provider>
    </>
  );
}

Notice I put wrap the update function with an useCallback hook so that it will not change with x and y values. I also put the animation function inside useSquare hook. If you move it back to App.js, make sure you set the dependencies array right.

Here is the updated sandbox: