TL;DR

Since everything is a square, we can use fake rows and columns to size the cells properly. Solution at the end of this post.

Explanation

Problem definition

If I understand your problem correctly, you want

• a W×H cells grid
• that fits inside a N×N pixels square
• where each cell is a square

Starting point, what already works

So in the case that you got working, say for example, W=3; H=2, we get a result that looks like this:

Refactoring for pure CSS

I assume that we're looking for a pure-CSS solution, since a JS solution would be both trivial and suboptimal. So before we proceed, let's modify the JS to just give us CSS custom properties and then we can reason in CSS only:

• we're defining --per-row, the number of cells in 1 row,
• we're defining --per-col, the number of cells in 1 column.

grid.style.setProperty('--per-row', width);
grid.style.setProperty('--per-col', height);

#grid {
  --per-row: 2;
  --per-col: 2;
  grid-template-columns: repeat(var(--per-row), 1fr);
  grid-template-rows: repeat(var(--per-col), 1fr);
}

PS: in every code snippet, I'll omit the lines that aren't relevant to the current discussion.

Understanding the issue

Now what doesn't work in your solution is when H > W, so let's take an example where W=2; H=3. We want our grid to look like this:

But since the parent (cyan blue box) is also a square, we can see that this example and the previous one have the same resulting cell dimensions. In the case of W=3; H=2 we kind of see an empty line at the bottom; and in the case of W=2; H=3 we kind of see an empty column on the right. In both cases, we lay out our cells in a 3×3 grid! This is simply max(width, height):

So let's create this grid with CSS max():

#grid {
  --per-row: 2;
  --per-col: 2;
  --max: max(var(--per-row), var(--per-col));
  grid-template-columns: repeat(var(--max), 1fr);
  grid-template-rows: repeat(var(--max), 1fr);
}

However, with this grid template, we'll always create 3 columns in both the 3×2 case and the 2×3 case. It's going to work great for W > H but the H > W will look wrong. We need to refactor how the columns template is calculated.

1. First, the number of column we actually want. That's the easy part, it's --per-row:

   grid-template-columns: repeat(var(--per-row), 1fr);
   

   But now our 2×3 grid looks like this because each column is 1fr, which results in 50% of the full width:

   

2. Now, we need to find the proper width of a column. We've seen that using 1fr doesn't work, but since everything is a square, we know that in both the 3×2 case and the 2×3 case we need each column to be ⅓ of the width, or 100% / 3, or calc(100% / var(--max)):

   grid-template-columns: repeat(var(--per-row), calc(100% / var(--max)));
   

tadaa!

We now have a layout that will work for any W and H with two simple lines of CSS:

#grid {
  grid-template-columns: repeat(var(--per-row), calc(100% / var(--max)));
  grid-template-rows: repeat(var(--max), 1fr);
}

Cleaning up

1. Because of how "vertical" layout works (technically, "block direction" in CSS flow layout), we can simplify how rows are created. grid-template-rows will create a fixed number of rows, but now that we have a strong definition for the number of columns in grid-template-columns, we can just say "create as many rows as needed to fit all of the children", which is exactly what grid-auto-rows does:

   grid-auto-rows: calc(100% / var(--max));
   

   This uses the same math as for the width of the columns, since everything here is a square.

2. Because support for CSS max() still isn't great, this is something we can do in JS without any performance issue:

   grid.style.setProperty('--per-row', width);
   grid.style.setProperty('--per-col', height);
   grid.style.setProperty('--max', Math.max(width, height));
   

3. You'll notice that at this point, we aren't using --per-col anymore thanks to grid-auto-rows, so we can remove it:

   grid.style.setProperty('--per-row', width);
   grid.style.setProperty('--max', Math.max(width, height));
   

   #grid {
     --per-row: 2;
     --max: 2;
   }
   

Solution

Here's your initial code, with the modifications explained in my answer.

rerender = (event) => {
  const height = document.getElementById("y-input").value;
  const width = document.getElementById("x-input").value;
  const grid = document.getElementById("grid");

  grid.style.setProperty('--per-row', width);
  grid.style.setProperty('--max', Math.max(width, height));

  grid.innerHTML = "";
  [...Array(height * width).keys()]
    .forEach(() => {
      const e = document.createElement('div')
      e.className = "cell";
      grid.appendChild(e)
    })
}

#container {
  width: 500px;
  height: 500px;
  background-color: aqua;
  padding: 8px;
}

#grid {
  display: grid;
  height: 100%;
  --per-row: 2;
  --max: 2;
  grid-template-columns: repeat(var(--per-row), calc(100% / var(--max)));
  grid-auto-rows: calc(100% / var(--max));
  gap: 2px;
}

.cell {
  background-color: blue;
  aspect-ratio: 1/1;
}

<div>
  <label for="x-input">width</label>
  <input value=2 min=1 max=50 type="number" name="x" id="x-input" style="width: 4ch;" onchange="rerender(event)">
  <label for="y-input">height</label>
  <input value=2 min=1 max=50 type="number" name="y" id="y-input" style="width: 4ch;" onchange="rerender(event)">
</div>

<div id="container">
  <div id="grid">
    <div class="cell"></div>
    <div class="cell"></div>
    <div class="cell"></div>
    <div class="cell"></div>
  </div>
</div>