React MUI, How to append to a Masonry Image List

I'm trying to use the React MUI Masonry Image List component with the infintie scroll component. They seem to work pretty great together.

However, the issue I'm having is appending to the Masonry image list. I successfully append new images, but the whole masonry moves jump and becomes jittery when new images are loaded.

I wonder if it's possible just to add the images to the bottom of the page without the whole thing spazzing out.

Here is a code the illustrates this, basically I just modified the Masonry image list demo:

import * as React from 'react';
import Box from '@mui/material/Box';
import ImageList from '@mui/material/ImageList';
import ImageListItem from '@mui/material/ImageListItem';
import InfiniteScroll from "react-infinite-scroll-component";
import { useEffect, useState } from "react";


export default function MasonryImageList() {

  const sleep = ms => new Promise(resolve => setTimeout(resolve, ms))
  const [items, setItems] = useState([]);
  const fetchMore = async (itemData) =>{
      await sleep(2000)
      setItems(prev_items => [...prev_items,...itemData])
  }

  useEffect(() => {
    fetchMore(itemData);
  }, []);


  return (
    <Box id="scrolableDiv" sx={{ width: 800, height: 800, overflowY: 'scroll' }}>
      <ImageList variant="masonry" cols={3} gap={8}>
      <InfiniteScroll
          dataLength={items.length}
          next={fetchMore(itemData)}
          hasMore={true}
          loader={<h4>Loading...</h4>}
          scrollableTarget= "scrolableDiv"
        >
        {items.map((item) => (
          <ImageListItem key={item.img}>
            <img
              src={`${item.img}?w=248&fit=crop&auto=format`}
              srcSet={`${item.img}?w=248&fit=crop&auto=format&dpr=2 2x`}
              alt={item.title}
              loading="lazy"
            />
          </ImageListItem>
          
        ))}
        </InfiniteScroll>
      </ImageList>
    </Box>
  );
}

const itemData = [
  {
    img: 'https://images.unsplash.com/photo-1549388604-817d15aa0110',
    title: 'Bed',
  },
  {
    img: 'https://images.unsplash.com/photo-1525097487452-6278ff080c31',
    title: 'Books',
  },
  {
    img: 'https://images.unsplash.com/photo-1523413651479-597eb2da0ad6',
    title: 'Sink',
  },
  {
    img: 'https://images.unsplash.com/photo-1563298723-dcfebaa392e3',
    title: 'Kitchen',
  },
  {
    img: 'https://images.unsplash.com/photo-1588436706487-9d55d73a39e3',
    title: 'Blinds',
  },
  {
    img: 'https://images.unsplash.com/photo-1574180045827-681f8a1a9622',
    title: 'Chairs',
  },
  {
    img: 'https://images.unsplash.com/photo-1530731141654-5993c3016c77',
    title: 'Laptop',
  },
  {
    img: 'https://images.unsplash.com/photo-1481277542470-605612bd2d61',
    title: 'Doors',
  },
  {
    img: 'https://images.unsplash.com/photo-1517487881594-2787fef5ebf7',
    title: 'Coffee',
  },
  {
    img: 'https://images.unsplash.com/photo-1516455207990-7a41ce80f7ee',
    title: 'Storage',
  },
  {
    img: 'https://images.unsplash.com/photo-1597262975002-c5c3b14bbd62',
    title: 'Candle',
  },
  {
    img: 'https://images.unsplash.com/photo-1519710164239-da123dc03ef4',
    title: 'Coffee table',
  },
];

If you see any glaring errors, please point them out, I've just started with React.