Webpack5 - Module parse failed: Unexpected character '�' (1:0) You may need an appropriate loader

Trying to import local images into my React project. I'm using statements like these, nothing fancy.

<Image source={require('./redvelvet.png')}></Image>

And here's the error it produces.

ERROR in ./src/components/redvelvet.png 1:0
Module parse failed: Unexpected character '�' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)
 @ ./src/components/MachineTemplate.js 157:16-42
 @ ./src/components/App.js 6:0-48 21:28-43
 @ ./src/index.js 1:0-35

I'm pretty sure I've narrowed it down to my webpack.config file, but for the life of me I can't figure out what I've done wrong.

const path = require("path");
const webpack = require("webpack");

module.exports = {
  entry: "./src/index.js",
  output: {
    path: path.resolve(__dirname, "./static/frontend"),
    filename: "[name].js",
  },
  module: {
    rules: [
      {
        test: /\.js$|jsx/,
        exclude: /node_modules/,
        use: {
          loader: "babel-loader",
        },
      },
    ],
  },
  optimization: {
    minimize: true,
  },
  plugins: [
    new webpack.DefinePlugin({
      "process.env": {
        // This has effect on the react lib size
        NODE_ENV: JSON.stringify("production"),
      },
    }),
  ],
};

Any help is appreciated. It feels like it's something simple I'm missing here.