Hydration failed because the initial UI does not match what was rendered on the server. React 18, next link, error in my button component?

here's my custom button component:

const variants = {
  primary: "btn-primary",
  secondary: "btn-secondary",
  delete: "btn-red",
  white: "btn-white",
  none: "",
};

type IconProps =
  | { startIcon: React.ReactElement; endIcon?: never }
  | { endIcon: React.ReactElement; startIcon?: never }
  | { endIcon?: undefined; startIcon?: undefined };

export type ButtonProps = React.ButtonHTMLAttributes<
  HTMLButtonElement | HTMLAnchorElement
> & {
  variant?: keyof typeof variants;
  isLoading?: boolean;
  asAnchor?: boolean;
} & IconProps;

export const Button = React.forwardRef<
  HTMLButtonElement | HTMLAnchorElement,
  ButtonProps
>(
  (
    {
      type = "button",
      className = "",
      variant = "primary",
      isLoading = false,
      startIcon,
      endIcon,
      asAnchor,
      ...props
    },
    ref
  ) => {
    return (
      <>
        {asAnchor ? (
          <a
            className={clsx(
              className,
              variants[variant],
              "appearance-none",
              props.disabled && "pointer-events-none opacity-90"
            )}
            {...props}
          >
            {!isLoading && startIcon}
            {!isLoading && props.children}
            {!isLoading && endIcon}
          </a>
        ) : (
          <button
            type={type}
            className={clsx(
              className,
              variants[variant],
              "disabled:pointer-events-none disabled:opacity-60"
            )}
            {...props}
          >
            {isLoading && (
              <span className="flex items-center justify-center space-x-2">
                <Spinner />
                <span>Loading</span>
              </span>
            )}
            {!isLoading && startIcon}
            {!isLoading && <>{props.children}</>}
            {!isLoading && endIcon}
          </button>
        )}
      </>
    );
  }
);

Button.displayName = "Button";

And if I want to render an <a> but use the styles of a Button, I can use it with next/link, I will use it like so:

<Link passHref href="/login">
  <Button className="text-xs" asAnchor>
    Login
  </Button>
</Link>

I get console logs such as: Expected server HTML to contain a matching <svg> in <a>.

Commenting out the above links removes the error, so im sure the problem is with how I render my links.

It was fine in react 17 but in react 18, I have this error.