ReactNode can't be JSX Element

I have a component that display different ReactNode based on the props passed. It basically looks like this

interface RestrictedFieldProps{
  restrictedTo: LogisticFeatureFlag[];
  fallback?: React.ReactNode;
  loadingComponent?: React.ReactNode;
}

export const RestrictedField = ({
  restrictedTo,
  fallback = null,
  loadingComponent = null,
  children,
}: React.PropsWithChildren<RestrictedFieldProps>) => {
  const { loading, isAllowed } = usePermissionCtx({ restrictedTo });

  if (loading) {
    return loadingComponent
  }

  if (!isAllowed) {
    return children
  }

  return fallback;
};

Now, I have an issue with this, when I use the component I get prompted.

'RestrictedBrandField' cannot be used as a JSX component. Its return type 'ReactNode' is not a valid JSX element. Type 'undefined' is not assignable to type 'Element | null'.ts(2786)

Fine, I don't see where I do return an undefined but I tried to change to.

  if (loading) {
    return <>{loadingComponent}</>;
  }

  if (!isAllowed) {
    return <>{children}</>;
  }

  return <>{fallback}</>;

But I get a warning from eslint

Fragments should contain more than one child - otherwise, there’s no need for a Fragment at all.

Not good also...

What should I do in such case ? Is eslint wrong ?