Yet another React + Typescript conditional props question

Pretty new to Typescript. I'm trying to make a Button component that builds a className string based on two props: variant and color. Using those values it will look up a className string on an object like the following:

const config = {
  primary: {
    blue: '...',
    green: '...',
  },
  secondary: {
    blue: '...',
    green: '...',
  },
  tertiary: {
    blue: '...',
  },
}

Here's where I've ended up on the Button component:

type Variant = keyof typeof config

type VariantButton<T extends Variant> = {
  variant?: T
  color?: T extends Variant ? keyof typeof config[T] : never
}

type Props =
  | VariantButton<'primary'>
  | VariantButton<'secondary'>
  | VariantButton<'tertiary'>

const Button: React.FC<Props> = ({
  variant = 'primary',
  color = 'blue',
  ...rest
}) => {
  const className = config[variant]?.[color] // TYPESCRIPT ERROR HERE

  return (
    <button
      {...rest}
      className={className}
    />
  )
}

The error is telling me that property green does not exist on type { blue: '...' }, which is true of course, but how do I get typescript to treat the variant/color combinations separately so that it only expects color properties from the passed variant?

Playground