How to set a custom color on the chakra UI colorScheme prop

I want to pass a custom color to the colorScheme prop but it's not working. An answer on stack overflow suggests I set custom colors in the theme object and then pass that to the chakraProvider, but I don't want to have chakras theme on my application. How do I fix this?