General approach

Usually in MaterialUI when we want to override MaterialUI styles that are applied conditionally, we may use the classes prop:

classes={{ root: <class to apply always>, error: <class to apply in error state>}}

You can read about the CSS API and which classes are applicable for a given component at the bottom of its API page.

Alternatively, we can use a more elaborate approach and simply apply a rule whenever the component is in error state:

className={ error ? <class to apply in error state> : undefined }

The Select component

The Select component seems to be written at another time than most of the other MaterialUI input components. This is my intuitive impression based on API and code style but I might be wrong. Nevertheless, the Select component uses one of Input, FilledInput or OutlinedInput depending on the value of the variant property. Since you use outlined, I will continue this answer with OutlinedInput in mind. I will also use the color green for the border instead of orange because its distinction from red is more clear.

Naively, I would tackle the problem by looking at the CSS API of the Select component, and then also look closer at the OutlinedInput to which remaining properties of the Select component are spread. No applicable CSS API class exists on the Select component to reach your goal, but the OutlinedInput has a class error with the following description:

Pseudo-class applied to the root element if error={true}.

If we inspect the OutlinedInput in the development tools, we can see that the element that has the border is a fieldset residing inside the root element of the OutlinedInput (note: not the root element of the Select). If we use the error class, we will end up with an additional class applied to the root element of the OutlinedInput which is not what we want. Nonetheless, we could apply a class that refers to the aforementioned child element... this would work.

There is also a css class notchedOutline which we can see is applied to this fieldset resulting in the border. The dilemma here is that if we would be able to add some css to this class, then it would be applied at all times on the fieldset, but we want to apply it ONLY in error state.

This gives us four possible strategies to affect the css via the CSS API:

• Add css conditionally (when in error) to the root class of the underlying OutlinedInput that targets child notchedOutline.
• Add css to the root class of the underlying OutlinedInput that targets error state and child notchedOutline.
• Add css to the error class of the underlying OutlinedInput and target child notchedOutline.
• Add css conditionally (when in error) to the notchedOutline class of the underlying OutlinedInput.

Note that the second one and third are preferable in that they do not require us to add a class conditionally; this is handled by MaterialUI.

In the best of worlds, I would want the Select component to forward all of its non-existing props to the element it uses internally, in this case an OutlinedInput, but apparently, it does not forward portions of elements that it partially supports (the classes prop). This means that if we supply classes.error or classes.notchedOutline, MaterialUI will complain with

Material-UI: The key `XXXX` provided to the classes prop is not implemented in ForwardRef(Select).

instead of forwarding the prop to the underlying OutlinedInput. I do not know if this is a bug or not, but this is how it works currently.

This implies that we can not use any of the above four strategies via the classes prop of the Select component. A workaround is to use the Select className prop which is forwarded to the OutlinedInput which in turn forwards it to InputBase which forwards it to the native element, which is the same as the root element of the OutlinedInput. So, if we could access the underlying OutlinedInput itself, we could use all of the above strategies, otherwise only the first two (and then via className prop of the Select).

The solutions

1. Set the global error color of the entire app

If you want the error color to be something else than red for the entire app, then provide a theme which changes the color used for errors. This is the best approach since it also changes the color of helper texts and labels which would otherwise have a different color than the border of your Select input.

import green from '@material-ui/core/colors/green';

const theme = createTheme({
    palette: {
        error: {
            main: green[500]
        },
    },
});

This is a clean and non-hacky way to do it and the one I recommend. However, if you DON'T want the general color for errors to be changed, this might not be a good solution, even though you could wrap just a local part of the app in a theme of its own.

2. Override the class used for notchedOutline in error state

This is a broad way to solve it since you can do this in several different ways using the four strategies described above. The downsides with all the solutions are that they are more or less hacky and that they only affect the border of the Select, implying that helper texts and labels will be differently colored than the border when the component is in error state (you could add styles to them too, making all of this very elaborate and hard to achieve).

2.a Global override / default for OutlinedInput

We can create a theme with a global override for the OutlinedInput. Since we then manipulate the OutlinedInput itself, we may use all four strategies from above.

Global override / default for OutlinedInput for root class targeting notchedOutline child

This theme would have to be applied conditionally only when the component is in error state. This is elaborate and not a good alternative.

Global override / default for OutlinedInput for root class targeting error state and notchedOutline child

const themeWithOverrides = createTheme({
    overrides: {
        MuiOutlinedInput: {
            root: {
                "&$error": {
                    "& $notchedOutline": {
                        border: "1px solid green"
                    }
                }
            }
        }
    }
})

This one works well and targets the OutlinedInput root class with css that applies only when it is in error state and applies to its notchedOutline child. The selectors refer to the css classes already defined by default in the theme for this component.

Global override / default for OutlinedInput for error class and notchedOutline child

const themeWithOverrides = createTheme({
    overrides: {
        MuiOutlinedInput: {
            error: {
                "& $notchedOutline": {
                    border: "1px solid green"
                }
            }
        }
    }
})

This solution results in an error message from MaterialUI

Material-UI: The `MuiOutlinedInput` component increases the CSS specificity of the `error` internal state.

which in turn recommends us to use the previous solution targeting the root class instead.

Global override / default for OutlinedInput for notchedOutline class

const themeWithOverrides = createTheme({
    overrides: {
        MuiOutlinedInput: {
            notchedOutline: {
                border: "1px solid green"
            }
        }
    }
})

This will apply the style at all times which we don't want, so the theme has to be applied conditionally which is a hassle so this solution is not recommended at all.

2.b Css classes for Select component

Here we could also, in theory, use all of the four strategies described above. However, since we are not manipulating the OutlinedInput itself, and since MaterialUI doesn't forward the classes props that are not defined for the Select to the underlying component, only two remains.

Adding css class conditionally to className prop of the Select which is applied to the OutlinedInput root component targeting child notchedOutline

const useStyles = makeStyles((theme) => ({
    errorState: {
        "& .MuiOutlinedInput-notchedOutline": {
            border: "1px solid green !important"
        }
    }
})) 

...

<Select
    ...
    className={ errorFlag ? classes.errorState : undefined }
    ...
</Select>

This solution both uses literal MaterialUI css classes and requires us to conditionally add the class. Finally, we have to use the !important modifier to override the specificity of the MaterialUI default behaviour. All in all, this has a slightly hacky flavour to it.

Adding css class to className prop of the Select which is applied to the OutlinedInput root component targeting child notchedOutline only in error state

const useStyles = makeStyles((theme) => ({
    errorState: {
        "&.Mui-error": {
            "& .MuiOutlinedInput-notchedOutline": {
                border: "1px solid green"
            }
        }
    }
})) 

...

<Select
    ...
    className={ classes.errorState }
    ...
</Select>

This is slightly better because we don't need to conditionally apply the style. Also, as long as this style is injected after the MaterialUI default style, we can omit the !important modifier since this rule achieves the same specificity as the MaterialUI default behaviour. This method is absolutely preferable over the previous.

2.c Css classes for underlying OutlinedInput component via input prop on Select

The Select component takes a prop input which lets you define your own input components to be used. Here we can use a custom component that returns an OutlinedInput with the modifications you want.

export default function CustomInput(props) {
  
    return (
        <OutlinedInput
            {...props}
            <CHANGES HERE>
        />
    )
}

Since we here can work directly towards the OutlinedInput, we can use its CSS API and thus use all the four strategies described earlier.

Note that to do it by the book, you should merge incoming classes.error and classes.notchedOutline with the classes you assign by default as well.

Adding css class conditionally to classes.root prop targeting child notchedOutline

Since we now have access to the classes.error prop, providing this class conditionally would not be a smart option.

Adding css class to classes.root prop targeting state error and child notchedOutline

Similarly, since we now have access to the classes.error prop, providing a class here targeting error state instead of targeting it directly would be an extra workaround and is not recommended.

Adding css class to classes.error prop targeting child notchedOutline

const useStyles = makeStyles((theme) => ({
    error: {
        "& .MuiOutlinedInput-notchedOutline": {
            border: "1px solid green !important"
        }
    }
}))

export default function CustomInput(props) {

    const classes = useStyles()
  
    return (
        <OutlinedInput
            {...props}
            classes={{ ...props.classes, error: classes.error }}
        />
    )
}

This is a good alternative that allows us to leave the decision of whether the component is in error state or not to MaterialUI. Slightly hacky to use the literal MaterialUI css classes and that we have to provide the !important flag to override the higher specificity of the MaterialUI default behaviour.

Adding css class conditionally to classes.notchedOutline prop when in error state

const useStyles = makeStyles((theme) => ({
    errorOutline: {
        border: "1px solid green !important"
    }
}))

export default function CustomInput(props) {

    const classes = useStyles()
  
    return (
        <OutlinedInput
            {...props}
            classes={{ ...props.classes, notchedOutline: errorFlag ? classes.errorOutline : undefined }}
        />
    )
}

This is equally good as the previous. It saves us from using literal MaterialUI css classes but it forces us to add the class conditionally. Also here we have to provide the !important flag to override the higher specificity of the MaterialUI default behaviour.

Conclusion

It's a shame that it seems to be quite complicated to change such a simple thing. Nonetheless, the best solutions seem to be:

Use global error color override if you want to have a robust solution with uniform appearence for all inputs in your app that harmonize with labels and helper texts.

import green from '@material-ui/core/colors/green';

const theme = createTheme({
    palette: {
        error: {
            main: green[500]
        },
    },
})

Use global override for OutlinedInput if you want a uniform error color across all OutlinedInputs and if you're ok with labels and helper texts that don't harmonize.

const themeWithOverrides = createTheme({
    overrides: {
        MuiOutlinedInput: {
            root: {
                "&$error": {
                    "& $notchedOutline": {
                        border: "1px solid green"
                    }
                }
            }
        }
    }
})

Use a custom input which conditionally overrides the classes.notchedOutline when the component is in error state if you need a one-time change on a single or couple of Selects / OutlinedInputs and if you can accept labels and helper texts that don't harmonize in color. I prefer this solution because I don't like to use literal MaterialUI css classes in my code.

const useStyles = makeStyles((theme) => ({
    errorOutline: {
        border: "1px solid green !important"
    }
}))

export default function CustomInput(props) {

    const classes = useStyles()
  
    return (
        <OutlinedInput
            {...props}
            classes={{ ...props.classes, notchedOutline: errorFlag ? classes.errorOutline : undefined }}
        />
    )
}

...
<Select input={<CustomInput error={errorFlag} />} />
...

Here is a code sandbox with some of these recommended examples

You can use this styles

const useStyles = makeStyles((theme) => ({
  select: {
    borderColor:'#f00',
    '&:focus': {
      borderColor: '#f00',
    },
  },
}));