You can use 'react-test-renderer':
like this:
const fakeNavigation = {
navigate: jest.fn(),
};
const tree = renderer.create(
<YourComponent navigation={fakeNavigation} />
);
after that you can use jest expect function to test if mock navigation.navigate is called someThing like this:
const instance = tree.getInstance();
instance.onSubmit();
expect(fakeNavigation.navigate).toBeCalledWith('MainPage')
for react-native-testing-library it's a bit different as you test mostly what user sees depending of the behavior of your component.
Basically you look for your button that calls onSubmit and then use fireEvent from react-native-testing-library to press it, after that you expect the text of your alert to be shown in screen or not.
If you have a button like this in your component=
<TouchableOpacity onPress={onSubmit}>
<Text>Submit</Text>
</TouchableOpacity>
You can test the behavior of you function like this:
const {getByText} = render(<YourComponent/>)
let buttonText = getByText('Submit');
fireEvent(buttonText.parent, 'press'); //parent to get to TouchableOpacity
alertText = getByText('Not Allowed');
expect(alertText).toBeDefined();