SAML2 and Form based login in same application with Spring Security

I'm trying to figure out how to support both form-based login for most users and saml2-based (spring security 5.4) login for some users in our application.

1. Users accessing app.exampleapp.com should be redirected to form login.
2. Users accessing sso.exampleapp.com should be redirected to SAML2 Identity Provider login.

I tried with a single Java configuration to handle both types but that did'nt work at all. So I'm trying with one configuration for form login and a separate configuration for saml2 login. But I can't get it to work either. One of them seems to take precedence over the other. If I disable the saml2 config I can login with form login and if I disable the form login config I can login using SSO/Keycloak. But I can't get both configs to work at the same time.

I have a RequestMatcher that matches on sub-domains in each security configuration.

Saml2 login config

http
    .requestMatcher(new SubdomainSecurityMatcher("sso"))
    .authorizeRequests()
    .antMatchers("/**")
    .authenticated()
    .and()
    .saml2Login(withDefaults())

Form login config

http
    .requestMatcher(new SubdomainSecurityMatcher("app"))
    .authorizeRequests()
    .antMatchers("/**")
    .authenticated()
    .and()
    .formLogin()

Do I need a custom AuthenticationEntryPoint to decide what login processing to perform?

I tried a lot of things but I'm not sure if or how it's supposed to work. Anyone managed to do something similar with the new SAML2 support in Spring Security 5.4?