Enabling Oauth2sso on Google App Engine

I am trying to get spring security oauth2 setup on my application in Google app engine. Everything seems to work fine locally but when i deploy to app engine things start to break down. After I authenticate through google its forwarding me to a Whitelabel error page. In the console I see this error:

http://my-application.appspot.com/login?state=t…m&session_state=8b67f5df659a8324430803973b9e1726e39fd454..1ae3&prompt=none 
401 (Unauthorized)

I setup my auth with this application.yml file:

security:
  oauth2:
client:
  clientId: client-key
  clientSecret: secret-key
  accessTokenUri: https://www.googleapis.com/oauth2/v4/token
  userAuthorizationUri: https://accounts.google.com/o/oauth2/v2/auth
  clientAuthenticationScheme: form
  scope:
    - openid
    - email
    - profile
    - https://www.googleapis.com/auth/cloud-platform
resource:
  userInfoUri: https://www.googleapis.com/oauth2/v3/userinfo
  preferTokenInfo: true

My security config looks somethign like this:

@Override
protected void configure(HttpSecurity http) throws Exception {
    http.csrf().csrfTokenRepository(CookieCsrfTokenRepository.withHttpOnlyFalse())
        .and()
            .authorizeRequests()
            .antMatchers("/static/**").permitAll()
            .antMatchers("/**").hasAuthority("ROLE_ADMIN")
            .anyRequest().authenticated()
        .and()
            .exceptionHandling()
            .accessDeniedPage("/403");
}

I have configured the Oauth ID on the google credential pages to allow authorized javascript origins to be:

http://my-application.appspot.com
https://my-application.appspot.com
http://localhost:8080

And the authorized redirect URIs to:

http://my-application.appspot.com/login
https://my-application.appspot.com/login
http://localhost:8080/login

Any ideas why i might be getting unauthorized errors once I deploy to GAE?

Thanks,

Craig