After logging in I am redirected to /robots.txt

Viewed 719

So since a couple of weeks now we have noticed some strange behaviour on a couple of projects that use Spring Boot and Spring Security.

Mostly only OSX users seem to have this issue but whenever we login to our application it randomly redirects back to robots.txt

I realise that this is probably not enough info, but sadly it's all I got. We have seen this pop up only on OSX so far and on 3 different projects (some live and some running locally)

server logs have this as request info (I removed/obfuscated some the real info):

Request parameters:


Request attributes:
javax.servlet.forward.request_uri: '/robots.txt'
javax.servlet.forward.context_path: ''
javax.servlet.forward.servlet_path: '/robots.txt'
javax.servlet.forward.mapping: 'org.apache.catalina.core.ApplicationMapping$MappingImpl@4352fb9f'
org.springframework.web.context.request.async.WebAsyncManager.WEB_ASYNC_MANAGER: 'org.springframework.web.context.request.async.WebAsyncManager@d83be2'
javax.servlet.error.status_code: '404'
org.springframework.web.servlet.DispatcherServlet.CONTEXT: 'org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6e521844: startup date [Tue Aug 28 16:09:57 CEST 2018]; root of context hierarchy'
org.springframework.web.servlet.resource.ResourceUrlProvider: 'org.springframework.web.servlet.resource.ResourceUrlProvider@f1e816f'
org.springframework.web.servlet.HandlerMapping.introspectTypeLevelMapping: 'false'
javax.servlet.http.HttpServletResponse: 'org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse@1609a68b'
__spring_security_session_mgmt_filter_applied: 'true'
characterEncodingFilter.FILTERED: 'true'
_csrf: 'SaveOnAccessCsrfToken [delegate=org.springframework.security.web.csrf.DefaultCsrfToken@aa7c61f]'
__spring_security_filterSecurityInterceptor_filterApplied: 'true'
org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter$TimingContext: 'org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter$TimingContext@5efd5ad7'
org.springframework.web.servlet.DispatcherServlet.THEME_SOURCE: 'org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6e521844: startup date [Tue Aug 28 16:09:57 CEST 2018]; root of context hierarchy'
org.springframework.web.servlet.HandlerMapping.producibleMediaTypes: '[text/html]'
org.springframework.web.servlet.DispatcherServlet.LOCALE_RESOLVER: 'org.springframework.web.servlet.i18n.FixedLocaleResolver@5602d5b5'
org.springframework.web.servlet.HandlerMapping.bestMatchingPattern: '/error'
org.springframework.web.servlet.DispatcherServlet.OUTPUT_FLASH_MAP: 'FlashMap [attributes={}, targetRequestPath=null, targetRequestParams={}]'
errorPageFilter.FILTERED: 'true'
org.springframework.web.servlet.HandlerMapping.pathWithinHandlerMapping: '/error'
org.springframework.web.servlet.DispatcherServlet.FLASH_MAP_MANAGER: 'org.springframework.web.servlet.support.SessionFlashMapManager@1fbb7dc1'
org.springframework.security.web.csrf.CsrfToken: 'SaveOnAccessCsrfToken [delegate=org.springframework.security.web.csrf.DefaultCsrfToken@aa7c61f]'
javax.servlet.error.request_uri: '/robots.txt'
org.springframework.web.servlet.HandlerMapping.uriTemplateVariables: '{}'
org.springframework.web.servlet.DispatcherServlet.THEME_RESOLVER: 'org.springframework.web.servlet.theme.FixedThemeResolver@445dbd43'

Request headers:
host: 'website.com';
connection: 'keep-alive';
Cache-Control: 'max-age=0';
Upgrade-Insecure-Requests: '1';
user-agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36';
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8';
referer: 'https://website.com/login';
Accept-Encoding: 'gzip, deflate, br';
Accept-Language: 'en-GB,en-US;q=0.9,en;q=0.8';
cookie: 'JSESSIONID=6909C15700507E59492ECE290D8288B4';
content-length: '0';

Session attributes:
SPRING_SECURITY_CONTEXT: org.springframework.security.core.context.SecurityContextImpl@2d2bdc98: Authentication: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@2d2bdc98: Principal: com.company.application.security.UserDetails@dc9e9848: Username: username; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_CUSTOMER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@0: RemoteIpAddress: 0.0.0.0; SessionId: 2CD3B5E833D51BE0AB34455CFCC3E644; Granted Authorities: ROLE_ADMIN, ROLE_CUSTOMER

Does anyone have any idea why this seems to be happening

1 Answers

I'm guessing that the browser or one of the plugins is requesting /robots.txt. For a user that this happens to, can you inspect the traffic and see if it is being requested.

The reason it only happens occasionally is it is likely a race condition. Spring Security will redirect to the last URL that was requested that was not public.

You can try fixing the issue by marking /robots.txt as permitAll

Java Config

http
    .authorizeRequests()
        .mvcMatchers("/robots.txt").permitAll()
        ...

XML Configuration

<http ...>
    <intercept-url pattern="/robots.txt" access="permitAll"/>
    ...
</http>
Related