303 cache is preventing headers from being sent

I am configuring a odoo server to block certain routes for people outside of some networks. I am using Nginx as a reverse proxy on this server.

My issue is with the route /web/session/lougout. When i add the two following blocks to my config, browsers start caching the 303 answer from the route and stop sending headers to the server. Since the headers are missing, it prevents the server from invalidating the session and leads to a bunch of issues.

location ~ ^/web/ {
        add_header Content-Security-Policy upgrade-insecure-requests;
        proxy_pass  http://odoo-xxx-test;
        proxy_buffering on;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        expires 864000;
        allow *some ip* ;
        allow *some ip* ;
        allow *some ip* ;
        deny all;
    }

    location ~ ^\/web\/(action\/|content\/|static\/|image\/|login|session\/|webclient\/) {
        add_header Content-Security-Policy upgrade-insecure-requests;
        proxy_pass  http://odoo-xxx-test;
        proxy_cache_valid 200 60m;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_buffering on;
        expires 864000;
    }

From my understanding, the first location should deny access to all routes outside of the 3 allowed ips and the second should allow everyone outside of those 3 ips to access those 7 routes. I dont understand how those two block affect the caching of the /web/session/logout route, removing those two directives fixes the issues but this behavior is in the requirements for the projects.

Any help would be welcome!