How can I combine rewrite and client_max_body_size in nginx?

I want to enable large file uploads to a single location, and that location is then rewritten to another location. It seems that the rewrite is resetting the other configurations.

My Config:

server {
    listen 80;

    server_name example.com;
    root /var/www;
    index index.php;

    # this location requires large file upload
    location /upload {
        client_max_body_size 512M;
        rewrite ^(.*)$ /index.php?request=$1 last;
    }

    # all other locations
    location / {
        rewrite ^(.*)$ /index.php?request=$1 last;
    }

    # pass the PHP scripts to FPM
    location ~ \.php$ {
        include /etc/nginx/includes/php;
    }
}

If I move the client_max_body_size out of the location and into the server, then it works.

If I put it in both the location /upload AND location ~ \.php$, then it also works. But I don't want other locations to be able to upload large files.

I was thinking I could direct to PHP directly in the location /upload, but once I run the rewrite it will look for another location anyway. Does that mean I would have to have two separate locations for php scripts? Is there any way to make the client_max_body_size retain through other locations after the rewrite?