mod_rewrite: If the file exists in another directory, serve that one instead

I have a website at example.com/test/. Lets say the website is laid out as such:

example.com
└── test/
    ├── assets/
    │   └─ stylesheet.css
    │
    ├── .htaccess
    └── index.php

index.php here is the router, as is apparently cool to do nowadays.

Whenever the user requests a page like example.com/test/stylesheet.css, I want to check to see if assets/ has that file, and if so, serve that file instead of giving the URL to index.php. Ideally, the following would work:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond assets/%{REQUEST_FILENAME} -f
RewriteRule ^(.+)$ assets/$1

But since %{REQUEST_FILENAME} is an absolute path, assets/%{REQUEST_FILENAME} turns out to be something like assets/home/public/test/stylesheet.css. %{REQUEST_URI} is no better: it turns into assets/test/stylesheet.css. I also looked at this question, but the answer didn't work either.

Is there any way, without resorting to PHP, to do this? (If not, I'll just use PHP's readfile, but I don't want to worry about LFI or anything.)