An explanation...
If I click on a link, say, ./page, it brings me to localhost/page instead of localhost/site/page.php
It would have internally rewritten the request to /page.php (in the document root), which would have presumably resulted in a 404. This is because of the RewriteBase / directive - which provides the URL-prefix for relative path substitutions, ie. $1.php.
Note the subdirectory is ignored ...
Because the .htaccess file is inside the /site subdirectory, the RewriteRule pattern only captures the part of the URL-path that comes after this. So, for example, if you request localhost/site/page (the resolved URL-path from your relative ./page link) then the RewriteRule pattern captures page (the value of the $1 backreference), which is rewritten to page.php, to which the RewriteBase is prefixed to become /page.php (in the document root).
I've tried changing RewriteBase / to RewriteBase /site/ and it doesn't seem to do anything. Any ideas?
Yes. In fact, this is all you needed to do.
Or, preferably, remove the RewriteBase directive entirely (which is what @anubhava did in his answer), since the .htaccess file resides in the same directory. In the absence of the RewriteBase directive, the default action for relative path substitutions is to add back the directory-prefix (ie. the file system path of the /site directory). So, given a request for /site/page (from the ./page link) it would naturally rewrite to /site/page.php without having to explicitly include the /site subdirectory.
Either way, changing the RewriteBase directive, in this case, must have done something. If not, then you were probably seeing a cached response.
My website's entry point is localhost/site/index.php
Providing the /site subdirectory is part of the visible URL, as you suggest, then the additional .htaccess file in the site root is not required (as in @anubhava's answer). Either way, you wouldn't need both.