after struggling with this several times, and coming across it right now, I figured I'd share my solution.
long story short: just install things into the default blender environment or install bpy into an anaconda environment
On linux you may be able to follow Failxxx's answer, or using mklink in Windows maybe the same (I couldn't get it to work, got weird "compiler" errors when trying to import bpy in blender).
You can also check here for various workarounds:
https://docs.blender.org/api/current/info_tips_and_tricks.html
In the end, I've had success building bpy as a module inside of an anaconda environment (follow tutorial in link), or installing whatever packages I want inside of my Blender python environment.
To do this.. find out where your Blender's python.exe is stored.. mine is here:
C:\Program Files\Blender Foundation\Blender 3.2\3.2\python\bin
then, open a cmd as admin (admin was extremely crucial for my machine?) in this directory and do something like:
python -m pip install pandas
this will install the package into your Blender python environment. Sometimes I had to uninstall and reinstall with admin incase anything had been installed to C:\Users\%USERNAME%\AppData\Roaming - Python or Blender folders.
if it's only a handful of packages, this is the best way in my opinion. Just know that this permanently alters the python environment that shipped with Blender.
Proof :) :
