ananconda: editing PKG_CONFIG_PATH doesn't work

I'm building a conda recipe. This was the requirements from meta.yaml and it works:

requirements:
  build:
    - {{ compiler('c') }}
    - {{ compiler('cxx') }}
    - sysroot_linux-64 2.17  # [linux64]
    - meson
    - mako
    - flex
    - bison
    - pkg-config

  host:
    - zlib
    - expat
    - libdrm                       # [linux]
    - elfutils                     # [linux]
    - ncurses
    - llvmdev 14
    - xorg-libx11

  run:
    - libllvm14

Now I hope to update it a bit to to use x11 lib from CentOS yum instead of conda:

So I delete xorg-libx11 from requirements/build and add PKG_CONFIG_PATH=/usr/lib64 to build (I have libX11.so located at /usr/lib64/libX11.so):

build:
  number: 5
  script_env:
   - PKG_CONFIG_PATH=/usr/lib64

requirements:
  build:
    - {{ compiler('c') }}
    - {{ compiler('cxx') }}
    - sysroot_linux-64 2.17  # [linux64]
    - meson
    - mako
    - flex
    - bison
    - pkg-config

  host:
    - zlib
    - expat
    - libdrm                       # [linux]
    - elfutils                     # [linux]
    - ncurses
    - llvmdev 14

  run:
    - libllvm14

Now I couldn't build the recipe as pkgconfig couldn't find x11:

Run-time dependency OpenMP found: YES 4.5
Pkg-config binary for 1 is cached.
Determining dependency 'x11' with pkg-config executable '$BUILD_PREFIX/bin/pkg-config'
env[PKG_BUILDNUM]: 5
env[PKG_VERSION]: 21.3.9
env[PKG_BUILD_STRING]: placeholder
env[PKG_CONFIG]: $BUILD_PREFIX/bin/pkg-config
env[PKG_CONFIG_PATH]: $PREFIX/lib/pkgconfig:$BUILD_PREFIX/lib/pkgconfig
env[PKG_NAME]: mesa
env[PKG_HASH]: 1234567
Called `$BUILD_PREFIX/bin/pkg-config --modversion x11` -> 1

CMake binary for 1 is cached as not found
CMake binary for machine 1 not found. Giving up.
Run-time dependency x11 found: NO (tried pkgconfig)

meson.build:2058:4: ERROR: Dependency "x11" not found, tried pkgconfig
Traceback (most recent call last):
  File "/home/liuwentao/anaconda3/bin/conda-build", line 11, in <module>

Why is PKG_CONFIG_PATH=/usr/lib64 not working?