Java exception when running pyspark on Macbook

I just started learning Spark and met some issues starting the interactive shell. When I ran pyspark in command line, I got the following output:

Python 3.8.2 (default, Dec 21 2020, 15:06:04)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:54)
    at org.apache.spark.internal.config.package$.<init>(package.scala:1095)
    at org.apache.spark.internal.config.package$.<clinit>(package.scala)
    at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$loadEnvironmentArguments$3(SparkSubmitArguments.scala:157)
    at scala.Option.orElse(Option.scala:447)
    at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:157)
    at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:115)
    at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1022)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1022)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @13545af8
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
    at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
    at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
    at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:56)
    ... 13 more
Traceback (most recent call last):
  File "/Users/yxiong/sys/python-venv/lib/python3.8/site-packages/pyspark/python/pyspark/shell.py", line 35, in <module>
    SparkContext._ensure_initialized()  # type: ignore
  File "/Users/yxiong/sys/python-venv/lib/python3.8/site-packages/pyspark/context.py", line 331, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/Users/yxiong/sys/python-venv/lib/python3.8/site-packages/pyspark/java_gateway.py", line 108, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
>>>

More details about my environment:

1. I am on macOS Catalina 10.15.7.

2. The python version is 3.8.2, and I use virtualenv to manage my python packages.

3. I installed pyspark with pip install pyspark from my virtualenv, and no error was raised during the installation. I also tried to download the pre-built spark-3.1.2-bin-haddop3.2 tarball from apache site and ran the bin/pyspark from their, but got the same error as above.

4. I installed jre-8u301 and jdk-16.0.2 from oracle website, and java -versions shows the following. I added export JAVA_HOME=$(/usr/libexec/java_home) in my ~/.bash_profile (which evaluates to /Library/Java/JavaVirtualMachines/jdk-16.0.2.jdk/Contents/Home), but it did not solve the problem above.

   $ java -version
   java version "16.0.2" 2021-07-20
   Java(TM) SE Runtime Environment (build 16.0.2+7-67)
   Java HotSpot(TM) 64-Bit Server VM (build 16.0.2+7-67, mixed mode, sharing)
   

How can I fix or troubleshoot this issue?