PySpark 2.4.5 is not compatible with Python 3.8.3, how do I solve this?

Viewed 7450

Code

from pyspark import SparkContext,SparkConf

conf=SparkConf().setMaster('local').setAppName('Test App')
sc=SparkContext(conf)

Error Message

    Traceback (most recent call last):
      File "C:\Users\Test\PycharmProjects\python-test\MainFile.py", line 5, in <module>
        from pyspark import SparkContext,SparkConf
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\__init__.py", line 51, in <module>
        from pyspark.context import SparkContext
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\context.py", line 31, in <module>
        from pyspark import accumulators
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\accumulators.py", line 97, in <module>
        from pyspark.serializers import read_int, PickleSerializer
      File "C:\Test\Python_3.8.3_Latest\lib\sit`enter code here`e-packages\pyspark\serializers.py", line 72, in <module>
        from pyspark import cloudpickle
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\cloudpickle.py", line 145, in <module>
        _cell_set_template_code = _make_cell_set_template_code()
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
        return types.CodeType(
    TypeError: an integer is required (got type bytes)
2 Answers

Although latest Spark doc says that it has support for Python 2.7+/3.4+, it actually doesn't support Python 3.8 yet. According to this PR, Python 3.8 support is expected in Spark 3.0. So, either you can try out Spark 3.0 preview release (assuming you're not gonna do a production deployment) or 'temporarily' fall back to Python 3.6/3.7 for Spark 2.4.x.

Spark 3.0 has been released for a while now and is compatible with Python 3.8.+. The error you experienced is no longer reproducible.

Related