No module named 'resource' installing Apache Spark on Windows

user_10643663 0
Viewed 9095

I am trying to install apache spark to run locally on my windows machine. I have followed all instructions here https://medium.com/@loldja/installing-apache-spark-pyspark-the-missing-quick-start-guide-for-windows-ad81702ba62d.

After this installation I am able to successfully start pyspark, and execute a command such as

textFile = sc.textFile("README.md")

When I then execute a command that operates on textFile such as

textFile.first()

Spark gives me the error 'worker failed to connect back', and I can see an exception in the console coming from worker.py saying 'ModuleNotFoundError: No module named resource'. Looking at the source file I can see that this python file does indeed try to import the resource module, however this module is not available on windows systems. I understand that you can install spark on windows so how do I get around this?

apache-spark windows python
4 Answers

I struggled the whole morning with the same problem. Your best bet is to downgrade to Spark 2.3.2

user_19268520

The fix can be found at https://github.com/apache/spark/pull/23055.

The resource module is only for Unix/Linux systems and is not applicaple in a windows environment. This fix is not yet included in the latest release but you can modify the worker.py in your installation as shown in the pull request. The changes to that file can be found at https://github.com/apache/spark/pull/23055/files.

You will have to re-zip the pyspark directory and move it the lib folder in your pyspark installation directory (where you extracted the pre-compiled pyspark according to the tutorial you mentioned)

user_46213230

Adding to all those valuable answers,

For windows users,Make sure you have copied the correct version of the winutils.exe file(for your specific version of Hadoop) to the spark/bin folder

Say, if you have Hadoop 2.7.1, then you should copy the winutils.exe file from the Hadoop 2.7.1/bin folder

Link for that is here

https://github.com/steveloughran/winutils

user_110452790

I edited worker.py file. Removed all resource-related lines. Actually # set up memory limits block and import resource. The error disappeared.

user_80365970
Related