Importing multiple py files in pyspark

user_8832641 0
Viewed 1064

I have the below folder structure in hdfs.

/home/myid -main.py -test1.py -test2.py

In main.py I am trying to import both test1 and test2.

from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
sc=SparkContext('local')
spark= SparkSession(sc)
from test1 import test_function
from test2 import new_function

test_function()
new_function()

All 3 files are in same folder.

Test1.py

def test_function():
    print("test1")

Test2.py

def new_function():
    print("test2")

When I try to execute main.py using spark-submit main.py, it says cannot import new_function.

It is able to import test1.py but not test2.py

I also tried with sc.addPyFile before import. Didn't work.

Tried by giving --py-files along with spark submit and it didn't work either.

Tried zipping both the test1 and test2 and added it along with spark context. It didn't work either.

Trying to understand why it's not working and how to resolve this.

hdfs apache-spark python
1 Answers

Option 1

Try giving with these syntax using spark-submit, make sure you are using comma

--py-files test1.py,test2.py

Option 2:

moreover you can zip them and include like this

First put them in a directory for instance myfiles/ ( In addition make empty __init__.py file at root level in this directory like myfiles/__init__.py )

From outside this directory,make a zip of it (for example myfiles.zip)

Now include this zip with sc.addPyFilefunction

sc.addPyFile("myfiles.zip")

Considering your have __init__.py , test1.py and test2.py in myfiles.zip

You can now use them as

from myfiles.Test1 import test_function
from myfiles.Test2 import new_function
user_36808310
Related