How do I load a local file onto Spark using a Jupyter notebook in an EMR cluster?

user_1175788 0
Viewed 2739

I'm struggling to load a local file on an EMR core node into Spark and run a Jupyter notebook. I keep getting errors from the task nodes saying that the file doesn't exist, but I've tried setting the spark configuration to be local, so I'm not sure how to fix this. The following works when I spin up a 1-node cluster, but fails when I have anything larger than that:

spark = SparkSession.builder \
        .master("local") \
        .appName("Test") \

df = spark.read.csv('/home/hadoop/dataset.csv')

df.show(n=5)

I've tried restarting the Jupyter notebook kernel, but that hasn't fixed anything. So I'd like to know how to either broadcast/share a local file onto the entire cluster, or create a SparkSession instance that works. I'm also using the new AWS JupyterHub, if that makes any difference

jupyter-notebook pyspark apache-spark
1 Answers

I could't get this to work either, here is my workaround:
(Assuming your problem is in Amazon EMR, too & you have the possibility to use S3 Buckets)

  1. (Services -> Storage -> S3 -> Create Bucket -> Bucket name: "emr-data-<account#>" *
  2. upload "dataset.csv" into your S3 Bucket "emr-data..."
  3. in JupyterNotebook: 
    #store URL: 
url = "s3n://emr-data-<account#>/dataset.csv"  

#read json from URL  
df = spark.read.csv(url, header=True)

#show first 5 rows of data frame
df.show(n=5)

the resulting spark data frame "df" can be used as usual

(* names of S3 Buckets must be unique, thus adding your own account number to the name is one easy way to achieve this. Also, adding the Region to the name (emr-bucket-123456789012-eu-central-1) is good practice)

user_149341640
Related