Pyspark EMR Notebook - Unable to save file to EMR environment

user_13136602 0
Viewed 880

This seems really basic but I can't seem to figure it out. I am working in a Pyspark Notebook on EMR and have taken a pyspark dataframe and converted it to a pandas dataframe using toPandas().

Now, I would like to save this dataframe to the local environment using the following code:

movie_franchise_counts.to_csv('test.csv')

But I keep getting a permission error:

[Errno 13] Permission denied: 'test.csv'
Traceback (most recent call last):
  File "/usr/local/lib64/python3.6/site-packages/pandas/core/generic.py", line 3204, in to_csv
    formatter.save()
  File "/usr/local/lib64/python3.6/site-packages/pandas/io/formats/csvs.py", line 188, in save
    compression=dict(self.compression_args, method=self.compression),
  File "/usr/local/lib64/python3.6/site-packages/pandas/io/common.py", line 428, in get_handle
    f = open(path_or_buf, mode, encoding=encoding, newline="")
PermissionError: [Errno 13] Permission denied: 'test.csv'

Any help would be much appreciated.

amazon-emr pyspark pandas
3 Answers

When you are running PySpark in EMR Notebook you are connecting to EMR cluster via Apache Livy. Therefore all your variables and dataframes are stored on the cluster and when you run df.to_csv('file.csv') you are trying to save CSV on the cluster and not in your local enviroment. I've struggled a bit, but this worked for me:

  1. Store your PySpark dataframe as temporary view: df.createOrReplaceTempView("view_name")
  2. Load SparkMagic: %load_ext sparkmagic.magics
  3. Select from view and use SparkMagic to load output to local (-o flag)
%%sql -o df_local --maxrows 10
SELECT * FROM view_name
  1. Now you have your data in Pandas dataframe df_local and can save it with df_local.to_csv('file.csv')
user_117704920

It depends on where exactly your Kernel is running, i.e, whether it is running locally or on remote cluster. In case of EMR Notebooks, for EMR release label 5.30, 5.32+(except 6.0 and 6.1), all kernels run remotely on the attached EMR cluster, hence when you try to save file, it is actually trying to save file on the cluster and you may not have access to that directory on the cluster. For release label other than those mentioned above, kernels run locally and hence for those release label you would be able to save file locally with your code.

user_57290170

I believe the best way to do it would be to save to s3 directly from a pyspark dataframe like this

df.repartition(1).write.mode('overwrite').csv('s3://s3-bucket-name/folder/', header=True)

Note: You don't need a file name here since pyspark will create a file with a custom name such as part-00000-d129fe1-7721-41cd-a97e-36e076ea470e-c000.csv

user_44195220
Related