How to upload large files from local pc to DBFS?

user_19318431 0
Viewed 448

I am trying to learn spark SQL in databricks and want to work with the Yelp dataset; however, the file is too large to upload to DBFS from UI. Thanks, Filip

databricks apache-spark-sql
1 Answers

There are several approaches to that:

  1. Use Databricks CLI's dbfs command to upload local data to DBFS.
  2. Download dataset directly from notebook, for example by using %sh wget URL, and unpacking the archive to DBFS (either by using /dbfs/path/... as destination, or using dbutils.fs.cp command to copy files from driver node to DBFS)
  3. Upload files to AWS S3, Azure Data Lake Storage, Google Storage or something like, and accessing data there.
Alex_Ott_186270
Related