mount S3 to databricks

user_4046073 0
Viewed 9092

I'm trying understand how mount works. I have a S3 bucket named myB, and a folder in it called test. I did a mount using 

var AwsBucketName = "myB"
val MountName = "myB"

My question is that: does it create a link between S3 myB and databricks, and would databricks access all the files include the files under test folder? (or if I do a mount using var AwsBucketName = "myB/test"does it only link databricks to that foldertestbut not anyother files that outside of that folder?)

If so, how do I say list files in test folder, read that file or or count() a csv file in scala? I did a display(dbutils.fs.ls("/mnt/myB")) and it only shows the test folder but not files in it. Quite new here. Many thanks for your help!

databricks amazon-web-services amazon-s3 scala
2 Answers

From the Databricks documentation:

// Replace with your values
val AccessKey = "YOUR_ACCESS_KEY"
// Encode the Secret Key as that can contain "/"
val SecretKey = "YOUR_SECRET_KEY".replace("/", "%2F")
val AwsBucketName = "MY_BUCKET"
val MountName = "MOUNT_NAME"

dbutils.fs.mount(s"s3a://$AccessKey:$SecretKey@$AwsBucketName", s"/mnt/$MountName")
display(dbutils.fs.ls(s"/mnt/$MountName"))

If you are unable to see files in your mounted directory it is possible that you have created a directory under /mnt that is not a link to the s3 bucket. If that is the case try deleting the directory (dbfs.fs.rm) and remounting using the above code sample. Note that you will need your AWS credentials (AccessKey and SecretKey above). If you don't know them you will need to ask your AWS account admin for them.

user_83283920

It only lists the folders and files directly under bucket.

In S3 

<bucket-name>/<Files & Folders>

In Databricks

/mnt/<MOUNT-NAME>/<Bucket-Data-List>

Just like below (Output for dbutils.fs.ls(s"/mnt/$MountName"))

dbfs:/mnt/<MOUNT-NAME>/Folder/  
dbfs:/mnt/<MOUNT-NAME>/file1.csv
dbfs:/mnt/<MOUNT-NAME>/file2.csv
user_21123820
Related