Databricks: Difference between dbfs:/ vs file:/

user_1468737 0
Viewed 1222

I am trying to understand the way Databricks stores files and I am a bit unsure of what the difference is between dbfs:/ and file:/ (see image below)

enter image description here

From what I have been able to deduce from here, file:/ seems to be the area where external files downloaded via curl/wget get downloaded into in the following folder path:

%fs ls "file:/databricks/driver"

But what is file:/ really and why does it exist and how is it different from dbfs:/?

For the record, I am using the community free edition of Databricks.

aws-databricks azure-databricks databricks
1 Answers

The Databricks Community Edition is hosted on Amazon Web Services.

When you create a cluster in databricks community edition, behind the scenes a Virtual machine (aws ec2 instance) created in the backend which acts a driver node to run your programs via notebooks.

A cluster management service. The service will spin up Amazon EC2 instances VM with Spark nodes already set up for you. Free 15GB memory cluster for the Community Edition and billed hourly per node for the regular version.

%fs ls "file:/" returns the folders available in the Virtual machine aws ec2 instance) which is running in the backend on the aws cloud.

enter image description here

%fs ls returns the Databricks File System (DBFS) is a distributed file system mounted into a Databricks workspace and available on Databricks clusters. DBFS is an abstraction on top of scalable object storage i.e aws S3 storage.

enter image description here

user_81884330
Related