Databricks Filesystem - %sh ls vs %fs ls

user_1539912 0
Viewed 2144

I have some files located in %sh ls and I would want to move those files into the filesystem of databricks (make them visible in %fs ls).

Does any of you know what the difference is between %sh ls and %fs ls, and how do I move the files between them?

I know we can utilize dbutils.fs.cp to move files that are already in %fs ls location.

Any help or pointers is appreciated.

databricks filesystems
1 Answers

when you are executing commands via %sh, they are executed on the driver node only, and showing the content on that machine. When you're doing %fs ls, it's by default shows you content of DBFS (Databricks File System), but it can also show the local content if you add the file:// prefix to the path.

You can copy or move files files as following:

  1. Using dbutils.fs.cp("file:///local-path", "dbfs-path") (or dbutils.fs.mv)
  2. By using so-called FUSE mount that mounts DBFS to local machines - you need to add the /dbfs/ prefix to a path that you want to have on DBFS, like, /dbfs/FileStore/.... (If you're using Community edition with DBR >= 7.x, then it may not work, so you have only the first method)

P.S. You can find more information in the documentation.

Alex_Ott_186270
Related