Julia: Can the HTTP.download function create a directory that doesn't exist?

Dean_Schulze_34372 0
Viewed 56

This code failed because the ../data/housing/ directory didn't exist:

function FetchHousingData(url::String, path::String)
    println("FetchHousingData " * Dates.format(now(), "HH:MM:SS") )
    HTTP.download(url, path)
    println("done "  * Dates.format(now(), "HH:MM:SS") )
end

housingUrl = "https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/housing/housing.tgz"
housingPathFile = "../data/housing/housing.tgz"

FetchHousingData(housingUrl, housingPathFile)

Is there a way to have HTTP.download() create a missing directory? I haven't been able to find the docs for HTTP.download.

julia download
2 Answers

You can get the docs from the REPL as usual by entering help mode (press ? at the REPL prompt):

help?> HTTP.download
  download(url, [local_path], [headers]; update_period=1, kw...)

  Similar to Base.download this downloads a file, returning the filename. If the local_path:

    •  is not provided, then it is saved in a temporary directory

    •  if part to a directory is provided then it is saved into that directory

    •  otherwise the local path is uses as the filename to save to.

  When saving into a directory, the filename is determined (where possible), from the rules of the HTTP.

    •  update_period controls how often (in seconds) to report the progress.
       • set to Inf to disable reporting

    •  headers specifies headers to be used for the HTTP GET request

    •  any additional keyword args (kw...) are passed on to the HTTP request.

There doesn't seem to be a kwarg in HTTP.download to create directories, but you might be interested in isdir:

help?> isdir

  isdir(path) -> Bool

  Return true if path is a directory, false otherwise.

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  julia> isdir(homedir())
  true
  
  julia> isdir("not/a/directory")
  false

  See also isfile and ispath.

and mkdir:

help?> mkdir

  mkdir(path::AbstractString; mode::Unsigned = 0o777)

  Make a new directory with name path and permissions mode. mode defaults to 0o777, modified by the current file creation mask. This function never creates more than one directory. If the directory already exists, or some intermediate
  directories do not exist, this function throws an error. See mkpath for a function which creates all required intermediate directories. Return path.

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  julia> mkdir("testingdir")
  "testingdir"
  
  julia> cd("testingdir")
  
  julia> pwd()
  "/home/JuliaUser/testingdir"
user_24998920

I prefer to use mkpath (also mentioned in the docs showed by Nils) as it creates parent directories and does not produce an error if the directory already exists:

housingPathFile = "../data/housing/housing.tgz"
mkpath(dirname(housingPathFile))
user_38014010
Related