Does Julia have an equivalent to R's list.files(recursive = TRUE)?

user_19297197 0
Viewed 77

I've written the folloeing function but it is ~50% slower than R's list.files(x, recursive = TRUE). Maybe there is also a way to make it faster.

function list_files(x)
    v = String[]
    for (root, dirs, files) in walkdir(x)
        for file in files 
            file = joinpath(root, file)
            push!(v, file)
        end
    end
    v
end
julia
3 Answers

I dunno, on my machine walkdir seems pretty close to optimal.

julia> using BenchmarkTools

julia> @benchmark list_files($".")
BenchmarkTools.Trial: 195 samples with 1 evaluation.
 Range (min … max):  22.098 ms … 54.305 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     25.395 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   25.636 ms ±  3.179 ms  ┊ GC (mean ± σ):  0.13% ± 0.87%

            ▄▂  ▄    ▄ ▅▅  ▅    ▂█                             
  ▆▃██▇▆▆▇█▇██▇▆██▇▇▇█▆██▆▆███▇▅██▅▇▅▁▆▁▅▅▇▅▆▆▁▅▃▁▁▁▃▁▃▁▁▁▁▁▃ ▃
  22.1 ms         Histogram: frequency by time        31.1 ms <

 Memory estimate: 989.60 KiB, allocs estimate: 8987.

julia> @benchmark split(read($`find . -type f`, String), '\n')
BenchmarkTools.Trial: 149 samples with 1 evaluation.
 Range (min … max):  26.661 ms … 47.466 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     32.541 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   33.547 ms ±  4.435 ms  ┊ GC (mean ± σ):  0.60% ± 3.90%

            ▄    █    ▂ ▂                                      
  ▃▁▆█▆▅▆█▇▃█▇▇▇▇██████▆█▃█▅▇▁▁▆▅▅▆▁▅▅▃▃▇▅▃▃▅▁▇▁▁▃▃▁▃▁▁▁▃▅▃▃▃ ▃
  26.7 ms         Histogram: frequency by time        45.3 ms <

 Memory estimate: 626.14 KiB, allocs estimate: 79.

If R is really much faster than find . -type f, then that's pretty impressive.

user_54964330

With regard to the other answer. Are you performing the measurements properly - disks and file managers have various caching methods. Hence two separate calls can have very different measurements.

Regarding the code you can use walkdir in a one liner. For an example:

julia> [(joinpath.(folder,files) for (folder,dir,files) in walkdir(raw"C:\Julia-1.8.0-rc3\lib"))...;]
3-element Vector{String}:
 "C:\\Julia-1.8.0-rc3\\lib\\libjulia.dll.a"
 "C:\\Julia-1.8.0-rc3\\lib\\libopenlibm.dll.a"
 "C:\\Julia-1.8.0-rc3\\lib\\julia\\sys.dll"```
user_99577100

Seemly for my use case (a folder with a lot of recursion) R is really faster.

R result:

  expression                              min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result memory     time       gc      
  <bch:expr>                         <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list> <list>     <list>     <list>  
1 list.files(path, recursive = TRUE)   16.5ms   20.3ms      45.1    1.25KB        0    23     0      510ms <chr>  <Rprofmem> <bench_tm> <tibble>

Julia result:

@benchmark [(joinpath.(folder,files) for (folder,dir,files) in walkdir(path))...;]

BenchmarkTools.Trial: 8 samples with 1 evaluation.
 Range (min … max):  574.158 ms … 976.682 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     598.369 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   649.435 ms ± 134.988 ms  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▁█ ▁▁ ▁      ▁                                              ▁  
  ██▁██▁█▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  574 ms           Histogram: frequency by time          977 ms <

 Memory estimate: 71.47 KiB, allocs estimate: 1019.
user_192971970
Related