How to use Pipe/HTTP/CSV to download, extract and import a zipped or tgz csv file from internet?

user_1586860 0
Viewed 73

For a CSV file, I am using a combination of Pipe, HTTP and CSV to download, (eventually) modify the file and import it as a DataFrame without any temporary writing on disk:

urlData = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.csv"
data = @pipe HTTP.get(urlData).body                |>
             replace!(_, UInt8(';') => UInt8(' ')) |>  # if we need to apply modifications to the file before importing 
             CSV.File(_, delim=' ')                |>
             DataFrame;

How can I use the same general approach (pipe without temp disk saving) when the file has been compressed with zip or tar (assuming a single file in the archive) ?

For example:

urlDataZ = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.zip"
urlDataT = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.tgz"

For zipped files I can use this approach:

data = @pipe ZipFile.Reader("data.zip").files[1] |>
             CSV.File(read(_), delim=';') |>
             DataFrame

But I can't arrive to put the three things (download, decompress and import) together without passing for an explicit temp file on disk.

julia dataframe csv download http
1 Answers

ZipFile.Reader accepts IO arguments too, so you can wrap things in an IOBuffer like this:

julia> @pipe HTTP.get(urlDataZ).body             |>
             IOBuffer(_)                         |>
             ZipFile.Reader(_).files[1]          |>
             CSV.read(_, DataFrame, delim = ';')
             
8×4 DataFrame
 Row │ Country  Year   forarea   forvol  
     │ String7  Int64  Float64   Float64 
─────┼───────────────────────────────────
   1 │ Germany   2000  11.354    3381.0
   2 │ France    2000  15.288    2254.28
   3 │ Italy     2000   8.36925  1058.71
   4 │ Sweden    2000  28.163    3184.67
   5 │ Germany   2020  11.419    3663.0
   6 │ France    2020  17.253    3055.83
   7 │ Italy     2020   9.56613  1424.4
   8 │ Sweden    2020  27.98     3653.91
sundar_81270
Related