How do I load multiple CSV into DataFrames in Julia?

Admin 1
Viewed 1063

I already know how to load a single CSV into a DataFrame:

using CSV
using DataFrames    
df = DataFrame(CSV.File("C:\\Users\\username\\Table_01.csv"))

How would I do this when I have several CSV files, e.g. Table_01.csv, Table_02.csv, Table_03.csv? Would I create a bunch of empty DataFrames and use a for loop to fill them? Or is there an easier way in Julia? Many thanks in advance!

julia
3 Answers

If you want multiple data frames (not a single data frame holding the data from multiple files) there are several options.

Let me start with the simplest approach using broadcasting:

dfs = DataFrame.(CSV.File.(["Table_01.csv", "Table_02.csv", "Table_03.csv"]))

or

dfs = @. DataFrame(CSV.File(["Table_01.csv", "Table_02.csv", "Table_03.csv"]))

or (with a bit of more advanced stuff, using function composition):

(DataFrame∘CSV.File).(["Table_01.csv", "Table_02.csv", "Table_03.csv"])

or using chaining:

CSV.File.(["Table_01.csv", "Table_02.csv", "Table_03.csv"]) .|> DataFrame

Now other options are map as it was suggested in the comment:

map(DataFrame∘CSV.File, ["Table_01.csv", "Table_02.csv", "Table_03.csv"])

or just use a comprehension:

[DataFrame(CSV.File(f)) for f in ["Table_01.csv", "Table_02.csv", "Table_03.csv"]]

(I am listing the options to show different syntactic possibilities in Julia)

user_12695670

This is how I have done it, but there might be an easier way.

using DataFrames, Glob
import CSV

function readcsvs(path)
    files=glob("*.csv", path) #Vector of filenames. Glob allows you to use the asterisk.
    numfiles=length(files)    #Number of files to read.
    tempdfs=Vector{DataFrame}(undef, numfiles) #Create a vector of empty dataframes.
    for i in 1:numfiles
        tempdfs[i]=CSV.read(files[i]) #Read each CSV into its own dataframe.
    end
    masterdf=outerjoin(tempdfs..., on="Column In Common") #Join the temporary dataframes into one dataframe.
end
user_134950490

A simple solution where you don't have to explicitly enter filenames:

using CSV, Glob, DataFrames
path = raw"C:\..." # directory of your files (raw is useful in Windows to add a \)
files=glob("*.csv", path) # to load all CSVs from a folder (* means arbitrary pattern)
dfs = DataFrame.( CSV.File.( files ) ) # creates a list of dataframes

# add an index column to be able to later discern the different sources
for i in 1:length(dfs)
    dfs[i][!, :sample] .= i # I called the new col sample
end

# finally, if you want, reduce your collection of dfs via vertical concatenation
df = reduce(vcat, dfs)
user_120522380
Related