Combine collection of DataFrames into one big DataFrame

Viewed 191

Suppose I have 3 DataFrames. One such DataFrame has a column name that's not in the other two.

using DataFrames

df1 = DataFrame([['a', 'b', 'c'], [1, 2, 3]], ["name", "id"])
df2 = DataFrame([['d', 'e', 'f'], [4, 5, 6]], ["name", "id"])
df3 = DataFrame([['x', 'y', 'z'], [7, 8, 9], [11, 22, 33]], ["name", "id", "num"])

And each DataFrame is an element of a Vector.

dfs = [df1, df2, df3]

I want to combine all of these DataFrames into one big DataFrame. Here's what I've tried:

df = reduce(x -> vcat(x, cols=:union), dfs)
# MethodError: no method matching (::var"#55#56")(::DataFrame, ::DataFrame)

So, how do I do it in Julia?

Bonus points: Can I do it in base Julia, without DataFrames.jl (note: df3 has more columns than the others)?

Since there are a ton of Julia packages that implement DataFrame-like constructs, I'm open to solutions using packages other than DataFrames.jl.


Here's how I'd do it in R:

df1 = data.frame(name = c("a", "b", "c"),
                 id = c(1, 2, 3))

df2 = data.frame(name = c("d", "e", "f"),
                 id = c(4, 5, 6))

df3 = data.frame(name = c("x", "y", "z"),
                 id = c(7, 8, 9),
                 num = c(11, 22, 33))

dfs = list(df1, df2, df3)

library(dplyr)
dplyr::bind_rows(dfs)

And Python 3.7:

import pandas as pd

df1 = pd.DataFrame({"name": ["a", "b", "c"],
                    "id": [1, 2, 3]})

df2 = pd.DataFrame({"name": ["d", "e", "f"],
                    "id": [4, 5, 6]})

df3 = pd.DataFrame({"name": ["x", "y", "z"],
                    "id": [7, 8, 9],
                    "num": [11, 22, 33]})

dfs = [df1, df2, df3]

pd.concat(dfs)
1 Answers

Just do:

julia> vcat(dfs..., cols=:union)
9×3 DataFrame
│ Row │ name │ id    │ num     │
│     │ Char │ Int64 │ Int64?  │
├─────┼──────┼───────┼─────────┤
│ 1   │ 'a'  │ 1     │ missing │
│ 2   │ 'b'  │ 2     │ missing │
│ 3   │ 'c'  │ 3     │ missing │
│ 4   │ 'd'  │ 4     │ missing │
│ 5   │ 'e'  │ 5     │ missing │
│ 6   │ 'f'  │ 6     │ missing │
│ 7   │ 'x'  │ 7     │ 11      │
│ 8   │ 'y'  │ 8     │ 22      │
│ 9   │ 'z'  │ 9     │ 33      │

or

julia> reduce(vcat, dfs, cols=:union)
9×3 DataFrame
│ Row │ name │ id    │ num     │
│     │ Char │ Int64 │ Int64?  │
├─────┼──────┼───────┼─────────┤
│ 1   │ 'a'  │ 1     │ missing │
│ 2   │ 'b'  │ 2     │ missing │
│ 3   │ 'c'  │ 3     │ missing │
│ 4   │ 'd'  │ 4     │ missing │
│ 5   │ 'e'  │ 5     │ missing │
│ 6   │ 'f'  │ 6     │ missing │
│ 7   │ 'x'  │ 7     │ 11      │
│ 8   │ 'y'  │ 8     │ 22      │
│ 9   │ 'z'  │ 9     │ 33      │
Related