Understanding the behavior of the colon in julia DataFrames.select()

user_6691780 0
Viewed 72

I have some data with many rows, that I want to reorder, and in some cases rename. Because of the number of columns I wouldn't want to select and rename every single on of them. But when I use the : operator to select the remaining columns I got a result that I did not expect. The columns that I renamed are included twice:

julia> data = [2  1  3  50
               52 51 53 100]

julia> names = ["col 2","col 1", "col_3", "col_50"]

julia> df = DataFrame(data, names)
2×4 DataFrame
 Row │ col 2  col 1  col_3  col_50 
     │ Int64  Int64  Int64  Int64  
─────┼─────────────────────────────
   1 │     2      1      3      50
   2 │    52     51     53     100


julia> select(df, "col 1" => :col_1, "col 2" => :col_2, :)
2×6 DataFrame
 Row │ col_1  col_2  col 1  col 2  col_3  col_50 
     │ Int64  Int64  Int64  Int64  Int64  Int64  
─────┼───────────────────────────────────────────
   1 │     1      2      1      2      3      50
   2 │    51     52     51     52     53     100

I was hoping for/expecting this

julia> select(df, "col 1" => :col_1, "col 2" => :col_2, :)
2×6 DataFrame
 Row │ col_1  col_2  col_3  col_50 
     │ Int64  Int64  Int64  Int64  
─────┼────────────────────────────
   1 │     1      2      3      50
   2 │    51     52     53     100

What do I misunderstand about the : operator. Is there a/another way to achieve the transformation I want?

julia dataframe
1 Answers

Turns out there is. Funny how people (I answer my own question here) can focus on using one function while I just could have used rename!() and then reorder them using select!():

julia> rename!(df, "col 1" => :col_1, "col 2" => :col_2)
2×4 DataFrame
│ Row │ col_2 │ col_1 │ col_3 │ col_50 │
│     │ Int64 │ Int64 │ Int64 │ Int64  │
├─────┼───────┼───────┼───────┼────────┤
│ 1   │ 2     │ 1     │ 3     │ 50     │
│ 2   │ 52    │ 51    │ 53    │ 100    │

julia> select!(df, :col_1, :col_2, :)
2×4 DataFrame
│ Row │ col_1 │ col_2 │ col_3 │ col_50 │
│     │ Int64 │ Int64 │ Int64 │ Int64  │
├─────┼───────┼───────┼───────┼────────┤
│ 1   │ 1     │ 2     │ 3     │ 50     │
│ 2   │ 51    │ 52    │ 53    │ 100    │

Or using a Pipe:

julia> using Pipe

julia> @pipe df |>
        rename!(_, "col 1" => :col_1, "col 2" => :col_2) |> 
        select!(_, :col_1, :col_2, :)
2×4 DataFrame
│ Row │ col_1 │ col_2 │ col_3 │ col_50 │
│     │ Int64 │ Int64 │ Int64 │ Int64  │
├─────┼───────┼───────┼───────┼────────┤
│ 1   │ 1     │ 2     │ 3     │ 50     │
│ 2   │ 51    │ 52    │ 53    │ 100    │

Regarding the behaviour of the : operator I have to thank bkamins for providing me with the answer on github

add in the place where : is placed all columns of the source data frame that have not been added to the result; the adding is based on column name (not contents)

Why it works like this: In general we allow for potentially very complex transformations in select etc. - columns can be created, renamed, added in any order. In order to keep the rules simple (so that users can build a correct mental model of what is going on and not too much magic happens) the approach is that columns are processed left to right and are identified by on their name in target data frame.

I agree that in your particular case it seems better to do what you propose, but if you consider a wider context (i.e. that in one select you can have dozens of different transformations combined) keeping the rules consistent without any special cases is I believe better.

I thought I share it here as well.

user_66917800
Related