Mutate lead variables

user_15001838 0
Viewed 46

This is my code:

data = @pipe data |> sort(_, :year) |> groupby(_,:id) |> 
 transform(_, :a1 => (x -> lead(x,1) .- x ) => :b1; ungroup = false) |>
 transform(_, :a2 => (x -> lead(x,2) .- x ) => :b2; ungroup = false) |>
 transform(_, :a3 => (x -> lead(x,3) .- x ) => :b3, ungroup = false) |>
 transform(_, :a4 => (x -> lead(x,4) .- x ) => :b4, ungroup = false) |>
 transform(_, :a5 => (x -> lead(x,5) .- x ) => :b5)

Is this the most efficient way to mutate lead variables on a very large dataset?

data is a DataFrame with columns a1, ..., :a5, and I want to have columns b1 , ..., :b5 where bi = lead(ai, i).

julia
1 Answers

The question about "most efficient" has many dimensions, but I assume you want to avoid as much allocations as possible, in this case do the following:

@pipe data |>
      sort!(_, :year) |>
      groupby(_,:id) |> 
      transform!(_, [Symbol(:a, i) => (x -> lead(x, i) .- x ) => Symbol(:b, i) for i in 1:5])

in this way you will update your data data frame in-place minimizing copying (your solution copies a lot).

user_12695670
Related