I have a dataframe,
df = ( a = 1:0.1:10,
b = 1:0.2:20
)
I would like to operate on the dataframe elements and find the difference of each element with respect to its previous element in columns a and b and store the results in the new columns da and db.
a b da db
1 1 Nan Nan
1.1 1.2 0.1 0.2
. . . .
. . . .
Please suggest a solution to this problem.
Thanks in advance!
You can use ShiftedArrays.jl to create lags and then generate the differenced variables:
using Pkg
Pkg.add("ShiftedArrays")
using ShiftedArrays
using DataFrames
df = DataFrame(a=rand(10),b=rand(10))
df[:,:d_a]= df[:,:a] .- lag(df[:,:a])
df[:,:d_b]= df[:,:b] .- lag(df[:,:b])
I would use ShiftedArrays for this:
julia> using DataFrames, ShiftedArrays
julia> df = DataFrame(a = 1:0.1:1.5, b = 1:0.2:2.0);
julia> df.da = df.a .- lag(df.a); df.db = df.b .- lag(df.b);
julia> df
julia> df
6×4 DataFrame
Row │ a b da db
│ Float64 Float64 Float64? Float64?
─────┼────────────────────────────────────────
1 │ 1.0 1.0 missing missing
2 │ 1.1 1.2 0.1 0.2
3 │ 1.2 1.4 0.1 0.2
4 │ 1.3 1.6 0.1 0.2
5 │ 1.4 1.8 0.1 0.2
6 │ 1.5 2.0 0.1 0.2
If you want NaN instead of missings, you can use lag(df.a, default = NaN).
Note that this also works nicely with grouped data, if you use combine(groupby(df, :groupvar), :a => lag => :a_L1) you will only get lagged values within groups.
Here is another option. Not as efficient as the one above, but showing you what kind of operations you can do in DataFrames.jl:
julia> using DataFrames
julia> df = DataFrame(a = 1:0.1:1.5, b = 1:0.2:2.0);
julia> transform!(df, [:a, :b] .=> (x -> [missing; diff(x)]) .=> [:da, :db])
6×4 DataFrame
Row │ a b da db
│ Float64 Float64 Float64? Float64?
─────┼────────────────────────────────────────
1 │ 1.0 1.0 missing missing
2 │ 1.1 1.2 0.1 0.2
3 │ 1.2 1.4 0.1 0.2
4 │ 1.3 1.6 0.1 0.2
5 │ 1.4 1.8 0.1 0.2
6 │ 1.5 2.0 0.1 0.2