how to write more robust pipes in julia

user_6691780 0
Viewed 111

It happens quite often that I'm adding a new line to a pipe that throws an error for one reason or another. Normaly this isn't a problem. I fix my code and it works. However, when using pipes, I often have to re-run all the code that created the DataFrame up to the point the pipe starts because the pipe itself changes the DataFrame in such a way it's no valid input anymore.

For example:

df = CSV.read("myfile.csv", DataFrame)

# ----
# all kinds of code working on df
# ----

@pipe df |>
      rename!(_, "A ha" => :Aha) |>  # This works fine the first time
      select!(_, :typo, :)           # throws an error for any reason

@pipe df |>
      rename!(_, "A ha" => :Aha) |>  # Now this throws an error
      select!(_, :fixed_the_typo, :) 
ArgumentError: Tried renaming :A ha to :Aha, when :A ha does not exist in the Index.

Is there a way to either make a pipeline atomic (it either all runs or nothing runs), or write my code in a way that prevents this problem?

I guess what I'm looking for is something like this:

@pipe df |>
      rename(_, "A ha" => :Aha) |> 
      select(_, :typo, :)       |>
      commit!(_)
julia dataframe
1 Answers

The issue is that running each of the piped commands does an in-place modification of the DataFrame. If you instead do

@pipe df |>
      rename(_, "A ha" => :Aha) |>
      select(_, :typo, :)

(notice I omitted the exclamation marks), then instead of modifying the DataFrame df directly in each operation it will create a new version to operate on.

For the exact behavior you asked for you could do

df = @pipe df |>
           rename(_, "A ha" => :Aha) |>
           select(_, :typo, :)

which assigns the result to df when it finishes.

Or, to only create a new DataFrame for the first operation, leave out the exclamation mark for the first operation, and leave it in for all the rest in the pipe:

df = @pipe df |>
           rename(_, "A ha" => :Aha) |>
           select!(_, :typo, :)

Now, in the first operation, a new DataFrame is created, and the same DataFrame is operated on from then on. This will give you the best possible performance while doing what you asked.

user_113249940
Related