Julia data frame - need to select rows based upon multiple columns being missing

user_5323145 0
Viewed 206

Help for the Julia newbie I have joined 2 dataframes and need to select rows that have columns that are missing. The following pulls from one column, but I need to pull multiples. I need to pull :md5 and :md5_1 and :md5_2.... that are missing.

@where(bwjoinout_1_2, findall(x -> (ismissing(x)), :md5)) # works

This pulls rows that have :md5 as missing.

I am syntactically challenged!!

Regards and stay safe Bryan Webb

julia rows dataframe
4 Answers

Not sure I completely understand what you want to do, but would this work for you?

julia> df = DataFrame(id = 1:3, x=[1, missing, 3], y=[1, 2, missing])
3×3 DataFrame
 Row │ id     x        y       
     │ Int64  Int64?   Int64?  
─────┼─────────────────────────
   1 │     1        1        1
   2 │     2  missing        2
   3 │     3        3  missing 

julia> df[ismissing.(df.x) .| ismissing.(df.y), :]
2×3 DataFrame
 Row │ id     x        y       
     │ Int64  Int64?   Int64?  
─────┼─────────────────────────
   1 │     2  missing        2
   2 │     3        3  missing
user_12256070

or

filter(row -> any(ismissing, row[names(df, r"^md5")]), df)

which will leave in df all rows that have a missing value in any of the columns whose name starts with "md5". This is not the most efficient way to do it, but I think it is simplest conceptually.

If you need maximum performance go along what François Févotte proposed, but it currently requires you to explicitly list columns you want to filter on (this PR will allow to make it more cleanly).

user_12695670

used

bwmissows = bwjoinout_1_2[ismissing.(bwjoinout_1_2.md5) .| ismissing.(bwjoinout_1_2.md5_1), :]

worked like a charm pulled rows that had a missing md5 or md5_1

Thanks for your help Stay safe!

i couldn't get the syntax

Regards bryan

user_53231450

I'd do bwjoinout_1_2[.!completecases(bwjoinout_1_2, r"md5"), :].

user_24131790
Related