Julia replace nested for-loops by comprehensions

Viewed 119

Sorry, but I am a complete beginner in Julia. I have tried unsuccessfully to replace a nested for loop with a comprehension in Julia:

dfc is a 12x8 DataFrame, empty_rows = Vector{Int64}(). The following comprehension

[[push!(empty_rows,j) for j in 1:12] for i in 1:8 if ismissing(dfc[j,i])]

got the error: ERROR: UndefVarError: j not defined

I would very much appreciate your help in solving the problem

2 Answers

I understand that you have a dataframe like below:

julia> df = DataFrame(rand([missing;1:8], 12, 8),:auto)
12×8 DataFrame
 Row │ x1      x2      x3      x4       x5       x6       x7      x8
     │ Int64?  Int64?  Int64?  Int64?   Int64?   Int64?   Int64?  Int64?
─────┼────────────────────────────────────────────────────────────────────
   1 │      5       7       3        6        7        8       2        8
   2 │      8       2       7        3        5        8       6        6
   3 │      2       2       4        7        4        3       4        7
   4 │      8       2       5        7        6        6       5        2
   5 │      1       3       1  missing        8        3       6        1
   6 │      2       8       7        2        3        3       8        3
   7 │      1       2       8        8        6  missing       2        4
   8 │      2       1       1        6        2  missing       4  missing
   9 │      3       2       8        8        6        2       8        5
  10 │      2       7       6        2        4        5       1        2
  11 │      3       7       5        2  missing        4       4        8
  12 │      2       4       5  missing        7        1       7        8

and you want to find rows with a missing value:

julia> findall(==(1), any(ismissing.(collect(row))) for row in eachrow(df))
5-element Vector{Int64}:
  5
  7
  8
 11
 12

Or even shorter (the result is the same):

findall(==(1),completecases(df))

I think it looks like you want to get a list of rows that don't have missing elements. I am not even sure that a list is what you really want, though.

For instance if you want a data frame with just complete rows, you can use a single list comprehension to generate indexes of good rows and then use that list of indexes to directly create a new dataframe. Like this:

julia> df = DataFrame(rand(20,4), :auto)
20×4 DataFrame
 Row │ x1        x2         x3        x4        
     │ Float64   Float64    Float64   Float64   
─────┼──────────────────────────────────────────
   1 │ 0.71893   0.505219   0.950092  0.444428
   2 │ 0.925687  0.0442746  0.159913  0.176289
   3 │ 0.927264  0.504218   0.661961  0.858979
   ...
  19 │ 0.233894  0.585126   0.509326  0.164148
  20 │ 0.734157  0.447377   0.238563  0.118466

julia> index = [rownumber(row) for row in eachrow(df) if all([x ≥ 0.05 for x in row])]
17-element Vector{Int64}:
  1
  3
  5
...
 19
 20

julia> df[index,:]
17×4 DataFrame
 Row │ x1        x2         x3        x4        
     │ Float64   Float64    Float64   Float64   
─────┼──────────────────────────────────────────
   1 │ 0.71893   0.505219   0.950092  0.444428
   2 │ 0.927264  0.504218   0.661961  0.858979
   3 │ 0.49163   0.330992   0.32438   0.325456
...
  16 │ 0.233894  0.585126   0.509326  0.164148
  17 │ 0.734157  0.447377   0.238563  0.118466

Note that I changed your ismissing to testing for small elements. The ideas are the same, of course.

Now if you really wanted a list of lists corresponding to rows, we can do that easily enough. Keep in mind, though, that a row of a dataframe is not a list. This is because DataFrames can be all kinds things under the covers and it wouldn't do to change the type too soon. Here is how to get a list of rows (each expressed as a list):

julia> [[x for x in row] for row in eachrow(df) if all([x ≥ 0.05 for x in row])]
17-element Vector{Vector{Float64}}:
 [0.7189299510308665, 0.5052191295159212, 0.9500917221166186, 0.44442775929136]
 [0.9272644252338568, 0.5042178110347096, 0.6619606618026813, 0.8589785313767418]
 [0.4916295966201112, 0.330991690382628, 0.32438004671565834, 0.32545637195862265]
...
 [0.23389358731823695, 0.5851259568533834, 0.5093258068548991, 0.1641483942765276]
 [0.7341572234303397, 0.4473771697876918, 0.2385629812047122, 0.11846580671284723]

Note that you can't broadcast over a row. If you could the condition in these expression could be simplified to all(x .≥ row). That would avoid a fair bit of wasted allocation since the optimizer can often see that you don't really need to allocate any space. The natural way to do this in a loop can avoid this allocation:

julia> index = []
Any[]
julia> for row in eachrow(df)
           flag = true
           for x in row
               if x < 0.05
                   flag = false
                   break
               end
           end
           if flag
               push!(index, rownumber(row))
           end
       end

julia> df[index,:]
17×4 DataFrame
 Row │ x1        x2         x3        x4        
     │ Float64   Float64    Float64   Float64   
─────┼──────────────────────────────────────────
   1 │ 0.71893   0.505219   0.950092  0.444428
   2 │ 0.927264  0.504218   0.661961  0.858979
   3 │ 0.49163   0.330992   0.32438   0.325456
...
  16 │ 0.233894  0.585126   0.509326  0.164148
  17 │ 0.734157  0.447377   0.238563  0.118466

Obviously, this is waay less fashionable than fancy comprehensions. But it is easy to code and easy to get right.

I hope this helps.

Related