Julia: segfault when assigning DataFrame column while using threads

Question

The following code creates a segfault for me - is this a bug? And if so, in which component?

using DataFrames
function test()
    Threads.@threads for i in 1:50
        df = DataFrame()
        df.foo = 1
    end
end
test()

(need to start Julia with multithreading support for this to work, eg JULIA_NUM_THREADS=50; julia)

It only generates a segfault if the number of iterations / threads is sufficiently high, eg 50. For lower numbers it only sporadically / never does so.

My environment:

julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 50

user_1269567 · Answer

It is most likely caused by the fact that you are using deprecated syntax so probably something with deprecation handling messes up things (I do not have enough cores to test it).

In general your code uses deprecated syntax (and produces something different than you probably expect):

~$ julia --depwarn=yes --banner=no
julia> using DataFrames

julia> df = DataFrame()
0×0 DataFrame


julia> df.foo=1
┌ Warning: `setproperty!(df::DataFrame, col_ind::Symbol, v)` is deprecated, use `df[!, col_ind] .= v` instead.
│   caller = top-level scope at REPL[3]:1
└ @ Core REPL[3]:1
1

julia> df # note that the resulting deprecated syntax has added the column but it has 0 rows
0×1 DataFrame


julia> df2 = DataFrame()
0×0 DataFrame

julia> df2.foo = [1] # this is a correct syntax - assign a vector
1-element Array{Int64,1}:
 1

julia> df2[:, :foo2] .= 1 # or use broadcasting
1-element Array{Int64,1}:
 1

julia> insertcols!(df2, :foo3 => 1) # or use insertcols! which does broadcasting automatically, see the docstring for details
1×3 DataFrame
│ Row │ foo   │ foo2  │ foo3  │
│     │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┤
│ 1   │ 1     │ 1     │ 1     │

The reason why df.foo = 1 is disallowed and df.foo = [1] is required follows the fact that, as opposed to e.g. R, Julia distinguishes scalars and vectors (in R everything is a vector).

Going back to the original question something e.g. like this should work:

using DataFrames
function test()
    Threads.@threads for i in 1:50
        df = DataFrame()
        df.foo = [1]
    end
end
test()

please let me know if it causes problems or not. Thank you!

Julia: segfault when assigning DataFrame column while using threads

1 Answers