Read cmd to stream to BSON in Julia

user_5224881 0
Viewed 97

I have curl command, whose input I want to load using BSON. For performance reason, I want to read the curl output directly to memory, without saving it to file. Also, I want to close the curl as soon as possible, so I want to read data from curl and then to pass them to BSON, we had some problems when curl was open because it was faster than consecutive parsing.

I know this works, but it keeps curl open for too long, which causes problems when we do this in parallel many times at once and the server where we download from is a bit busy.

using BSON
cmd = `curl <some data>`
BSON.load(open(cmd))

To close cmd ASAP, I have this:

# created IOBuffer to wrap bytes
import BSON.load
function BSON.load(bytes::Vector{UInt8})
    io = IOBuffer()
    write(io, bytes)
    seekstart(io)
    BSON.load(io)
end
cmd = `curl <some data>`
BSON.load(read(cmd))

which works, but I consider it very ugly. Also I'm not sure if this doesn't have some performance penalty.

Is there a more elegant way to do this? Can I read(cmd) into some IO structure, which could be then passed to BSON.load?

I realized the exactly same problem holds for Serialization.deserialize. My solution for deserialization is same, but I welcome any improvoements.

julia buffer io
1 Answers

It's a little unclear what your question means when you say that it "keeps curl open for too long", but here are two different ways to do this:

julia> using BSON

julia> url = "https://raw.githubusercontent.com/JuliaIO/BSON.jl/master/test/test.bson"
"https://raw.githubusercontent.com/JuliaIO/BSON.jl/master/test/test.bson"

julia> open(BSON.load, `curl -s $url`)
Dict{Symbol,Any} with 2 entries:
  :a => Complex{Int64}[1+2im, 3+4im]
  :b => "Hello, World!"

julia> BSON.load(IOBuffer(read(`curl -s $url`)))
Dict{Symbol,Any} with 2 entries:
  :a => Complex{Int64}[1+2im, 3+4im]
  :b => "Hello, World!"

The first version is similar to your first version but closes the curl process immediately when done downloading. The second version reads the result of the curl call into a byte vector, wraps it in an IOBuffer and then calls BSON.load on that.

user_6592480
Related