Julia: Streaming transformations of IO/file data

user_3375695 0
Viewed 124

It seems to be a simple question, but I’m not able to figure out how to do it in Julia. We receive text files, but they have ‘\01’ or ‘\00’ as newline. The files are partially large (~10 GB) which is why we’d like to open a stream for file content, map ‘\01’ to ‘\r’ and then continue reading the lines. Something like (pseudo code):

open("myfile.dat", read=true) do fd
    fd = replace(fd, '\01' => '\r')
    for line in eachline(fd)
        ...
    end
end

Preferably it should work with input from files, cmds and IOBuffer.

I know we could invoke an external command to do the translation, but we are using Windows as well as Linux, and we’d like an approach that easily works on both.

Given this is a a concret use cases, we are wondering how IO stream processing generally works in Julia. Is there anything available already that is easy to use?

Thanks a lot for your help

julia io
1 Answers

If '\01' is the only delimiter, you can user readuntil:

open("foo.txt"; read=true) do fd
    while !eof(fd)
        x = readuntil(fd, '\01')
        @show x
    end
end

If you have multiple delimiters, you can use a modified version of Julia's readuntil:

function multireaduntil(s::IO, delims::Vector{<:AbstractChar}; keep::Bool=false)
    out = IOBuffer()
    for c in readeach(s, Char)
        if c in delims
            keep && write(out, c)
            break
        end
        write(out, c)
    end
    return String(take!(out))
end

which you can use like this:

open("foo.txt"; read=true) do fd
    while !eof(fd)
        x = multireaduntil(fd, ['\00', '\01'])
        @show x
    end
end

If you need more advanced parsing, I would suggest using something like TextParse.jl

user_70771170
Related