Scanning text file for up to N words on each line, filling array with columns and rows of words in Julia. The number of words isn't known in advance

user_10022338 0
Viewed 108

I have a partial solution only

data = split.(readlines(file))
a =  map(x -> x[1],data)
b =  map(x -> x[2],data)
c =  map(x -> x[3],data)
d =  map(x -> x[4],data)
e =  map(x -> x[5],data)
f =  map(x -> x[6],data)

But it works only if all the input lines contain the full expected number of words. Unfortunately, the lines can be shorter and I want to fill the "empty words" with empty string "" or some native Julia goodness for the missing value.

I'm Julia newbie as can be seen.

Thank you

julia
3 Answers

Here is an example how you can do it (I put comments with explanations):

julia> str = """
       a b c
       a b
       a b c d
       a
       a b c d e
       """ # source data
"a b c\na b\na b c d\na\na b c d e\n"

julia> data = split.(readlines(IOBuffer(str))) # use IOBuffer to turn string into file-like object
5-element Vector{Vector{SubString{String}}}:
 ["a", "b", "c"]
 ["a", "b"]
 ["a", "b", "c", "d"]
 ["a"]
 ["a", "b", "c", "d", "e"]

julia> res = fill("", length(data), maximum(length, data)) # pre-fill matrix with "" as you requested - this should probably be easiest for you to work with
5×5 Matrix{String}:
 ""  ""  ""  ""  ""
 ""  ""  ""  ""  ""
 ""  ""  ""  ""  ""
 ""  ""  ""  ""  ""
 ""  ""  ""  ""  ""

julia> foreach(((out, in),) -> copyto!(out, in), zip(eachrow(res), data)) # fill each row of res with consecutive elements of data using copyto!

julia> res # the result you wanted is now in res
5×5 Matrix{String}:
 "a"  "b"  "c"  ""   ""
 "a"  "b"  ""   ""   ""
 "a"  "b"  "c"  "d"  ""
 "a"  ""   ""   ""   ""
 "a"  "b"  "c"  "d"  "e"

If something is unclear please comment. As a result you will have a matrix whose each row is row of your source data.

A bit more advanced pattern for the key operation would be:

foreach(Base.splat(copyto!), zip(eachrow(res), data))

If both are unclear to you you can alternatively do double loop e.g.:

for i in 1:length(data)
    row = data[i]
    for j in 1:length(row)
        res[i, j] = row[j]
    end
end
user_12695670

For minimum allocations, it's better to return a Vector of Vectors instead of a Matrix, and use data[i] as a row instead of data[i,:]. If a Matrix is strictly necessary, you'll need a simple conversion like mapreduce(permutedims, vcat, data) to get a Matrix.

So, build the Vectors directly from lines of the file and later append ""s as required.

function build_array(file)
    data = split.(readlines(file))
    N = maximum(length, data)
    for d in data
        for j = 1:N-length(d)
            push!(d, "")
        end
    end
    data
end

Using the following as input text file:

Scanning text file for 
up to N words on each line, 
filling array with columns 
and rows of words in Julia. 
The number of words isn't 
known in advance.

The output will look like this:

build_array("input.txt")

6-element Vector{Vector{SubString{String}}}:
 ["Scanning", "text", "file", "for", "", "", ""]
 ["up", "to", "N", "words", "on", "each", "line,"]
 ["filling", "array", "with", "columns", "", "", ""]
 ["and", "rows", "of", "words", "in", "Julia.", ""]
 ["The", "number", "of", "words", "isn't", "", ""]
 ["known", "in", "advance.", "", "", "", ""]

And if used mapreduce(permutedims, vcat, data), you'll get:

6×7 Matrix{SubString{String}}:
 "Scanning"  "text"    "file"      "for"      ""       ""        ""
 "up"        "to"      "N"         "words"    "on"     "each"    "line,"
 "filling"   "array"   "with"      "columns"  ""       ""        ""
 "and"       "rows"    "of"        "words"    "in"     "Julia."  ""
 "The"       "number"  "of"        "words"    "isn't"  ""        ""
 "known"     "in"      "advance."  ""         ""       ""        ""
user_39431700

Another option, which could be faster on long files is to regard the text as CSV with " " as a delimiter. As follows:

using CSV, DataFrames

CSV.read("test.txt",DataFrame;delim=" ", header=false, silencewarnings=true,types=String)

This outputs:

6×8 DataFrame
 Row │ Column1   Column2  Column3   Column4   Column5   Column6   Column7   Column8 
     │ String15  String7  String15  String7?  String7?  String7?  String7?  Missing 
─────┼──────────────────────────────────────────────────────────────────────────────
   1 │ Scanning  text     file      for       missing   missing   missing   missing 
   2 │ up        to       N         words     on        each      line,     missing 
   3 │ filling   array    with      columns   missing   missing   missing   missing 
   4 │ and       rows     of        words     in        Julia.    missing   missing 
   5 │ The       number   of        words     isn't     missing   missing   missing 
   6 │ known     in       advance.  missing   missing   missing   missing   missing

on @AboAmmar's sample file.

Because CSV reading is optimized for zero-copy processing, this could potentially be faster than other solutions. But reading the documentation of CSV.read is best before using - and may offer more useful features that you need.

user_35808700
Related