Any suggestions on how to optimize this string -> double parser further, in F#?

Viewed 58

Part of testing, I have to load massive CSV files (200+mb each) and I'm trying to cut down the load time since this is adding up.

As the CSV files are all floating point numbers, I tried to see if I could speed that up. One important point: I do NOT need precision beyond 5 digits after the comma. The code I post here parses it anyways to give a fair comparison to .NET's parser, but any shortcut that preserves that precision is good.

The code I came up with:

open System
open System.Diagnostics

[<EntryPoint>]
let main _ =

    let r = Random()

    let numbers =
        seq {
            for _ in 0 .. 10000000 do
                let n = (1000. * r.NextDouble()) - 500.
                yield (n, sprintf "%.8f" n)
        }
        |> Seq.toList


    let parseString (number: string) : double =
        let len = number.Length
        let rec parse (index: int) (nSign: double) (signMultiplier: double) (nAccumulator: int64) : float =
            if index < len then
                match number.[index] with
                | x when x >= '0' && x <= '9' -> parse (index + 1) (signMultiplier * nSign) signMultiplier (nAccumulator * 10L + int64 x - int64 '0')
                | x when x = '.'              -> parse (index + 1) nSign 0.1 nAccumulator
                | x when x = '-'              -> parse (index + 1) -nSign signMultiplier nAccumulator
                | _                           -> parse (index + 1) nSign signMultiplier nAccumulator
            else
                double nAccumulator * nSign

        parse 0 +1. 1. 0L



    let benchmark name (func: string -> double) =
        let allowedError = 0.00001
        printf "checking %s - " name
        let sw = Stopwatch()
        sw.Start()
        numbers
        |> List.iter (fun (num, str) ->
            let parsed = func str
            let delta = num - parsed
            if abs delta > allowedError then
                failwithf "(%f, %s) not matching %f" num str parsed
        )
        sw.Stop()
        printfn "%i ms" sw.ElapsedMilliseconds


    benchmark ".net parser" (fun x -> double x)
    benchmark "parser1" (fun x -> parseString x)

    0

Is there anything obvious I missed to speed this up?

Additionally, there is something I do not understand: I tried to make the test numbers an array instead of a list and suddenly the processing took longer, while I'd expect the opposite. If anyone has some insights as to why it is happening, I'd be happy to know. You can just change the toList to toArray and replace the List with Array in the benchmark loop to test this.


Edit:

to load the data, here is the code I use:

// load a csv file
let loadCSV (stream: Stream) =
    seq {
        use s = new StreamReader (stream)
        while not s.EndOfStream do
            yield s.ReadLine ()
    }
    |> Seq.map (fun line -> line.Split(",") |> Array.toList)
    |> Seq.toList
0 Answers
Related