Loading a CSV in memory with Cassava

Question

I am trying to load a CSV into memory as a Vector of Vector with Cassava. My program does work but uses a huge amount of memory for a 50MB csv file and I don't understand why.

I know that working with Data.Csv.Streaming should be better for big files, but I thought 50MB would still be ok. I tried both Data.Csv and Data.Csv.Streaming with more or less canonical examples from the github project page, I also tried to implement my own parser that outputs Vector of Vector (I based my code on attoparsec-csv https://hackage.haskell.org/package/attoparsec-csv), and all these solutions use about 2000MB of memory! I am sure there is something wrong in what I am doing. What is the right way of doing this?

My final goal is to have the data fully loaded into memory for further processing later on. For example, I could split my data into interesting matrices and work with those using Hmatrix.

Here are the 2 programs I tried with Cassava:

1/ Using Data.Csv

import qualified Data.ByteString.Lazy as BL
import qualified Data.Vector as V
import Data.Csv
import Data.Foldable


main = do
   csv <- BL.readFile "train.csv"
   let Right res = decode HasHeader csv :: Either String (V.Vector(V.Vector(BL.ByteString)))
   print $ res V.! 0

2/ Using Data.Csv.Streaming

{-# LANGUAGE BangPatterns #-}

import qualified Data.ByteString.Lazy as BL
import qualified Data.Vector as V
import Data.Csv.Streaming
import Data.Foldable


main = do
   csv <- BL.readFile "train.csv"
   let !a = decode HasHeader csv :: Records(V.Vector(BL.ByteString))
   let !res = V.fromList $ Data.Foldable.toList a
   print $ res V.! 0

Note that I don't give you the program I made based on attoparsec-csv because it is almost exactly the same with Vector instead of List. The memory usage of that solution is still quite poor.

Interestingly, in the Data.Csv.Streaming solution, if I simply print my data using a Data.Foldable.for_, everything is super fast with a 2MB memory usage. This made me think that my problem is linked to the way I construct my Vector. Probably accumulating thunks instead of stacking raw data into a compact data structure.

Thank you for your help,

Antoine

Loading a CSV in memory with Cassava

1 Answers