So, I’m having a lot of trouble writing a program to make a covariance matrix from several large integer vectors, stored in separate files. I started by writing
mean xs = realToFrac (sum xs) / realToFrac (length xs)
cov xs ys = mean (zipWith (*) xs ys) - mean xs * mean ys
covmat vectors = [cov xs ys | ys <- vectors, xs <- vectors]
which worked for small input, but you can see how inefficient even just “mean” is. It keeps all the xs in memory when doing the sum, because they're going to be used by "length". That’s fixable, as so:
mean xs = realToFrac thisSum / realToFrac thisLength
where (thisSum, thisLength) = foldl' (\(s,l) y-> (s+y,l+1)) (0,0) xs
but then I run into the same problem with “cov”. When I rewrote "cov" in this style, it didn't end up using my "mean" function. And I still have one level to go up when I write the "covmat" function, which will become extremely complicated.
So, I have two goals, which seem to be in conflict:
Traverse each list once, without keeping it in memory
Break down "covmat" into simpler, meaningful functions, specifically "cov" and "mean"
I don't see any way to unify these two goals with what I know of Haskell. But conceptually it seems simple: all of these functions need to "listen" to the values of the same few lists as they come in. Is there any way in Haskell to organize it like this? If this requires a different data structure or an additional library, I'm open to that.