The following functions parallelize the processing of a list by first decomposing a list into large chunks and then processing each chunk.
let chunkList chunkSize (xs : list<'T>) =
query {
for idx in 0..(xs.Length - 1) do
groupBy (idx / chunkSize) into g
select (g |> Seq.map (fun idx -> xs.[idx]))
}
let par (foo: 'T -> 'S) (xs: list<'T>) =
xs
|> List.map (fun x -> async { return foo x })
|> Async.Parallel
|> Async.RunSynchronously
|> Array.toList
let parChunks chunkSize (f: 'T -> 'S) (xs: list<'T>) =
chunkList chunkSize xs |> Seq.map List.ofSeq |> List.ofSeq
|> par (List.map f)
|> List.concat
This function was used to test parChunks:
let g i = [1..1000000] |> List.map (fun x -> sqrt (float (1000 * x + 1))) |> List.head
Running the standard List.Seq and `parChunk`` with a chunk size equal to 1/2 the list size there was a performance gain:
List.map g [1..100];; // Real: 00:00:28.979, CPU: 00:00:29.562
parChunks 50 g [1..100];; // Real: 00:00:23.027, CPU: 00:00:24.687
However, with a chunk size equal to 1/4 the size of the list the performance was almost the same. I did not expect this since my processor (Intel 6700HQ) has four cores.
parChunks 25 g [1..100];; // Real: 00:00:21.695, CPU: 00:00:24.437
Looking at the Performance app in Task Manager one sees that the four cores are never used.
Is there a way to make all four cores participate in this computation?