process a large file via multithreading

Viewed 2099

There is a quite large file (>10G) on the disk, each line inside the fie is composed of a line-number and a person's name, like this:

1 Jane
2 Perk
3 Sime
4 Perk
.. ..

I have to read this large file, and find the frequency of each name, finally output the results in descending order of each name's frequency, like this:

Perk 2
Jane 1
Sime 1

As the interviewer requested, the above job should be done as efficiently as possible, and multithreading is allowed. And my solution is something like this:

  1. Because the file is too large, I partition the file into several small files, each small file is about 100M, via lseek I can locate the begin and the end of each small file (beg, end);

  2. For these small files, there is a shared hash-map using person's name as key and how many times it shows so far as value;

  3. For each small file, there is a single thread go through it, every time the thread encounters a person's name, it will increment its corresponding value in the shared hash-map;

  4. When all threads finish, I think it's time to sort the hash-map according to the value field.

But because there might be too many names in that file, so the sorting would be slow. I didn't come up with a good idea about how to output the names in descending order.

Hope anyone can help me with the above problem, give me a better solution on how to do the job via multithreading and the sorting stuff.

5 Answers
Related