There is a quite large file (>10G) on the disk, each line inside the fie is composed of a line-number and a person's name, like this:
1 Jane
2 Perk
3 Sime
4 Perk
.. ..
I have to read this large file, and find the frequency of each name, finally output the results in descending order of each name's frequency, like this:
Perk 2
Jane 1
Sime 1
As the interviewer requested, the above job should be done as efficiently as possible, and multithreading is allowed. And my solution is something like this:
Because the file is too large, I partition the file into several small files, each small file is about
100M, vialseekI can locate the begin and the end of each small file(beg, end);For these small files, there is a shared hash-map using person's name as key and how many times it shows so far as value;
For each small file, there is a single thread go through it, every time the thread encounters a person's name, it will increment its corresponding
valuein the shared hash-map;When all threads finish, I think it's time to sort the hash-map according to the
valuefield.
But because there might be too many names in that file, so the sorting would be slow. I didn't come up with a good idea about how to output the names in descending order.
Hope anyone can help me with the above problem, give me a better solution on how to do the job via multithreading and the sorting stuff.