I have a high number of files, all in the same structure. In the first line of every file is a key (in this example the key of a movie), followed by the records of user-id, rating and date.
Example file 1:
1:
1488844,3,2005-09-06
822109,5,2005-05-13
885013,4,2005-10-19
Example file 2:
2:
2059652,4,2005-09-05
1666394,3,2005-04-19
1759415,4,2005-04-22
1959936,5,2005-11-21
In order to process the data in pig and get the highest ratings and average rating per movie or year, I would need something like this:
1,1488844,3,2005-09-06
1,822109,5,2005-05-13
1,885013,4,2005-10-19
2,2059652,4,2005-09-05
2,1666394,3,2005-04-19
2,1759415,4,2005-04-22
2,1959936,5,2005-11-21
How can I manage this? Thanks!!!