I am very new to hadoop so please bear with me. Any help would be appreciated.
I need to join 2 tables, Table 1 will have pagename , pagerank for eg. Actual data set is huge but with the similar pattern
pageA,0.13
pageB,0.14
pageC,0.53
Table 2, it is a simple wordcount kind of table with word , pagename for eg. actual dataset is huge but with similar pattern
test,pageA:pageB
sample,pageC
json,pageC:pageA:pageD
Now if user searches for any word from second table, I should give him the results of pages based on their pagerank from table 1.
Output when searched for test,
test = pageB,pageA
My approach was to load the first table into distributed cache. Read second table in map method get the list of pages for the word, sort the list using the pagerank info from first table which is loaded into distributed cache. This works for the dataset i am working but wanted to know if there was any better way, also would like to know how can this join be done with pig or hive.