I have 3 cascading pipes (one to join against other two) described as follows,
- LHSPipe - (larger size)
- RHSPipes - (smaller size that could possibly fit to memory)
Psuedocode as follows, This example involves two joins
IF F1DecidingFactor = YES then Join LHSPipe with RHS Lookup#1 BY (LHSPipe.F1Input = RHS Lookup#1.Join#F1) and set the lookup result (SET LHSPipe.F1Output = Result#F1) Otherwise SET LHSPipe.F1Output = N/A
The same logic applies for F2 computation.
The expected output,
This scenario forced me to go with Custom Join operation as IF-ELSE decides whether to Join or not.
Considering the above scenario, I would like to go for MAP-SIDE join (keeping RHSPipe in memory of MAP task node), I was thinking of the below possible solutions, each has its pros and cons. Need your suggestions on these.
Option#1:
CoGroup - We can build custom join logic using CoGroup with BufferJoiner followed by custom join (operation), but that wouldnt ensure MAP-SIDE join.
Option#2:
HashJoin - It ensures MAP-SIDE join, but as far as I see custom join cannot be built using this.
Please correct my understanding and suggest your opinions to work on this requirement.
Thanks in advance.