Perform Left, Right and Inner join in Hazelcast Jet

user_3458271 0
Viewed 156

I am working on Hazelcast jet application and I trying to join two Sources using Left, Right or Inner Join but I am stuck at below:

Here is my code:

BatchStage<Map<String,Object>> batch1= pipeline.readFrom(companyListBatchSource);
BatchStage<Map<String,Object>> batch2= pipeline.readFrom(employeeListBatchSource);

//Getting group by key
 BatchStageWithKey<Map<String,Object>, Object> jdbcGroupByKey = batch1.groupingKey(a -> a.getSource1().get(col1));
 BatchStageWithKey<Map<String,Object>, Object> fileGroupByKey = batch2.groupingKey(b -> b.getSource1().get(col2));

//trying to join but not sure what exactly is happening.
BatchStage<Entry<Object, Tuple2<List<Map<String,Object>>, List<Map<String,Object>>>>> d = jdbcGroupByKey.aggregate2(AggregateOperations.toList(),fileGroupByKey,AggregateOperations.toList());

From above code how can achieve data in BatchStage<Map<String,Object>> format? How we can apply different kind of joins here?

hazelcast-jet hazelcast java
1 Answers

Your Map<String, Object> represents a single item, let's call that entire type E. So your input is two streams of type E: companies and employees.

The result of a JOIN isn't of the type E, but, in full generality, Tuple2<List<E>, List<E>>.

From here on the details depend on the cardinalities of each side of the join. If you group by company on both sides, you should have a one-to-many association and the result type Tuple2<E, List<E>>. In this case, use AggregateOperations.pickAny() for the company stream, this will result in a single item and the result of the join will come out as Entry<Object, Tuple2<List<E>, List<E>>>. This is the equivalent of a LEFT OUTER JOIN. You get an inner join from that if you filter out all results where the list of employees is empty: joined.filter(e -> !e.getValue().f1().isEmpty())

user_11038720
Related