We are working with spark 1.6 and we are trying to keep global identity for similar events. There can be few "groups"of events with identical ID (in the example as number. letters are added just for uniqueness). And we know that some of these events are similar so we are able to connect them. We want to keep something like:
Z -> 1, 2, 3
X -> 4
so in a future if some events with id 4 will come we can assign X as a global identity.
Please check example for better illustration:
Let's say we have some streaming data coming into spark job.
1a
1b
2c
2d
2e
3f
3g
3h
4i
Since event 1 is our first appearance we want to assign 1 to Z.
Next what we know is that 1b and 2c are similar. so we want to keep somewhere 2->1 mapping. Same thing is for 2e and 3f so we need mapping 3-2. So for now we have 3 pairs 1->Z, 2->1, 3->2.
And we want to create "historical" path: Z <- 1 <- 2 <- 3
At the end we will have all events with ID = Z.
1a -> Z
1b -> Z
2c -> Z
2d -> Z
2e -> Z
3f -> Z
3g -> Z
3h -> Z
4i -> X
We tried to use mapwithstate but only thing we were able to do was that 2->1 and 3->2. With mapwithstate we were not able to get state for "parent" in state for current event - eg. current event 3 with parent 2 and not able to get 2 -> 1 and neither 1 -> Z.
Is it possible to have some global mapping for this? We already tried accumulators and broadcast but looks like not very suitable. And we were not able to replace events 1 for first mapping and events 2 for second mapping with Z.
If new event 5 will come and it is similar with 3h for example we need to assign mapping 5->Z again.