apache beam with python pardo with multi pCollections input

user_13512325 0
Viewed 659

I'm new in apache_beam and I'm trying to develop a pipeline. I have 2 pCollections with same format and I have another pCollection with other format. I try to do a ParDo function that for each item in pCollection 3 depending a value o this item search if item exist in pCollection 1 or 2 to complete output with info of pCollection 1 or 2. But I dont know how do it this ParDo function.

This is my code:

output = (
      pCollection1, pCollection2, pCollection3
      | 'ParDo function' >> beam.ParDo(SearchData()))

And this is my ParDo fucntion:

class SampleScores(beam.DoFn):
    def process(self,element):
      
      # here I don't know how call a collection because I have only a "element"

      return xxx

thanks

apache-beam python
1 Answers

Solved.

Have you looked at Side inputs? beam.apache.org/documentation/programming-guide/#side- inputs If I understand your question correctly, what you want is to have process(self, element, pcoll1, pcoll2), side inputs might help you with that. – Milan Cermak yesterday

Thanks @MilanCermak

user_135123250
Related