Apache_beam[gcp] - GroupBy result to seperate folder(GCP)

Viewed 36

Input data needs to be segregated based on particular column and store in seperate folders in GCP. In below code snippet, planning to write to GCP from writeEachGroupToGCP class. Tried creating a child pipeline from this class, or write directly to gcp, all are failing. if there is any better option, please suggest

InputData: Employee data with columns (ID, Ename, HireDate, ManagerName)

class writeEachGroupToGCP(DoFn):
  def process(self, data,gcp_out_prefix):
    (partition, tble)=data
    #TODO: Write to GCP, gcp_out_prefix+"/Manager={}".format(partition)
    
    return data

p1 = beam.Pipeline()

(
    p1
     | beam.io.ReadFromText("indata/dept_data.txt")
     | beam.Map(lambda x: str(x).split(","))
     | beam.Map(lambda x: (x[3], x)) 
     | beam.GroupByKey()
     | beam.ParDo(writeEachGroupToGCP())
)
p1.run()
0 Answers
Related