Input data needs to be segregated based on particular column and store in seperate folders in GCP. In below code snippet, planning to write to GCP from writeEachGroupToGCP class. Tried creating a child pipeline from this class, or write directly to gcp, all are failing. if there is any better option, please suggest
InputData: Employee data with columns (ID, Ename, HireDate, ManagerName)
class writeEachGroupToGCP(DoFn):
def process(self, data,gcp_out_prefix):
(partition, tble)=data
#TODO: Write to GCP, gcp_out_prefix+"/Manager={}".format(partition)
return data
p1 = beam.Pipeline()
(
p1
| beam.io.ReadFromText("indata/dept_data.txt")
| beam.Map(lambda x: str(x).split(","))
| beam.Map(lambda x: (x[3], x))
| beam.GroupByKey()
| beam.ParDo(writeEachGroupToGCP())
)
p1.run()