Beam Python SDK: pd.merge left join error (valueError: Attempted to encode null for non-nullable field)

user_16076481 0
Viewed 326

I am using beam python SDK to read two CSV files using apache_beam.dataframe.io.read_csv library.

subscriber_data = (p | 'Read Subscriber File' >> read_csv('subscriber_data.csv'))

address_data = (p | 'Read address File' >> read_csv('address_data.csv'))

then trying to merge these files using pd.merge left join using the below code

subscriber_address_df = subscriber_data.merge(address_data.set_index('address_id').state,
                                                         right_index=True,
                                                         left_on='address_id',
                                                         how='left')

once I tried to print the subscriber_address_df using the below code

to_pcollection(subscriber_address_df , include_indexes=False) | beam.Map(print)

I got this error:

ValueError: Attempted to encode null for non-nullable field "state". [while running 'Unbatch 'merge_DataFrame_2137267343520'/ParDo(_UnbatchNoIndex)']

How can I solve this error?

My understanding is, this error happened because of null values generated from left join in non-nullable column state in the data frame schema.

I tried to switch the dataframe to Pcollection using to_pcollection and assign schema with nullable columns for that pcollection using .with_output_types then switch back to dataframe to_dataframe, but it did not work

apache-beam python
1 Answers

This looks like a bug. I filed BEAM-12587. In the meantime you can work around this by doing to_pcollection(..., yield_elements='pandas') which will result in a PCollection of Pandas Dataframe objects, which you can then split in to rows using your own logic.

user_5823330
Related