UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 10 error while creating data flow template

Viewed 230

Can someone please help me with the error I am getting while running pipeline locally to create a data flow template.

I am trying to load data from big query table in a CSV file and while running the pipeline locally you need to use Python 2.7.

RuntimeError: UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 10: ordinal not in range(128) [while running 'CSV format']

I tried options such as adding below in my python script

import sys
reload(sys)
sys.setdefaultencoding('utf8')

also below before running the python script to create data flow template, I also ran below.

from __future__ import absolute_import
import apache_beam as beam
import argparse
import pickle
import logging
#import sys
#reload(sys)
#sys.setdefaultencoding('utf8')
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io.gcp.internal.clients import bigquery
from datetime import date
today = date.today()
current_date = today.strftime("%Y%m%d")
def createFile(argv=None):
    parser = argparse.ArgumentParser()
    known_args, pipeline_args = parser.parse_known_args(argv)
    p = beam.Pipeline(options=PipelineOptions(pipeline_args))
    (p | 'ReadTable' >> beam.io.Read(beam.io.BigQuerySource(query="SELECT CUST_ROLE,CUST_ROLE_TXT,CUST_C4C_CUSTID,CUST_FNAME,CUST_LNAME,CUST_EMAIL,CUST_MOBILE,DL_BILL_TO_ADD,DL_SHIP_TO_ADD,CUST_STATE,CUST_GENDER_TXT,CUST_LOCALITY,CUST_GROUP,CUST_GROUP_TXT,CUST_DOB,CUST_SHIP_TO_EMAIL,CUST_CITY,CUST_WEDDING_ANNIVERSARY,CUST_PINCODE,CUST_UUID,CUST_ENTITY_ID,CONSENT_EMAIL,CONSENT_WHATSAPP,CONSENT_MOBILE FROM `whr-asia-datalake-nonprod.WHR_DATALAKE.CONSUMER_EXTRACT`",use_standard_sql=True))
       | 'read values' >> beam.Map(lambda x: x.values())
       | 'CSV format' >> beam.Map(lambda row:'|'.join(['"'+ str(column) +'"' for column in row]))
       | 'Write_to_GCS' >> beam.io.WriteToText('gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master'+''+ str(current_date),file_name_suffix='.csv',header='CUST_ROLE|CUST_ROLE_TXT|CUST_C4C_CUSTID|CUST_FNAME|CUST_LNAME|CUST_EMAIL|CUST_MOBILE|DL_BILL_TO_ADD|DL_SHIP_TO_ADD|CUST_STATE|CUST_GENDER_TXT|CUST_LOCALITY|CUST_GROUP|CUST_GROUP_TXT|CUST_DOB|CUST_SHIP_TO_EMAIL|CUST_CITY|CUST_WEDDING_ANNIVERSARY|CUST_PINCODE|CUST_UUID|CUST_ENTITY_ID|CONSENT_EMAIL|CONSENT_WHATSAPP|CONSENT_MOBILE'))
    p.run().wait_until_finish()
if __name__ == '__main__':
    logging.getLogger().setLevel(logging.INFO)
    createFile()

Any suggestion please? export PYTHONIOENCODING=utf8

The error I am getting is as below

"/home/radhika_sharma_ibm/Consumer_Extract.py", line 20, in <lambda> RuntimeError: UnicodeEncodeError: 'ascii' codec can't encode character u'\xc3' in position 8: ordinal not in range(128) [while running 'CSV format']

My python script has below code

0 Answers
Related