Problem publishing JSON output to PubSub. A 'u' is being added to every field

user_10604442 0
Viewed 769

I have a script that pulls from the city if Chicago and grabs a json file and then published to Pub Sub. Once the data gets into pub sub I have a dataflow template that pulls the data into Google Big Query. The final data move to BQ is failing and when I print the output in the script. I am getting a u' in front of all the fields which I believe is messing up the field match. Has anyone else had this issue and know what is wrong with my code and how to possibly remove the 'u'. I have tried multiple fix but none of them have worked. A sample out put is listed below:

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals
from sodapy import Socrata
import json
from io import StringIO
from google.oauth2 import service_account
from oauth2client.client import GoogleCredentials
from google.cloud import pubsub_v1
import time
import datetime
import urllib
import urllib.request
import argparse
import base64

credentials = GoogleCredentials.get_application_default()
# change project to your Project ID
project="xxxx"
# change topic to your PubSub topic name
topic="xxxx"


res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')
res_body = res.read()
traffic=json.loads(res_body)
publisher = pubsub_v1.PublisherClient()
topicName = 'projects/' + project + '/topics/' + topic
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project,topic)
for key in traffic:
        publisher.publish(topicName,str.encode(str(key)))
        print(key.items())
google-cloud-pubsub publish-subscribe google-bigquery python json
1 Answers

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']
[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'
True

Now when you say "I am getting a u' in front of all the fields which I believe is messing up the field match." Where do you see this? Is this part of the Python code, or do you see these u' show up on the BigQuery web UI too?

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"
'a'
>>> from __future__ import unicode_literals
>>> "a"
u'a'
user_1324380
Related