Python converting tweepy tweet and user IDs to scientific notation

user_1667034 0
Viewed 634

I was trying to do some data mining with twitter and ran into this problem. Python keeps converting the tweet's large numerical id's to scientific notation when I try to write it to a CSV file. For example, if the ID was 9381435503399854 python will convert it to 9.381435503399854E+17. I tried using format(int(tweet.id), ".0f") but it is giving me the same result. Format(int(tweet.id), "f") seems like it works but appends ".000000" to the end of the id. Any suggestions would be greatly appreciated. Here is some sample code:

writeExtended(count, tweet.id, tweet.full_text.encode('utf8'), tweet.display_text_range, tweet.created_at)

def writeExtended(id, idstr, full_text, display_text_range, created_at):
    #Write Extended tweet details to CSV file
    with open('Extended.csv', mode='a+') as employee_file:
        employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
        employee_writer.writerow([id,idstr, full_text, display_text_range, created_at])
twitter csv python-3.x python variables
2 Answers

My guess is your script is working perfectly and that the effect you are seeing (i.e. conversion to scientific format) is a result of the conversion of the CSV file into Excel (or other spreadsheet application). You should try opening the CSV file in a text editor such as Notepad.

As a slightly bizarre workaround, you can convert it to text and prefix a tab character to it. This should stop the conversion:

def writeExtended(id, idstr, full_text, display_text_range, created_at):
    #Write Extended tweet details to CSV file
    with open('Extended.csv', mode='a+', newline='') as employee_file:
        employee_writer = csv.writer(employee_file)
        employee_writer.writerow([id, '\t{}'.format(idstr), full_text, display_text_range, created_at])

writeExtended(count, tweet.id, tweet.full_text.encode('utf8'), tweet.display_text_range, tweet.created_at)
user_49857330

MS Excel, Libre Office and Google Spreadsheets import numeric-only cells as the number, so if it is a long string of a number (e.g. tweetid) then they convert it to scientific notation.

A workaround solution is to append a non-numeric text ( underscore character _ for example) to the number when you output your csv file from Python.

Another solution that works only with Libre Office Calc, when you open a csv file it will pop-up a menu before loading it. So if select the column you want to be treated as text, and then from the menu just above change the "Column type" to "Text", and hit OK, then it will load the column without converting it to scientific notation. See the image below for visual illustration.

enter image description here

user_22341610
Related