Using pandas to save csv files with extended ascii separators

user_12339133 0
Viewed 387

I have a file that is riddled with almost all basic ascii characters, and therefore, to convert it to a meaningful csv to be imported by our database, I have changed my computers default list seperator to ¬ (alt+170).

If I use excel to open an xlsx, and then save as a csv, the program does fine. However, if I read a xlsx or csv into pandas, then call (the code below), it creates a file with two characters in place of ¬.

dataframe.to_csv(filepath, sep = '¬', index = False)

Opening these files as ¬ separated does not work and editing them in notepad shows me extended ascii character 182, as seen here:
https://theasciicode.com.ar/extended-ascii-code/letter-a-circumflex-accent-ascii-code-182.html

How can I get it to save extended ascii properly using pandas?

pandas ascii python
1 Answers

use the python unicode characters to solve this issue

import pandas as pd 
df = pd.DataFrame({'column_1':['a','a','a','a','a','a','b','b','b','b','b','b','b','b','b','c','c','d','d']
                   ,'column_2':[1,1,1,2,2,2,1,1,1,2,2,2,3,3,3,1,2,2,3]
                  })

df.to_csv('test.csv', sep = u'\xac', index = False)

Now here the u'\xac' is the unicode for the logical not operator

and to read it again

df2 = pd.read_csv('test.csv', sep = u'\xac')

This should fix your problem as you have a data frame that you can read and write using the character you need, this is all in the UTF-8 encoding. If you want to use another encoding it is up to you.

BTW this is where I get my encoding mappings https://www.fileformat.info/info/unicode/char/00ac/index.htm

user_54783730
Related