How to remove double qoutes when reading csv from pandas?

user_9879869 0
Viewed 31

I have a csv and trying to have it as a list of list.

operational_band,alternative_band
V490,null
V550,null
V620,null
V670,null
V708,V710
N740,null
N753,N760
N778,N780
N865,N860

I am reading the file as:

df = pd.read_csv("op_bands.csv")
op_list = df.values.tolist()
print(op_list)

I get op_list as:

[['V620', ' "null"'], 
 ['V670', ' "null"'], 
 ['V708', ' "V710"'], 
 ['N740', ' "null"'], 
 ['N753', ' "N760" '], 
 ['N778', ' "N780"'], 
 ['N865', ' "N860"']]
  1. My 1st and 2nd entry were ignored, why? And how to fix that?
  2. The 2nd entries from each list have double quotes and space e.g. ' "N860"'. How to ignore these quotes and extra space/s?
  3. My CSV format is right, isn't it?
pandas csv python
2 Answers

When I tried to replicate this, by copy-pasting what you wrote into a csv file, I did not have the same problem.

import pandas as pd

df = pd.read_csv('dummy.csv')
op_list = df.values.tolist()
print(op_list)
    
>>>[['V490', nan], ['V550', nan], ['V620', nan], ['V670', nan], ['V708', 'V710'], ['N740', nan], ['N753', 'N760'], ['N778', 'N780'], ['N865', 'N860']]

Looks like an encoding error. Are you sure your input was encoded with utf-8? Here are python's standard encodings: https://docs.python.org/3/library/codecs.html#standard-encodings You can these encodings like:

df = pd.read_csv('dummy.csv', encoding='latin1')
user_28272620

When I copy pasted your data I didn't have your problem. This is my code:

df = pd.read_csv('data.csv', keep_default_na=False)
lsls = df.values.tolist()

use keep_default_na=False to correctly cast 'null' to string and not to nan. Try upgrading to last version of Pandas.

user_100323700
Related