how to store 2d array in pandas df and read it without it turning into strings

user_3983454 0
Viewed 275

I have a df where a column is array where each cell is 1*50 dimension and there are 20 rows.

import pandas as pd
df = pd.DataFrame(zip(list(range(0, 20, 1)), np.random.rand(20, 50)),
             columns=['id', 'array'])

At this point there is no issue to use the array column for any array operations (addition, multiplication, division etc) with other array.

But if one saves the df as csv and reads it in another notebook (which I don't have a good way to demo here), each cell in the array column turns into list wrapped strings and using either ast literal_eval or to_numpy doesn't help.

'[1.2 -2.3 2.1 ... 4.1]'

How to prevent array turning into strings here?

pandas arrays python
1 Answers

For me the usage of converters in pandas.read_csv works. The converters get a lambda function, which strips each string, and removes the linebreaks. Afterwards, I can apply np.fromstring with space as seperator.

pd.read_csv("file.csv",converters={"array":lambda x: np.fromstring(x.strip("][").replace("\n", ""), sep=" ")})
user_66538420
Related