Pandas Column Consists of Arrays, Plot This Using plt.imshow()

Viewed 3607

So I have data that is structured in a slightly unusual way. Here's an example of what I'm working with:

import numpy as np
import pandas as pd

dict = {'data': [[1,2,3], [2,3,4], [3,4,5]], 'parameter': [10, 11, 12]}
df = pd.DataFrame(dict)

In other words, each row in df['data'] contains an array.

I need to plot this using matplotlib's imshow() function, but I'm having difficulties working with the arrays in df['data']. I've tried using df['data'].to_numpy(), but this returns a dtype=object, which imshow can't handle. I get this error when trying to plot it:

TypeError: Image data of dtype object cannot be converted to float

To be exact, this is what I've tried to run:

import pylab as plt
plt.imshow(df['data'].to_numpy())

I've read around and can't seem to find anyone with a similar example.

To be clear: I need to go from the DataFrame to imshow, I cannot plot directly from the dictionary in my code. I also do not want to do any appending to new lists, as my dataset is large and will slow things down considerably.


EDIT 1:

To answer a question in the comments, this is the type of plot I'm after. The x-axis contains the arrays in df['data'], while the y-axis will ultimately be df['parameter']. enter image description here


EDIT 2:

I think I need to further clarify my question. I'm looking to reproduce this plot exactly.

enter image description here

To make this plot, I did the following:

rows = []
for i in df['data']: 
    rows.append(i)
plt.imshow(rows)

This solution works with my data, but I'm looking for a more efficient way of doing the same thing. i.e., a method that doesn't involve looping and appending.

1 Answers

You could do the following:

use df.explode to unlist the data column to different rows and then plot.

x = df.explode('data').reset_index()
plt.imshow(x[['data','parameter']].to_numpy().astype('float').T)

This will produce:

enter image description here

Basically, x[['data','parameter']].to_numpy().astype('float').T will give you the desired numpy array where you can plot as you wish.

Edit based on user comment:

plt.imshow(np.array(df["data"].values.tolist()).astype('float'))

This will produce:

enter image description here

Related