Read numpy into pandas dataframe and back to file

user_1912594 0
Viewed 74

I have a binary file that is packed and built as repeated:

struct Record
{
    uint32_t a;
    double   b;
} __attribute__ ((packed));

I'm reading this binary file with python like this:

def read_file(filename):
    dt = np.dtype([('a', np.uint32),
                   ('b', np.float64)])

    data = np.fromfile(filename, dtype=dt)

    df = pd.DataFrame(data=data, columns=dt.names)

    return df

Afterwards, I'm manipulating the data in the dataframe and then would like to save it into a file in the same format.

I'm trying this:

df.to_numpy(dtype=dt).tofile('/tmp/out2.bin')

But it the file is than what I'd expect, because it looks like to_numpy(dtype=dt) with unmodified array does not result in the original array. Here is a reproduceable complete example:

import numpy as np
import pandas as pd
import io

dt = np.dtype([
    ('a', np.uint32),
    ('b', np.float64)
])

data = np.frombuffer(b'\1\2\3\4\x00\x00\x00\x00\x00\x00E@', dtype=dt)
print(data)
df = pd.DataFrame(data=data, columns=dt.names)
print(df)
from_df = df.to_numpy(dtype=dt)
print(from_df)

The first print prints an array of one struct element:

[(67305985, 42.)]

and the second a dataframe

          a     b
0  67305985  42.0

Now I would expect to to_numpy(dtype=dt) operation on unmodified dataframe make an array that equals to the first one, but instead I get something that prints out

[[(67305985, 6.7305985e+07) (      42, 4.2000000e+01)]]

How to convert the Pandas dataframe back to the original one-dimensional array of structs?

pandas numpy c python
1 Answers

I don't know of any good way to get pandas to make a struct dtype.

dt = np.dtype([('a', np.uint32),
               ('b', np.float64)])

df = read_file(...)

You're going to have to create an empty array and populate it yourself.

d = np.empty(df.shape[1], dtype=dt)
d['a'], d['b'] = df['a'], df['b']

d.tofile('/tmp/out2.bin')
user_73409480
Related