How to create a DataFrame from a list of dictionaries with inconsistent key dimensions in Dask?

user_2461913 0
Viewed 209

Somewhat new to Dask but since most of the operations are lazy, how can I make a basic case like this work at scale?

import dask.dataframe as dd
import dask.bag as db

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
    # etc...
]

dag_data = db.from_sequence(dataset)
dag_data.to_dataframe()

In a Pandas-only world, I could map pd.Series but the problem is these operations aren't calculated until after they are computed. The above code produces a DataFrame with only features from the first record ("a", "b", "c").

Exepcted result:

| a | b | c | d | e | q | x | y | z |
---------------------------------------
| 1 | 2 | 3 | - | - | - | - | - | - |
| 3 | 4 | - | 5 | 5 | - | - | - | - |
| 2 | - | - | - | - | 5 | 1 | 2 | 3 |
dask-dataframe dask python
1 Answers

Please take this as an extended comment rather than an answer.

In pandas only you have

import pandas as pd

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
]

df = pd.DataFrame(dataset)

and df is

   a    b    c    d    e    x    y    z    q
0  1  2.0  3.0  NaN  NaN  NaN  NaN  NaN  NaN
1  3  4.0  NaN  5.0  5.0  NaN  NaN  NaN  NaN
2  2  NaN  NaN  NaN  NaN  1.0  2.0  3.0  5.0

But if you move to dask the only possible solution I found is the following

import pandas as pd
import dask.dataframe as dd
from dask import delayed, compute

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
]

def fun(d):
    return pd.DataFrame(d, index=[0])

lst = [delayed(fun)(l) for l in dataset]

df = dd.concat(compute(lst)[0])

This doesn't seem to me efficient at all. It will be interesting to see if there is a proper way to obtain the same output.

user_48193760
Related