I have a dataframe with a column with str and another one with int data
import pandas as pd
from databricks import koalas as ks
df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6, 7],
'b': ['7', '6', '5', '4', '3', '2', '1'],
'letter': ['a', 'a', 'b', 'a', 'b', 'b', 'a']})
kdf = ks.from_pandas(df)
The following operation works in pandas...
df.groupby('letter')[['a', 'b']].apply(lambda x: x.values.tolist())
... but not in koalas :(
kdf.groupby('letter')[['a', 'b']].apply(lambda x: x.values.tolist())
It throws the error:
ArrowInvalid: Could not convert '7' with type str: tried to convert to int64
It is not clear to me why the error and how to solve it. Any idea?
UPDATE:
Simplifying the problem, I notice that kdf.apply(lambda x: [1, 2], axis=1) works, but kdf.apply(lambda x: [1, "2"], axis=1) throw the error. It seems not possible to have a list with elements of different types in koalas dataframe. It is the case? What is an alternative?