count total number of list elements in pandas column

Viewed 17843

I have a pandas dataframe A with column keywords as (here Im showing only 4 rows but in actual there are millions) :-

 keywords
 ['loans','mercedez','bugatti']
 ['trump','usa']
 ['galaxy','7s','canon','macbook']
 ['beiber','spiderman','marvels','ironmen']

I want to sum total number of list elements in column keywords and store it into some variable. Something like

total_sum=elements in keywords[0]+elements in keywords[1]+elements in 
          keywords[2]+elements in keywords[3]

total_sum=3+2+4+4
total_sum=13

How I can do it in pandas?

7 Answers

IIUC

Setup

df = pd.DataFrame()
df['keywords']=[['loans','mercedez','bugatti'], 
                ['trump','usa'], 
                ['galaxy','7s','canon','macbook'], 
                ['beiber','spiderman','marvels','ironmen']]

Then juse use str.len and sum

df.keywords.str.len().sum()

Detail:

df.keywords.str.len()

0    3
1    2
2    4
3    4
Name: keywords, dtype: int64

Ps: If you have strings that look like a list, use ast.literal_eval to convert to list first.

df.keywords.transform(ast.literal_eval).str.len().sum()

Using sum and map:

sum(map(len, df.keywords))

Sample

df = pd.DataFrame({
    'keywords': [['a', 'b', 'c'], ['c', 'd'], ['a', 'b', 'c', 'd'], ['g', 'h', 'i']]
})

sum(map(len, df.keywords))

12

Timings

df = pd.concat([df]*10000)

%timeit sum(map(len, df.keywords))
1.87 ms ± 52.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit df.keywords.map(len).sum()
13.5 ms ± 661 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df.keywords.str.len().sum()
14.3 ms ± 272 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Validation

>>> sum(map(len, df.keywords)) == df.keywords.map(len).sum() == df.keywords.str.len().sum()
True

A bit of a disclaimer: using pandas methods on columns that contain lists is always going to be inefficient (which is why using non-pandas' methods is so much faster here), since DataFrames are not meant to store list. You should try to avoid this whenever possible.

You can try this one:

df.keywords.map(len).sum()

I want to sum total number of list elements in column keywords

This is different from what you pseudo-coded. I believe you mean to call the size function for dataframes:

total_sum = keywords.size

Method 1:

len([item for sublist in df.keywords for item in sublist]

Method 2:

df.keywords.apply(len).sum()

.

df = [{"item": "a", "item_price": [1,1.5,2]}, {"item": "b", "item_price": [0.5,0.75,1]}]
df = pd.DataFrame(df)
print(df)
print("Ans:",len([item for sublist in df.item_price for item in sublist]))

OUTPUT

df

    item    item_price
0   a       [1, 1.5, 2]
1   b       [0.5, 0.75, 1]

Ans:6

More like a list flatten problem

import itertools
len(list(itertools.chain(*df.keywords.values.tolist())))
Out[57]: 13

Simple as that.

Maybe Pandas evolved since then.

df['len_of_list'] = df.my_columns_with_list.agg([len])

Cheers,

Related