How to find quantity of individual characters in Pandas

Viewed 91

I would need the program to accomplish getting the quantity of individual characters from list data that has been stored in a series. Here is an example of what I would like the program to accomplish:

char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
#end result (higher to lower frequency)
S : 4 
T : 3
L : 2
G : 2
N : 1
A : 1
M : 1
F : 1
P : 1

I know this can be accomplished with regular Python, but I am trying to find the Pandas equivalent. So far I have gotten:

pd.Series(char_list).value_counts())

But this returns the quantity of the whole sequences versus their individual characters. Any help would be great.

3 Answers

pandas isn't particularly well suited for this sort of problem. It can be done, but in the end you're either just going to iterate over the Series, as opposed to the list (really not different) or if you truly want to do it in a different pandas way it will involve some slow .str. methods.

The easiest way you get what you want is collections + itertools. Use a Counter and chain all the separate words together. Then you can make a Series out of the dict and sort it.

from collections import Counter
from itertools import chain
import pandas as pd

pd.Series(Counter(chain.from_iterable(char_list))).sort_values(ascending=False)

S    4
T    3
L    2
G    2
N    1
A    1
M    1
F    1
P    1
dtype: int64

If you want to use pandas you could split the words into each letter, drop the empty string at the start and the end, then explode and take the value_counts. But this is a lot more complicated and likely slower than the above.

s = (pd.Series(char_list)
       .str.split('')
       .str[1:-1]
       .explode()
       .value_counts())

S    4
T    3
L    2
G    2
N    1
A    1
M    1
F    1
P    1
dtype: int64

The core issue in solving this problem is that each cell contains a string value, so any pandas opertaions will consider the whole cell as a single unit. To remedy this, it is necessary to transform the data so that each cell is a single-character string.

There are many approaches with varying efficiencies, but they all fundamentally rely on turning a column of strings into a column of single-character strings.

  1. Creating a chain.from_interable then applying Series.value_counts:
from itertools import chain

import pandas as pd

char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
output = pd.Series(chain.from_iterable(char_list)).value_counts()
  1. map each string into a list, then Series.explode and value_counts:
import pandas as pd

char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
output = pd.Series(map(list, char_list)).explode().value_counts()
  1. Series.apply or Series.map list then explode and value_counts:
import pandas as pd

char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
output = pd.Series(char_list).apply(list).explode().value_counts()
import pandas as pd

char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
output = pd.Series(char_list).map(list).explode().value_counts()

All options produce: output:

S    4
T    3
L    2
G    2
N    1
A    1
M    1
F    1
P    1
dtype: int64

Some timing information via %timeit:

%timeit pd.Series(chain.from_iterable(char_list)).value_counts()
273 µs ± 10.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pd.Series(map(list, char_list)).explode().value_counts()
358 µs ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pd.Series(char_list).map(list).explode().value_counts()
428 µs ± 19.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pd.Series(char_list).apply(list).explode().value_counts()
508 µs ± 36.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

From @ALollz's answer:

%timeit pd.Series(Counter(chain.from_iterable(char_list))).sort_values(ascending=False)
215 µs ± 7.61 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pd.Series(char_list).str.split('').str[1:-1].explode().value_counts()
677 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
import pandas as pd
char_list = ['LNTS', 'AGTS', 'LMFS', 'PGST']
pd.Series(x for x in ''.join(char_list)).value_counts()
Related