Search for identical strings of certain length in Python

user_12773841 0
Viewed 59

I have a question concerning a large dataset containing a column 'text' with strings. My goal is to search for identical strings with a minimum of 20 characters in Python. The dataset that I need thus contains al these identical strings.

For example the following dataset named df:

  idnr       text 

      1      This sentence is more than 20 characters long
      2      Shorter sentence
      3      Short
      4      Short
      5      This sentence is more than 20 characters long
      6      Shorter sentence
      7      This sentence is more than 20 characters long
      8      Shorter sentence
      9      Short
     10      This sentence is more than 20 characters long
     11      This sentence is more than 20 characters long
     12      Shorter sentence

The output should be:

  idnr       text 

      1      This sentence is more than 20 characters long
      5      This sentence is more than 20 characters long
      7      This sentence is more than 20 characters long
     10      This sentence is more than 20 characters long
     11      This sentence is more than 20 characters long

What script can I use for this in Python?

pandas python
1 Answers

You can filter rows by Series.str.len with greater like 20 chained by Series.duplicated for 2 or more values:

df1 = df[df['text'].str.len().gt(20) & df['text'].duplicated(keep=False)]

print (df1)
    idnr                                           text
0      1  This sentence is more than 20 characters long
4      5  This sentence is more than 20 characters long
6      7  This sentence is more than 20 characters long
9     10  This sentence is more than 20 characters long
10    11  This sentence is more than 20 characters long

For testing is possible use:

m1 = df['text'].str.len().gt(20)
m2 = df['text'].duplicated(keep=False)
m3  = df['text'].str.len().gt(20) & df['text'].duplicated(keep=False)

print (df.assign(m1=m1, m2=m2, both=m3))
    idnr                                           text     m1    m2   both
0      1  This sentence is more than 20 characters long   True  True   True
1      2                               Shorter sentence  False  True  False
2      3                                          Short  False  True  False
3      4                                          Short  False  True  False
4      5  This sentence is more than 20 characters long   True  True   True
5      6                               Shorter sentence  False  True  False
6      7  This sentence is more than 20 characters long   True  True   True
7      8                               Shorter sentence  False  True  False
8      9                                          Short  False  True  False
9     10  This sentence is more than 20 characters long   True  True   True
10    11  This sentence is more than 20 characters long   True  True   True
11    12                               Shorter sentence  False  True  False
user_29010020
Related