Return rows w/ string content that don't have words exceeding a certain max length while retaining and filtering out words w/ certain content

user_11564795 0
Viewed 80

Here is my dataframe

Input

        qid                     question_stemmed    target  question_length total_words
443216  56da6b6875d686b48fde    mathfracint1x53x5 tantanboxedint1x01x2 sumvarp...   1   589 40
163583  1ffca149bd0a19cd714c    mathoverbracesumvartheta8infty vecfracsumkappa...   1   498 31
522266  663c7523d48f5ee66a3e    httpgooglecom check out the content of the www..    0   449 66
522379  756678d3d48f5ee66a3e    mark had a great day he plans to go fishing with.   0   310 23

I am using the following logic to only return records from the df whose question_text column has

  • any word whose length shouldn't exceed 15 characters (note: not string length) (using negation)
  • any word that shouldn't contain numeric values while above condition is true (using negation)
  • while making sure words with http or www values are retained (while above 2 conditions are still true)

df = df[(~df['question_stemmed'].str.len() > 15) & (~df['question_stemmed'].str.contains(r'[0-9]')) & (df.question_stemmed.str.match('^[^\http]*$'))]

Getting error error: bad escape \h at position 3

Expected output

        qid                     question_stemmed     target    question_length  total_words
522266  663c7523d48f5ee66a3e    httpgooglecom check out the content of the www..    0   449 66
522379  756678d3d48f5ee66a3e    mark had a great day he plans to go fishing with.   0   310 23

Also, wondering if the logic above will work or not to satisfy all 3 conditions Appreciate any help

pandas python regex string
1 Answers

I suggest using

df = df[~df['question_stemmed'].str.contains(r'(?<!\S)(?!\S*(?:http|www\.))\S{15}')]

See the regex demo

Details

  • (?<!\S) - whitespace or start of string should come immediately before the current location
  • (?!\S*(?:http|www\.)) - no 0 or more non-whitespace chars followed with http or www. substrings allowed immediately to the right of the current location
  • \S{15} - fifteen non-whitespace chars.
user_38329700
Related