Delete strings that contain only numbers from the rows of a dataframe

user_16485940 0
Viewed 168

My dataframe contains many lines consisting of:
  • Numbers only;
  • Numbers and spaces;
  • Strings mixed with numbers (see Hell0 guys34 g00d j0b.);
+--------------------------+--------+
|   Content                | Score  |
+--------------------------+--------+
| Hello my name is Tim.    | 3      |
| 081332111333             | 4      |
| 08 97 5524               | 1      |
| 10 08 2021               | 1      |
| Great App                | 3      |
| Hell0 guys34 g00d j0b.   | 5      |
    . . . . . . . . . .      . . .
| 122345324 2331 111111    | 3      |
+--------------------------+--------+

Is there a way to clean up, that is to delete these 3 types of possible rows that appear in my dataframe, using the withColumn, or any other quick method?
I had created a for loop that iterates over all my rows in the Dataframe, but being over 1 million rows, it takes a long time. My environment is Google Colab
Thanks a lot!!

bigdata pyspark google-colaboratory apache-spark dataframe
1 Answers

Using pyspark, I would simply do something like this:

#assuming df is your dataframe
df.where(~df["content"].rlike("\d")).show()

\d Matches any decimal digit. Equivalent to [0-9].

user_50137520
Related