How to delete all values in a column with lowercase

user_1592334 0
Viewed 60

I have a dataset I am working on and it has a description column where items for sale are represented in uppercase and expenses are represented in lowercase. I am trying to drop all rows with lower case.

Sample data :

invoiceid   description(string datatype)
100         WHITE METAL LANTERN
200         post expenses
300         BLACK WIRE

what I want to achieve

invoiceid   description
100         WHITE METAL LANTERN
300         BLACK WIRE

I tried using the following codes but I kept getting errors

from pyspark.sql.functions import upper, lower col

df4 = df3.where(col('Description').islower())
pyspark
1 Answers

One way; extract the caps into a column. The lower case will return blank. Filter out the blanks and drop the extract column to clean df

 df.withColumn('filt', regexp_extract('description(string datatype)','[A-Z]+', 0)).filter("filt != ''").drop('filt').show()

+---------+----------------------------+
|invoiceid|description(string datatype)|
+---------+----------------------------+
|      100|         WHITE METAL LANTERN|
|      300|                  BLACK WIRE|
+---------+----------------------------+
user_89869750
Related