I have 2 lists one looks like
['sdf_ratio_dffv','ewdef','edef_ratio_dcdc']
2nd one looks like
['%ratio%','%de%']
I want to find all the values that matches with the second one maybe with a likely function.
Is that possible in Pyspark/python? Can someone share their expertise here please?
Not sure what you mean by lists. But if you have a data frame and want to find rows that match a list of regexes, you could use "rlike".
df = spark.createDataFrame([["some-ratio-some", 20], ["dd-ratio-dd", 30], ["dd-something-dd", 30], ["dd-de-dd", 30], ["other-other-dd", 30]], ["name", "age"])
from pyspark.sql import functions as F
df.filter(F.col("name").rlike(".*ratio.*|.*de.*")).show()
+---------------+---+
| name|age|
+---------------+---+
|some-ratio-some| 20|
| dd-ratio-dd| 30|
| dd-de-dd| 30|
+---------------+---+