How to pass arguments dynamically to filter function in Apache Spark?

Question

I have a employees file which have data as below:

Name:   Age:
David   25
Jag     32
Paul    33
Sam     18

Which I loaded into dataframe in Apache Spark and I am filtering the values as below:

employee_rdd=sc.textFile("employee.txt")
employee_df=employee_rdd.toDF()
employee_data = employee_df.filter("Name = 'David'").collect()

+-----------------+-------+
|            Name:|   Age:|
+-----------------+-------+
|David            |25     |
+-----------------+-------+

But when I am trying to do something like this:

emp_Name='Sam' and passing this Name to filter like below:

employee_data = employee_df.filter("Name = 'emp_Name'").collect

but this is giving me empty list.

user_7918500 · Answer

Try the following:

emp_Name='Sam'
employee_data = employee_df.filter(employee_df["Name"] == emp_Name).collect()

user_10224456 · Answer

from pyspark.sql.functions import col
filter_condition= col("Name").isin(["Sam","John"])
employee_data = employee_df.filter(filter_condition).collect