How to pass a list list1:
list1 = [1,2,3]
into a spark.sql statement
spark.sql(f'select * from tbl where id IN list1')
Pass a list of values from a variable into IN statement sparksql
Viewed 1362
2 Answers
One way is to convert the list to tuple.
>>> list1
[1, 2, 3]
>>> str = r'select * from tbl where id in ' + format(tuple(list1))
>>> str
'select * from tbl where id in (1, 2, 3)'
To do that you need to format it correctly, and will be more trickier if you use the strings instead of numbers.
If you use PySpark, then it's better to use isin function instead, with something like this:
import pyspark.sql.functions as F
list1 = [1,2,3]
df = spark.read.table("tbl").filter(F.col("id").isin(*list1)
then you can use different data types the same way
Related