I need to filter out rows based on 3 rules, and there are correlations between them.
Basically, the final dataset should match rule1&&(rule2||rule3).
- If pid is "aaa" for all the rows in col1, then filter out all of them.
- If pid is "aaa" and col1 is null, then filter it out.
- If pid is "aaa" for all rows in (col2 and col3) key, then filter it out.
Example:
pid col1 col2 col3
abaa 111 apple red
aaa 111 apple red
aaa 222 banana yellow
aaa 222 apple green
aaa 333 apple green
aaa 333 null green
aaa 444 apple red
1&&(2||3) = (1&&2)||(1&&3)
Row: aaa 333 null green satisfied rule1&2, should be filter out.
Rows:
no_fdl 222 banana yellow
no_fdl 222 apple green
no_fdl 333 apple green
no_fdl 444 apple red
satisfied rule1&3, should be filter out.
And union those filter out rows, then what's left are
acc_id netg_set_id netting_id gfcid
abaa 111 apple red
no_fdl 111 apple red
How can I filter out these at same time to get correct answer by using spark api?