Java Spark SQL, how to merge 3 datasets like 1&&(2||3)?These datasets have exactly same columns

Viewed 26

I need to filter out rows based on 3 rules, and there are correlations between them.

Basically, the final dataset should match rule1&&(rule2||rule3).

  1. If pid is "aaa" for all the rows in col1, then filter out all of them.
  2. If pid is "aaa" and col1 is null, then filter it out.
  3. If pid is "aaa" for all rows in (col2 and col3) key, then filter it out.

Example:

pid col1 col2 col3
abaa 111 apple red aaa 111 apple red aaa 222 banana yellow aaa 222 apple green
aaa 333 apple green aaa 333 null green aaa 444 apple red

1&&(2||3) = (1&&2)||(1&&3)

Row: aaa 333 null green satisfied rule1&2, should be filter out.

Rows: no_fdl 222 banana yellow no_fdl 222 apple green
no_fdl 333 apple green no_fdl 444 apple red

satisfied rule1&3, should be filter out.

And union those filter out rows, then what's left are

acc_id netg_set_id netting_id gfcid
abaa 111 apple red no_fdl 111 apple red

How can I filter out these at same time to get correct answer by using spark api?

0 Answers
Related