Reading Parquet dataset using PyArrow filter is not working

user_7585417 0
Viewed 34

I want to implement below query using PyArrow filter,

'(salary == 150280.17 or country == "Finland" ) and (first_name == "Amanda" or last_name == "Gray")'

dataset = pq.ParquetDataset(
  parquit_file,
  (use_legacy_dataset = False),
  (filters = [
    ([("salary", "==", 150280.17)], [("country", "==", "Canada")]),
    ([("first_name", "==", "Amanda")], [("last_name", "==", "Gray")]),
  ])
);
dataset.read().to_pandas();

but it is giving me error .

ValueError: not enough values to unpack (expected 3, got 1)

pyarrow
1 Answers

The filters should be a list[tuple] or a list[list[tuple]]:

dataset = pq.ParquetDataset(
  parquet_file,
  use_legacy_dataset = False,
  filters = [
      [
        ("salary", "==", 150280.17), 
        ("country", "==", "Canada"),
      ],
      [
        ("first_name", "==", "Amanda"),
        ("last_name", "==", "Gray"),
      ]
  ]
)
dataset.read().to_pandas()

You had an extra [].

user_1095250
Related