How to limit max number of partitions in scope of partitioning by column

Viewed 48

I have a PySpark dataframe like:

City  SomeValue

NY     5
NY     4   
WA     1
WA     3
CA     7
SF     8

I want to write it to for example 2 files, but all records with the same value of City should be in the same file (number of files should be less than number of unique cities). How can I do it using pySpark?

0 Answers
Related