Pyspark: Reorder only a subset of rows among themselves

user_7518708 0
Viewed 58

my data frame:

+-----+--------+-------+
| val |  id    | reRnk |
+-----+--------+-------+
| 2   | a      | yes   |
| 1   | b      | no    |
| 3   | c      | no    |
| 8   | d      | yes   |
| 7   | e      | yes   |
| 9   | f      | no    |
+-----+--------+-------+

In my desired output I will re-rank only the columns where reRnk==yes, ranking will be done based on "val" I don't want to change the rows where reRnk = no, for example at id=b we have reRnk=no I want to keep that row at row no. 2 only. my desired output will look like this:

+-----+--------+-------+
| val |  id    | reRnk |
+-----+--------+-------+
| 8   | d      | yes   |
| 1   | b      | no    |
| 3   | c      | no    |
| 7   | e      | yes   |
| 2   | a      | yes   |
| 9   | f      | no    |
+-----+--------+-------+
apache-spark ranking row dataframe python
1 Answers

From what I'm reading, pyspark DF's do not have an index by default. You might need to add this.

I do not know the exact syntax for pyspark, however since it has many similarities with pandas this might lead you into a certain direction:

df.loc[df.reRnk == 'yes', ['val','id']] = df.loc[df.reRnk == 'yes', ['val','id']].sort_values('val', ascending=False).set_index(df.loc[df.reRnk == 'yes', ['val','id']].index)

Basically what we do here is isolating the rows with reRnk == 'yes', sorting these values but resetting the index to its original index. Then we assign these new values to the original rows in the df.

for .loc, https://spark.apache.org/docs/3.2.0/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.loc.html might be worth a try.

for .sort_values see: https://sparkbyexamples.com/pyspark/pyspark-orderby-and-sort-explained/

user_71944740
Related