Setting x and y indexes in pySpark dataframe plot

Question

I have a pySpark dataframe on which I apply some SQL queries and want to plot the results.

state_grouped = "SELECT customer_state, AVG(review_score), SUM(review_score), AVG(order_products_value) FROM global_temp.olist_table GROUP BY customer_state ORDER BY AVG(review_score) DESC"

spark.sql(state_grouped).show()

To plot this, I convert it to pandas dataframe using toPandas and then do a plot

spark.sql(state_grouped).toPandas().plot(kind='barh', figsize=(12,11), logx=True)

The resulting diagram looks like below:

As you see in the y - axis, the indexes are basically numbers representative of the customer_state column. Instead of those numbers, I want to show the actual customer_state such as PR, SP etc.

How do I do that?

I know if I were using pd.Dataframe to convert to pandas dataframe, I could specify index=['PR','SP',...] but since I am using .toPandas instead, I am not sure a way to specify the actual Y axis indexes.

Can anyone please suggest?

user_11840002 · Answer

You need to set the ".set_yticklabels" and for that you need the y tick labels as a list which can be done by getting the column you want to be the y tick labels:

ylabels = example_df.select("Example_Col").rdd.flatMap(list).collect()

Then you can set the y ticks labels

example_df.toPandas().plot(kind='barh', figsize=(12,11), logx=True).set_yticklabels(ylabels)

user_6623907 · Answer

try this out

ax = spark.sql(state_grouped).toPandas().plot(kind='barh', figsize=(12,11), logx=True)
ax.set_xlabel("x label")
ax.set_ylabel("y label")

or if you looking in one line code

spark.sql(state_grouped).toPandas().plot(kind='barh', figsize=(12,11), logx=True).set(xlabel="x label", ylabel="y label")

it should be something like :

you can also add a title on your plot by addind 'title' parameter in your code :

spark.sql(state_grouped).toPandas().plot(kind='barh', figsize=(12,11), title='My Plot', logx=True).set(xlabel="x label", ylabel="y label")

the output will be :

Setting x and y indexes in pySpark dataframe plot

2 Answers