About tidyverse median in summarize

Viewed 141

I have the following code, that generates a plot:

library(tidyverse)

plastics <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-01-26/plastics.csv')

plastics %>% 
  group_by(year, country) %>%
  summarize(hdpe = mean(hdpe)) %>%
  ungroup() %>%
  group_by(year) %>%
  top_n(10, hdpe) %>%
  ungroup() %>%
  mutate(country = fct_reorder(country, hdpe)) %>%
  ggplot(aes(country, hdpe)) + geom_col() + coord_flip() + facet_wrap(~year, scales = "free_y")

enter image description here

However, when I try to select the median of the hdpe, I don't get the 20 rows I would expect.

library(tidyverse)

plastics <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-01-26/plastics.csv')

plastics %>% 
  group_by(year, country) %>%
  summarize(hdpe = median(hdpe)) %>%
  ungroup() %>%
  group_by(year) %>%
  top_n(10, hdpe) %>%
  ungroup() %>%
  mutate(country = fct_reorder(country, hdpe)) %>%
  ggplot(aes(country, hdpe)) + geom_col() + coord_flip() + facet_wrap(~year, scales = "free_y")

enter image description here

Could somebody explain what's going on here?

1 Answers

I suspect what's going on is that you have a lot of cases where "hdpe = 0" and, as a result, the median is 0 even though the mean is > 0. If you filter hdpe for values > 0 you may see the pattern you're looking for, but the actual answer won't be "correct" (median = 0 is the "correct" answer, even though it's not useful for selecting top_n values). E.g.

library(tidyverse)

plastics <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-01-26/plastics.csv')
#> 
#> ── Column specification ────────────────────────────────────────────────────────
#> cols(
#>   country = col_character(),
#>   year = col_double(),
#>   parent_company = col_character(),
#>   empty = col_double(),
#>   hdpe = col_double(),
#>   ldpe = col_double(),
#>   o = col_double(),
#>   pet = col_double(),
#>   pp = col_double(),
#>   ps = col_double(),
#>   pvc = col_double(),
#>   grand_total = col_double(),
#>   num_events = col_double(),
#>   volunteers = col_double()
#> )

plastics %>% 
  group_by(year, country) %>%
  filter(hdpe > 0) %>% 
  summarize(hdpe = median(hdpe)) %>%
  ungroup() %>%
  group_by(year) %>%
  top_n(10, hdpe) %>%
  ungroup() %>%
  mutate(country = fct_reorder(country, hdpe)) %>%
  ggplot(aes(country, hdpe)) + geom_col() + coord_flip() + facet_wrap(~year, scales = "free_y")
#> `summarise()` has grouped output by 'year'. You can override using the `.groups` argument.

Created on 2021-08-16 by the reprex package (v2.0.0)

To try and illustrate this, you can create a fake dataset with a lot of zeros and check the mean and median:

library(tidyverse)
df <- data.frame(names = rep(LETTERS[1:2], 50),
                 values = rpois(n = 100,lambda = c(0.5)))
head(df, 20)
#>    names values
#> 1      A      2
#> 2      B      0
#> 3      A      0
#> 4      B      0
#> 5      A      1
#> 6      B      0
#> 7      A      1
#> 8      B      0
#> 9      A      0
#> 10     B      0
#> 11     A      0
#> 12     B      0
#> 13     A      0
#> 14     B      0
#> 15     A      0
#> 16     B      1
#> 17     A      0
#> 18     B      0
#> 19     A      0
#> 20     B      1

df %>% 
  group_by(names) %>% 
  summarize(mean = mean(values))
#> # A tibble: 2 x 2
#>   names  mean
#>   <chr> <dbl>
#> 1 A      0.36
#> 2 B      0.46

df %>% 
  group_by(names) %>% 
  summarize(median = median(values))
#> # A tibble: 2 x 2
#>   names median
#>   <chr>  <dbl>
#> 1 A          0
#> 2 B          0

Created on 2021-08-16 by the reprex package (v2.0.0)

Does that make sense?

Related