To use this family of functions you need to provide a function that returns a an object with class "ScaleContinuous" "Scale" "ggproto" "gg" (i.e. the equivalent output to scale_fill_viridis_c)!
scale_fill_custom <- function (..., alpha = 1, begin = 0, end = 1, direction = 1,
option = "D", values = NULL, space = "Lab", na.value = "grey50",
guide = "colourbar", aesthetics = "fill") {
continuous_scale(aesthetics, scale_name = "custom",
palette = scales:::gradient_n_pal(c("red", "blue", "grey50", "black"),
values, space), na.value = na.value,
guide = guide, ...)
}
ggplot(data = dat, aes(x = cyl, y = mean_hp, size = n, fill = n)) +
geom_point(shape = 21) +
scale_size_binned(breaks = c(8, 10, 12), guide = guide_bins(show.limits = T)) +
scale_fill_binned(breaks = c(8, 10, 12), guide = guide_bins(show.limits = T),
type = scale_fill_custom) +
labs(x = "Cylinder", y = "Mean hp", fill = "Nb of cars", size = "Nb of cars") +
theme_minimal()

Note that you are using colour as a scale to be translated by the eye into numerically meaningful difference. The colours are interpolated between the manually applied points, so will not actually be your exact colours. If you wish to band your averages by colour it would be preferable to create a factor, then manually apply your theme.
ggplot(data = mutate(dat, n = cut(n, breaks = c(0, 8, 10, 12, 20))),
aes(x = cyl, y = mean_hp, size = n, fill = n)) +
geom_point(shape = 21) +
scale_size_discrete() +
scale_fill_manual(values = c("red", "blue", "grey50", "black")) +
labs(x = "Cylinder", y = "Mean hp", fill = "Nb of cars", size = "Nb of cars") +
theme_minimal()
