Usage of palette argument in scale_*_manual

Background

The help of ?scale_fill_manual says:

 ...: Arguments passed on to ‘discrete_scale’
          palette A palette function that when called with a single
              integer argument (the number of levels in the scale)
              returns the values that they should take.

Code

Thus, I tried:

library(ggplot)
library(dplyr)

my_pal <- colorRampPalette(1:3)

(p <- ggplot(mtcars %>% 
              group_by(cyl) %>% 
              summarise(mpg = mean(mpg)), 
             aes(x = factor(cyl), y = mpg, fill = factor(cyl))) +
   geom_col())

Problem

But if I want to add my palette function to the scale it does not work:

p + scale_fill_manual(palette = my_pal)
# Error in as.vector(x, "character") : 
#   cannot coerce type 'closure' to vector of type 'character'

Question

How do I use the palette argument properly? I know how I can set the color properly, but I want to understand what the palette argument is for and how to use it. If it is an internal argument, I was wondering why it is available in the scale_*_manual API in the first place.