Passing std::function type which has a templated return type

I have a function

template<typename P = int_fast16_t, typename I>
std::vector<std::vector<P>> f(
    I xb, I xe, I yb, I ye,
    std::function<P(typename std::iterator_traits<I>::value_type,
                    typename std::iterator_traits<I>::value_type)> &&score_function,
    P ID = -1, P S = -1, P M = 1)

It takes two pairs of iterators and a function that should compare two elements of the value_type of the iterators and returns a value of type P.

This gives me an error

./h.hpp:47:32: note: candidate template ignored: could not match 'function<type-parameter-0-0 (typename iterator_traits<type-parameter-0-1>::value_type, typename iterator_traits<type-parameter-0-1>::value_type)>' against 'stringAlgorithms::scoring::plus_minus_one'
   std::vector<std::vector<P>> nw_score_matrix(I xb, I xe, I yb, I ye, 

Now, if I change this to use the specific return type P

template<typename P = int_fast16_t, typename I>
   std::vector<std::vector<P>> nw_score_matrix(I xb, I xe, I yb, I ye, std::function<int_fast16_t(typename std::iterator_traits<I>::value_type, typename std::iterator_traits<I>::value_type)> &&score_function, P ID = -1, P S = -1, P M = 1)

this compiles.

In this case the function plus_minus_one is

  struct plus_minus_one {
     template<typename T, typename R = int_fast16_t>
     R operator()(const T &x, const T &y) { return x == y ? 1 : -1; }
  };

and passed using

scoring::plus_minus_one matchScoring;

and

nw_score_matrix(x.begin(), x.end(), y.begin(), y.end(), matchScoring);

I realize that I could just declare a typename F in the template and make score_function a

F &&score_function

however I want to ensure that if someone creates a functor/lambda that's specific to some types that the function handles the right types. So why doesn't the declaration compile?

EDIT: a full example is found https://github.com/meconlen/stringAlgorithms/tree/soq