GHC Prefer one specialization over another

I have a Haskell typeclass function that looks like the following:

class Foo m where
   foo :: Num a => m a -> a

I want to generate two specializations of foo, one for anytime that m ~ X Int and the other when m ~ X Int, a ~ Int. I wrote the following code for this:

instance Foo (X a) where
   {-# SPECIALIZE instance Foo (X Int) #-}
   foo = ...
   {-# SPECIALIZE foo :: X Int Int -> Int #-}

From what I can tell, in the file that contains this code I am getting both specializations. However, in the module that is using Foo it seems to always point to the less generic specialization, i.e. foo :: Num a => X Int a -> a even when a ~ Int.

Is there a way to tell GHC to prefer the most specialized version of foo rather than the partially specialized version?

Thanks.