Lost in interface variance. IInterface<in T1, out T2> is not working as expected

Hi, I have a variant factory interface. Now, I am deriving factories from it and would like to be able to pass a factory of more derived type as a factory of less derived type. It doesn't work and I am really trying to figure out why as from the little I know about variance, it should.

Please consider this code:

public interface IFactory<in TParam, out TValue> { }

public class Fruit { }

public class Apple : Fruit { }

public class BoxOfFruits { }

public class BoxOfApples : BoxOfFruits { }

public class BoxOfFruitsFactory : IFactory<Fruit, BoxOfFruits> { }

public class BoxOfApplesFactory : IFactory<Apple, BoxOfApples> { }

public class FruitPacker
{
    public FruitPacker(IFactory<Fruit, BoxOfFruits> factory) { }
}

public class ApplePacker : FruitPacker
{
    //This works fine as expected, as there is no variance in play.
    public ApplePacker(BoxOfFruitsFactory factory) : base(factory) { }

    //This complains about not being able to convert BoxOfApplesFactory to IFactory<Fruit, BoxOfFruits>
    public ApplePacker(BoxOfApplesFactory factory) : base(factory) { }

    //This is not working either
    public ApplePacker(IFactory<Apple, BoxOfApples> factory) : base(factory) { }

    //But this does work
    public ApplePacker(IFactory<Fruit, BoxOfApples> factory) : base(factory) { }
}

Could someone please explain, why this is not working? My assumtions are that "in" lets me pass in more derived type and "out" lets me output less derived type.

Thanks