Symfony3 form validation not called for field use by a model transformer

Edit: here's a github with the full code to reproduce the problem

I have the following entity

class Place
{
    use Traits\HasId;

    /**
     * Used for form.
     *
     * @Assert\Image(
     *     mimeTypes = {"image/png", "image/jpeg"},
     *     minWidth = 50,
     *     maxWidth = 1000,
     *     minHeight = 50,
     *     maxHeight = 1000,
     *     maxSize = "1M"
     * )
     */
    private $imageFile = null;

    /**
     * @ORM\OneToOne(targetEntity="MyImage", orphanRemoval=true, cascade={"persist"})
     */
    protected $image;

}

With the following form

class AdminPlaceType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $transformer = new HasImageTransformer();
        $builder->add('imageFile')->addModelTransformer($transformer);
    }

    public function configureOptions(OptionsResolver $resolver)
    {
        $resolver->setDefaults(['data_class' => 'AppBundle\Entity\Place']);
    }
}

And the following model transformer

class HasImageTransformer implements DataTransformerInterface
{   
    public function transform($hasImage)
    {
        return $hasImage;
    }

    /**
     * reverse transforms.
     */
    public function reverseTransform($hasImage)
    {   
        $file = $hasImage->getImageFile();
        $myImage = new MyImage();
        $myImage->setData(file_get_contents($file->getPathName()))
        $myImage->setMimeType($file->getMimeType());

        $hasImage->setImage($myImage);
    }
}

I can upload a correct image, and the form is correctly saved in database.

However if I submit an incorrect image (for example a .txt file), the form is still saved in database without any error

However if I remove the addModelTransformer from the Form, then I got the correct validation error

This file is not a valid image

as my transformer does not modify the original imageFile field, I'm wondering what could cause this problem.

I'm using php7 and symfony3.3.4