Symfony form required only on custom validator fields

I am struggling to find a way to make required only those fields, who have my own custom validator attached. Is there any way to achieve this?

/**
 * @var string
 * @ORM\Column(type="string", length=255)
 * @AdminAssert\CustomNotBlank(          // <--- this is my custom validator
 *     groups={
 *         //...
 *     }
 * )
 */
protected $name;

/**
 * @var string
 * @ORM\Column(type="string", length=255)
 * @Assert\Email(
 *     groups={
 *         //...
 *     }
 * )
 */
private $email;

So in the example above only $name must have 'required' => true, because it was Validated by my own validator CustomNotBlank. And no field which does not have this validator no matter what other validation settings they have.

The FormType does not really matter because I have many fields of many Form Types (this is just a simple example of two text type fields)