Turn off validation for form type if checkbox is checked in Symfony

I'm trying to create an address form with a payment and shipping address. When a checkbox on the shipping address is checked, I want to skip form validation from occurring on that address.

I have created the form type below with a toggle option that will display and deal with the checkbox, however the form is still validated even when checked.

Symfony has documentation on how to implement such a form, and even though I almost have the exact same code, validation is not turned off when checked. I am not using validation groups, so I just disable the default group to disable validation on the entity.

The AddressType building a form for the Address class (which has annotation constraints on certain fields like NotBlank and Callback).

class AddressType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        if ($options["toggle"]) {
            $builder->add("toggle", CheckboxType::class, [
                "mapped" => false,
                "required" => false,
                "label" => $options["toggle"]
            ]);
        }

        $builder
            ->add("name", TextType::class, [
                "required" => !$options["toggle"]
            ])
            ->add("address", TextType::class, [
                "required" => !$options["toggle"]
            ])
            ->add("zipcode", TextType::class, [
                "label" => "Postcode",
                "required" => !$options["toggle"]
            ])
            ->add("city", TextType::class, [
                "required" => !$options["toggle"]
            ])
            ->add("countryCode", ChoiceType::class, [
                "choices" => Address::COUNTRY_CODES
            ]);
    }

    public function configureOptions(OptionsResolver $resolver)
    {
        $resolver->setDefaults([
            "toggle" => false,
            "data_class" => Address::class,
            "validation_groups" => function(FormInterface $form) {
                if ($form->has("toggle") && $form->get("toggle")->getData() === true) {
                    return [];
                }

                return ["Default"];
            }
        ]);

        $resolver->setAllowedTypes("toggle", ["bool", "string"]);
    }
}

I use the type like this:

$addressForm = $this
    ->createFormBuilder([
        "paymentAddress" => $paymentAddress,
        "shippingAddress" => $shippingAddress
    ])
    ->add("paymentAddress", AddressType::class, [
        "label" => false
    ])
    ->add("shippingAddress", AddressType::class, [
        "label" => false,
        "toggle" => "Use payment address"
    ])
    ->add("submit", SubmitType::class, [
    ])
    ->getForm();

I've been going over this for a few hours now but I am not able to deduce why validation is not being turned off, and I am not willing to botch up a form over this little detail.

Why is validation for the AddressType not turned off by the closure in configureOptions? If this is just not how it works, what would be a better solution to accomplish partially turning off validation in a tidy way?

EDIT: Even if setting "validation_groups" => false in the defaults, in the children created in the builder, or in the usage of the form, validation will still happen. It does not have to do with the closure. Every online resource, including Symfony's own resource states that it should work though...