Detect if a button attribute is deleted and cancel the request? (Javascript)

I have this little code to create my own captcha with PHP and JavaScript. If the sum of 2 numbers is correct, it changes from "disabled" to "enabled" the button that makes the AJAX request. The problem I found is that if I manually remove the "disabled" attribute or simply remove the input captcha, the AJAX request is made.

What is the best way to fix this? I do NOT need to change my button type to "submit" since I don't want my page to refresh.

PHP:

<?php
$min  = 1;
$max  = 300;
$num1 = rand( $min, $max );
$num2 = rand( $min, $max );
$sum  = $num1 + $num2;
?>

HTML:

<form>
<label for="quiz" class="col-sm-3 col-form-label">
<?php echo $num1 . '+' . $num2; ?>?
    </label>
    <div class="col-sm-9">
        <input type="text" class="form-control quiz-control" id="quiz">
    </div>
<div class="col-md-6">
    <button id="botonAjax" data-res="<?php echo $sum; ?>" type="button" class="btn btn-dark w-100 fw-bold" disabled>Send</button>
</div>
</form>

Javascript:

const submitButton = document.getElementById('botonAjax');
const quizInput = document.querySelector(".quiz-control");
quizInput.addEventListener("input", function(e) {
    const res = submitButton.getAttribute("data-res");
    if ( this.value == res ) {
        submitButton.removeAttribute("disabled");
    } else {
        submitButton.setAttribute("disabled", "");
    }
});