Yup validation on two input fields with same name

I am using formik in react to add multiple address at run time. On click of 'Add more address' I simply append the input filed with same name as first one but with index. I am using names as array so on submit i will have all values as array in single variable here is my form component code

const [forms,setForms] = useState(0);
const getErrors = (n,msg)=>{
    const errObj={0:msg};
    for (let i = 1; i < n; i++) {
        errObj[i] = msg;
    }
    return errObj;
}
const notEmpty = (value)=>{
    return value.length===forms+1 && value.every(v=>v!=null && v!=="");
}
<Formik
            initialValues={initialValues}
            validationSchema={Yup.object({
                city:Yup.array().test('required',getErrors(forms+1,"This field is required"),(value)=>notEmpty(value)),
            })}
            onSubmit={async (values) => {
                handleSubmit(values);   
            }}
            <Form>
                <TextInput 
                       label='City'
                       required='true'
                       name='city[0]'
                       className='col xl-6'
                />
               {
                 [...Array(forms)].map((formNo,index)=>{
                    return(
                            <TextInput 
                                label='City'
                                required='true'
                                name={`city[${index+1}]`}
                                className='col xl-6'
                            />
                              )
                            })
                        }
               <Submit
                    type='submit'
                    isSubmitting={isSubmitting}
                    text='Save Address'
                    submittingText='Saving...'
               />
               <button 
                   onClick={()=>setForms(forms+1)} 
                   type="button"
                   className="add-more-btn">
                   Add Address<span><Plus height={30} width={30} color="#0FE3C7"/> 
                   </span>
              </button>
      </Form>
</Formik>

Now the issue is that when i don't specify the values for new fields i entered at runtime. Instead of getting validation errors on those fields I am getting error on above fields which are already filled. un till i click any of new entered fields. once i click on newly entered fields I am getting errors there too. but errors on previous fields still appear. I want errors to be appear on only invalid fields.

Any clue would be appreciated.. :)