Two way binding for Formik Field and custom component in React

I am trying to pass my own functional component as a component attribute to a Formik Field in React. The problem is I don't know how to bind the Formik Field's value with the value in my custom component. Here's what I tried:

Form.js

// other Formik Form code...
<Field
    className="form-control"
    name="NetWeight"
    component={IntegerComponent}
></Field>
//...

IntegerComponent.js

function IntegerComponent({ field, className }) {
    const onChangeHandler = event => {
      const returnVal = (event.target.validity.valid) ? event.target.value : inputValue;
      setInputValue(returnVal);
    };

    return <input
      type="text"
      pattern="[0-9]*"
      className={className}
      onChange={onChangeHandler}
      value={inputValue}
    />
}

I want to bind Formik's field.value which I am destructuring from the props, to my NumberComponent's input's value. I tried doing field.value = returnVal but it didn't work as well. I know I can use onBlur and onChange handlers for above purpose than creating my functional component but the way my project is built, if I can make this work, it will help reducing a lot of redundant code. Any thoughts or suggestions?