Setting frame position in video capture between each capture causes h264 errors "co located POCs unavailable" and "mmco: unref short failure"

I am reading a video file with OpenCV for image recognition where I'm just interested in 1 of 60 frames every second, so instead of capturing 60 frames before getting to the next one I thought I could speed up the code by setting frame position between each iteration

while (capture.isOpened()):

    capture.set(cv2.CAP_PROP_POS_FRAMES, frame_count)

    ret, frame = capture.read()
    frame_count += round(fps)

    if ret:
...

However, when I do it this way the following errors keep spamming in the terminal:

[h264 @ 0x21fc89c0] co located POCs unavailable
[h264 @ 0x21fba0a0] mmco: unref short failure

and it looks like it's reading two frames, then waiting a little or slowing down before the next frame, and then reading two frames fast after each other again:

If I do it the "normal" way where I read all frame but only running image recognition on every 60. frame it works totally normal, but if I increase the interval between running image recognition (for example 1 frame per minute or less) I experience that it actually takes quite a lot of time to just capture through all the frames that I don't use for anything.

Any inputs on this would be GREATLY appreciated, I've searched for a solution for quite some time now and yet havn't found anything :)