Python: What's the fastest way to load a video file into memory?

First some background

I am trying to write my own set of tools for video analysis, mainly for detecting render errors like flashing frames and possibly some other stuff in the future.

The (obvious) goal is to write a script, that is faster and more accurate than me watching the file in real time.

Using OpenCV, I have something that looks like this:

import cv2


vid = cv2.VideoCapture("Video/OpenCV_Testfile.mov", cv2.CAP_FFMPEG)
width = 1024
height = 576
length = vid.get(cv2.CAP_PROP_FRAME_COUNT)

for f in range(length):
    blue_values = []
    vid.set(cv2.CAP_PROP_POS_FRAMES, f)
    is_read, frame = vid.read()
    if is_read:
        for row in range(height):
            for col in range(width):
                blue_values.append(frame[row][col][0])
    print(blue_values)
vid.release()

This just prints out a list of all blue values of every frame. - Just for simplicity (My actual script compares a few values across each frame and only saves the frame number when all are equal)

Although this works, it is not a very fast operation. (Nested loops, but most important, the read() method has to be called for every frame, which is rather slow. I tried to use multiprocessing but basically ended up having the same crashes as described here:

how to get frames from video in parallel using cv2 & multiprocessing in python

I have a 20s long 1024x576@25fps Testfile which performs as follows:

• mov, ProRes: 15s
• mp4, h.264: 30s (too slow)

My machine is capable of playing back h.264 in 1920x1080@50fps with mplayer (which uses ffmpeg to decode). So, I should be able to get more out of this. Which leads me to

my Question

How can I decode a video and simply dump all pixel values into a list for further (possibly multithreaded) operations? Speed is really all that matters. Note: I'm not fixated on OpenCV. Whatever works best.

Thanks!