Let's say you have video of the night sky taken over hours. You want to get the brightness of a certain region of sky as a function of time. How would you efficiently get this information programmatically (say, in python)?
I only know how to process videos at the level of splitting them into frames first, then process each frame individually. But this seems horribly inefficient, and I feel that there's got to be a way to process videos for scientific purposes as efficiently as video codecs do when they replay them. There's a lot of information about video processing online, but it's hard to find information that addresses my question. Generally, how is scientific video processing done in a way that's more efficient than frame-by-frame?