Better heuristics to discard faces in images of sports competitions

I am currently working on a problem of re-identification of athletes in sports competitions, I have a large dataset of images like this:

runners and public

We are using retinafaces for face detection, and some heuristics to discard the faces of the audience and get only the runners,the heuristics I'm working with are:

1. Use the score that retinafaces gives us -> very bad heuristic
2. Use the size of the face to choose the largest or those that are greater than a certain threshold -> sounds good but it's bad
3. Use the distance of the face from the center of the image. -> sounds good but it's bad
4. Combinations between them -> the best but not very good

I was wondering if anyone has worked with a similar problem or knows of any creative solutions for this.(We have more models to detect faces but the same thing happens with all of us)

Thank you for reading!