How number of classes impact thespeed FPS in a YOLO network

I am a newbie about Computer Vision and Neural Networks. I trained a network using Yolo-tiny, using a subset of the COCO dataset because I wanted to train my network only with one class (Person). The training worked successfully, the accuracy not so good (I know Yolo-tiny is faster but lower accuracy than YOLOV3) and I get around 50-60FPS on videos (with size 416x416 in the cfg file) but very low FPS if I test in a real time application through a cam (around 2-3FPS). By training again the network with higher number of classes, such performances haven't improved. My question is basically, how the number of classes we choose to train the network impact the speed/FPS of detection? If I train the network against 1 class or against 80 classes, does this also lead an improvement to the performance of speed, or not? If not, why?