In most of the papers, like retinanet(https://arxiv.org/abs/1708.02002), it computes iou between each anchor and ground truth. And give each anchor the ground truth assignment according to the iou threshold. for example,
Specifically, anchors are assigned to ground-truth object boxes using an intersection-over-union (IoU) threshold of 0.5; and to back-ground if their IoU is in [0, 0.4). As each anchor is assigned to at most one object box, we set the corresponding entry in its length k label vector to 1 and all other entries to 0.If an anchor is unassigned, which may happen with the overlap in [0.4, 0.5), it is ignored during training
here is the problem, if the anchor is ignored when training, it will not be updated when doing backward propagation. so many of them can not update the parameters.
so, what is the reason for not including all the anchors when training?
thank you