I'm trying to train the object detection model for annotated data containing imbalanced labels. Since the classes are imbalanced, the trained model will be overfitted. To avoid this problem I'm picking up equal number of classes from annotated data for training the model. If this performs well I can balance the other classes with augmentation.
I wrote below script to choose certain number of classes which satisfies required number from annotated data
read_file = open('read.csv','r')
write_file = open('write.csv','w')
csv_read = csv.reader(read_file)
csv_write = csv.writer(write_file)
counter = 0
class_list=[]
class_dic = {}
no_of_samples_count = 100
for cnt, read in enumerate(csv_read):
if len(class_list) == 100:
if read[1] in class_dic:
if class_dic[read[1]]== 101:
continue
file=read[0]
class_name=read[1]
xmin=read[2]
xmax=read[3]
ymin=read[4]
ymax=read[5]
shutil.copy(file,'images')
csv_write.writerow([file,class_name,xmin,xmax,ymin,ymax])
if class_name in class_list:
class_dic[class_name]+=1
continue
else:
class_dic[class_name]=1
class_list.append(class_name)
sum = 0
for x in class_dic.values():
sum+=x
if sum == no_of_samples_count*no_of_samples_count:
break
else:
print('sum:%d'%sum)
continue
print(class_dic)
here I'm giving length of class_list as 100 to pick only 100 class labels and no_of_samples_count as 100 to pick 100 labeled box dimension from selected 100 class_list
problem with this script is it creates 100 samples but not balancing the no_of_samples_count I get class labels of uneven number of samples.
suppose
class_names = ['one','two','three','four','five','six','seven','eight','nine','ten']
and there number of samples dimensions are
class_names_count = {'one':7,'two':2,'three':9,'four':11,'five':3,'six':20,'seven':18,'eight':3,'nine':90,'ten':1}
if length of class_list is 3 no_of_samples_count is 10
then I should get output as
{'four':10,'six':10,'seven':10}
as you can see I'm collecting only 3 random labels as given length of class_list is 3 which has count more than 10 as given no_of_samples_count is 10