Length of train image dataset and train image loader are different?

user_18336458 0
Viewed 109

image_datasets Is a dictionary containing both train and test data.

Code below:

transforms= transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

image_datasets = {'train': datasets.CIFAR10(root=data_dir, train=True, download=True, 
                            transform=transforms),
                  'test': datasets.CIFAR10(root=data_dir, train=False, download=True,
                            transform=transforms)
                  }

image_datasets

OUTPUT:

{'test': Dataset CIFAR10
     Number of datapoints: 10000
     Root location: ../Data
     Split: Test
     StandardTransform
 Transform: Compose(
                ToTensor()
                Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
            ), 'train': Dataset CIFAR10
     Number of datapoints: 50000
     Root location: ../Data
     Split: Train
     StandardTransform
 Transform: Compose(
                ToTensor()
                Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
            )}


#CREATING DATA LOADER 
data_loaders = {
    'train': DataLoader(image_datasets['train'], 10, shuffle=True),
    'test': DataLoader(image_datasets['test'], 10)}

When I call len(data_loaders['train']) it returns

5000

When defining my data loader I am using a batch_size=10. Is the length of my data_loader being divided by my batch_size. New to coding and just wanted to double check.

conv-neural-network pytorch python
1 Answers

In brief,

  • len(data_loaders['train'].dataset)gives you how many instances in the dataset e.g., 50000 in CIFAR10.
  • len(data_loaders['train']) gives you the number of batches in this dataloader e.g., 5000 in CIFA10 if your batch_size=10. Number of batches is calculated by len(dataset)/(batch_size).

Therefore, when we calculate the accuracy in each epoch we divide number of correct ones over len(data_loaders['train'].dataset) not len(data_loaders['train']) because I fixed this bug in Stack Overflow for someone whose accuracy goes beyond 100% and the cause is division over len(data_loaders['train']) .

user_167331010
Related