Gradients of custom module parameter remains zero after loss.backward()

I am exploring the synthetic petri dish method (https://paperswithcode.com/paper/synthetic-petri-dish-a-novel-surrogate-model) as a performance estimator for a NAS. Below I have written a small script that emulates the problem I have in my program on a smaller scale.

class SynthData(torch.nn.Module):
    def __init__(self):
        super(SynthData, self).__init__()
        self.synthetic_data = torch.nn.Parameter(torch.rand(1, 3, 16, 16))
        self.synthetic_classes = ['Apple']

synthetic_pool = SynthData()
synthetic_pool = synthetic_pool.to('cpu')
print(f"synthetic_image: {synthetic_pool.synthetic_data}")

optimizer = torch.optim.Adam(params=synthetic_pool.parameters(), lr=0.0001, weight_decay=0.000001)
criterion = torch.nn.MSELoss()

motif_validation_acc = [0.0, 0.1, 0.1, 0.1, 0.0]
ground_truth_validation_acc = [0.821213940779368, 0.9302884737650553, 0.9124933083852133, 0.8845486243565878,
                               0.9230769276618958]
# Covert to tensors
motif_validation_acc_tensor = torch.tensor(motif_validation_acc, requires_grad=True).float().to('cpu')
ground_truth_validation_acc_tensor = torch.tensor(ground_truth_validation_acc).float().to('cpu')
print(f"motif_validation_acc: {motif_validation_acc_tensor}")
print(f"ground_truth_validation_acc: {ground_truth_validation_acc_tensor}")

optimizer.zero_grad(set_to_none=True)
loss = criterion(motif_validation_acc_tensor, ground_truth_validation_acc_tensor)
print(f"Outer_loop loss: {loss}")
loss.backward()

for param in synthetic_pool.parameters():
    print(param.grad)

# Optimize the synthetic image
optimizer.step()

A little context:

• The SynthData() class holds different images as parameters which are later used in an optimizer.
• The loss is calculated using MSE between motif_validation_acc and ground_truth_validation_acc. This loss is used to optimize the images stored in SynthData() in order to get the validation acc of the motifs as close as possible to the validation acc of the ground truth (which are the architectures in nasbench 101).

The problem is that the gradient of the parameter in the SynthData class remains None even after calling loss.backward() and I have no clue what is causing this.