I'm using Renyi (ε, δ)-differential privacy as implemented in PyTorch DNNs. I do this to guarantee the privacy of the training set to the degree ε. In order to implement this, I use PyTorch-DP. As the network fits more and more on the trainingset and the accuracy goes up, the amount of privacy decreases. However, at some turning point, the privacy decreases and the accuracy decreases too; I assume that the accuracy eventually goes down because too much (privacy) noise is added to the mean of the clipped gradients over a batch of training data and/or because it overfits on the training set.
Now what I want to achieve is the following: I want to train a model to reach (ε, δ)-differential privacy with a differing value of ε (i.e. 0.5, 1, 1.5, 2). Mind that the bigger the ε-value, the less privacy the model guarantees. In an ideal situation, the accuracy would improve as ε increases. Yet this does not happen because after that turning point, the accuracy decreases. My question is, is there any way to guarantee/influence this so that the accuracy is still at a near-optimum at the point that the algorithm reaches the specified ε. Or is this not really as controllable as I'd like and is the best way to achieve this through empirically changing the learning rate, or changing the amount of additive clipping noise as well as the number of epochs so that more/less privacy is added each epoch but for a shorter/longer time.
I'm asking for ideas really.