From how I understand it, the only reason a staging site would require restart is if there exists app settings or connection strings configured as slot settings.

Although this doesn't always seem to be true. One of our applications will restart regardless. I have used Powershell cmdlets to be sure that there are no slot settings "hidden" from the Portal (because apparently this is a thing).

What other factors can determine whether the staging site will be restarted during a swap?

(I posted this on the kudu GitHub repository, but haven't heard back)

EDIT:

In response to Byron's answer:

I have done some pretty thorough testing (see my testing repo for this) and I am able to perform slot swapping (with and without Preview) without either production or staging slot restarting. This is obviously with no slot settings configured on the Web Apps.

Does this mean that restarts are not guaranteed? The issue I'm raising here is that if no restart occurs will IIS' Application Initialisation module still ensure that the application has indeed warmed up (as these things could take a couple of minutes)?