Error Connection Time out for the first visit on my site

My site runs on Openlitespeed (Cyberpanel) server on a VPS, and it is a wordpress site. I am getting error connection time out on every first visit from a different device,. and also after a long time on the same device. How do I diagnose what is wrong on my site? I don't run any caching plugin enabled, etc. Although I have them installed and was running previously. I tried rebooting VPS, this happened even with a alternate vps panel. So This is not related to VPS panels.