I have Jenkins controller running on prem at the company I work for. We are moving to the cloud and have started by migrating our jenkins agents to run on AWS EKS (kubernetes). We use quite powerfull intsances which should have plenty of CPU horsepower and RAM and even if I assign every agent 4Cpus and >8 gigs of RAM the jobs take more than double the time that it took with agents on the same host as the controller which had less CPU and RAM.
First we thought that the issue is network related but we introduced package caches, jenkins tool caches and anything else we could cache.
Also weirdly even executing tests takes more than twice as long on the cloud instances. Tests run locally without fetching any artifacts from the web or from our company repository.
Does anyone have an idea why everything is so slow? Is it maybe because the controller and agent have to communicate a lot? Is there any way to make things faster? How can I find the reason why things take so long?