gunicorn increasing response time

The current flask setup is as follows, this is not the exact code, just a brief overview example

src/app.py

from gevent.pywsgi import WSGIServer
app = Flask(
    __name__,
    template_folder="../data",
    static_folder="../data",
)
http_server = WSGIServer(("0.0.0.0", 5001), app)
http_server.serve_forever()

and we start it via the following command

python -m src.app

We recently started using gunicorn as follows

gunicorn --bind=127.0.0.1:5001 --workers=5 --access-logfile=- --log-level=debug 'src.app:app'

for 2 CPU 6 RAM

Before using gunicorn avg response time was 700~800 ms But, after using gunicorn, avg response time is 1100~1200 ms for single request only

Any suggestions why so?

Even If we set workers to 1, then also getting 1100~1200 ms