Gatling vs Jmeter - Getting huge difference in response time per request

Viewed 3087

I have created exactly the same demo scenarios (Landing on the website homepage) using Firefox and used Gatling and Jmeter to test the tools.

When I execute both of them (one by one) with 500 VU for 30 sec, I am getting huge difference between the response time (800 ms with Gatling and over 3000 ms with Jmeter) and JMeter is showing failures of over 29% while Gatling shows only 1%

Which one to trust?

As the application is still under development, Gatling seems to be unreliable as we are aware of response time is approx 2.5 seconds usually.

4 Answers

In response to @DmitriT point 1. and 2.

With gatling there is another way of downloading resources in a more controlled way: User the .resources() method, and list as many http("") requests as you want, they will all be executed in parallel. And to make it more realistic, combine it with HttpProtocolBuilder settings like : .maxConnectionsPerHostLikeChrome to be closer to reality. I use this approach since my resources might also have dynamic links got from previous requests, ex :

With gatling there is another way of downloading resources in a more controlled way:  User the **.resources()** method, and list as many http("") requests as you want, they will all be executed in parallel. And to make it more realistic, combine it with HttpProtocolBuilder settings like : **.maxConnectionsPerHostLikeChrome** to be closer to reality.    I use this approach since my resources might also have dynamic links got from previous requests, ex :  `tryMax(1) {
  group("<-- EMP : Login Page -->") {
    exec(
      http("EMP : Login Page - login.html ")
        .get("/admin-ng/login.html")
        .check(regex("/admin-ng/scripts/login.[^\"]*.js").find.saveAs("login_js"))
        .check(regex("/admin-ng/login.[^\"]*.css").find.saveAs("login_css"))
        .check(regex("/admin-ng/scripts/loginVendors.[^\"]*.js").find.saveAs("loginVendors_js"))
        .resources(
          http("EMP : Login Page => Asset 1 - login.xxx.css")
            .get(session => session("login_css").as[String])
            .check(regex("([^\\.]*?\\.woff2)").findAll.saveAs("listOfFontsWoff2"))
            .check(regex("([^\\.]*?\\.woff(?=[^2]))").findAll.saveAs("listOfFontsWoff")),
          http("EMP : Login Page => Asset 2 - login.xxx.js")
            .get(session => session("login_js").as[String]),
          http("EMP : Login Page => Asset 3 - loginVendors.xxx.js")
            .get(session => session("loginVendors_js").as[String])
        )
    )
      .exec(
        http("")
          .get("")
          .silent
          .resources(
            http("EMP : Login Page => Asset 4 - .min.js")
              .get("/nr-spa-974.min.js"),
            http("EMP : Login Page => Asset 5 - woff2 (index: 3)")
              .get(session => session("listOfFontsWoff2").as[Seq[String]].apply(3)),
            http("EMP : Login Page => Asset 6 - woff2 (index: 2)")
              .get(session => session("listOfFontsWoff2").as[Seq[String]].apply(2)),
            http("EMP : Login Page => Asset 7 - woff2 (index: 5)")
              .get(session => session("listOfFontsWoff2").as[Seq[String]].apply(5))
          )
      )
  }
}`

This code executes in 2 phases same as in the browser:

  1. The main link to the webpage, which contains some link to the JS and CSS resources, i extract them, and in a .resources() method download them in parallel.
  2. Then the next phase i download the resources that are extracted from the previous resources, in this case from login_css

This way i get pretty close timings to the real browser (ofc it's never 100% accurate, regardless of the tool, every one of them will lack something in a particular area which you will face eventually)

And remember to use in Gatling the group() method, as it will count all the requests and resource parallel requests under it as one entity (like a web page loading from a to z) with the option in the report to see sub-requests of the group separately if needed

Related