Assertions

Learn how to set assertions on metrics like response time or number of failed requests, and export these results to a JUnit compatible format

Concepts#

The Assertions API is used to verify that global statistics, like response time or number of failed requests, match expectations for a whole simulation.

Assertions are registered for a simulation using the method assertions on the setUp. For example:

setUp(scn.injectOpen(injectionProfile))
  .assertions(
    global().responseTime().max().lt(50),
    global().successfulRequests().percent().gt(95.0)
  );

setUp(scn.injectOpen(injectionProfile))
  .assertions(
    global().responseTime().max().lt(50),
    global().successfulRequests().percent().gt(95.0)
  );

setUp(scn.injectOpen(injectionProfile))
  .assertions(
    global().responseTime().max().lt(50),
    global().successfulRequests().percent().gt(95.0)
  )

setUp(scn.inject(injectionProfile))
  .assertions(
    global.responseTime.max.lt(50),
    global.successfulRequests.percent.gt(95)
  )

This method takes as many assertions as you like.

The API provides a dedicated DSL for chaining the following steps:

1. defining the scope of the assertion
2. selecting the statistic
3. selecting the metric
4. defining the condition

All the assertions are evaluated after running the simulation. If at least one assertion fails, the simulation fails.

Scope#

An assertion can test a statistic calculated from all requests or only part of them.

• global: use statistics calculated from all requests.
• forAll: use statistics calculated for each individual request.
• details(path): use statistics calculated from a group or a request. The path is defined like a Unix filesystem path.

For example, to perform an assertion on the request MyRequest, use:

details("MyRequest");

details("MyRequest");

details("MyRequest")

details("MyRequest")

and to perform an assertion on the request MyRequest in the group MyGroup, use:

details("MyGroup", "MyRequest");

details("MyGroup", "MyRequest");

details("MyGroup", "MyRequest")

details("MyGroup" / "MyRequest")

For WebSockets it takes the name of the check and not the name of the request. ws.checkTextMessage("use this name")

Statistics#

• responseTime: target the response time in milliseconds.
• allRequests: target the number of requests.
• failedRequests: target the number of failed requests.
• successfulRequests: target the number of successful requests.
• requestsPerSec: target the rate of requests per second.

Selecting the metric#

Applicable to response time#

• min: perform the assertion on the minimum of the metric.
• max: perform the assertion on the maximum of the metric.
• mean: perform the assertion on the mean of the metric.
• stdDev: perform the assertion on the standard deviation of the metric.
• percentile1: perform the assertion on the 1st percentile of the metric, as configured in gatling.conf (default is 50th).
• percentile2: perform the assertion on the 2nd percentile of the metric, as configured in gatling.conf (default is 75th).
• percentile3: perform the assertion on the 3rd percentile of the metric, as configured in gatling.conf (default is 95th).
• percentile4: perform the assertion on the 4th percentile of the metric, as configured in gatling.conf (default is 99th).
• percentile(value: Double): perform the assertion on the given percentile of the metric. Parameter is a percentage, between 0 and 100.

Applicable to number of requests (all, failed or successful)#

• percent: use the value as a percentage between 0 and 100.
• count: perform the assertion directly on the count of requests.

Condition#

Conditions can be chained to apply several conditions on the same metric.

• lt(threshold): check that the value of the metric is less than the threshold.
• lte(threshold): check that the value of the metric is less than or equal to the threshold.
• gt(threshold): check that the value of the metric is greater than the threshold.
• gte(threshold): check that the value of the metric is greater than or equal to the threshold.
• between(thresholdMin, thresholdMax): check that the value of the metric is between two thresholds.
• between(thresholdMin, thresholdMax, inclusive = false): same as above but doesn’t include bounds
• around(value, plusOrMinus): check that the value of the metric is around a target value plus or minus a given margin.
• around(value, plusOrMinus, inclusive = false): same as above but doesn’t include bounds
• deviatesAround(target, percentDeviationThreshold): check that metric is around a target value plus or minus a given relative margin
• deviatesAround(target, percentDeviationThreshold, inclusive = false): same as above but doesn’t include bounds
• is(value): check that the value of the metric is equal to the given value.
• in(sequence): check that the value of metric is in a sequence.

Putting it all together#

To help you understand how to use assertions, here is a list of examples:

// Assert that the max response time of all requests is less than 100 ms
setUp(scn.injectOpen(injectionProfile))
  .assertions(global().responseTime().max().lt(100));

// Assert that every request has no more than 5% of failing requests
setUp(scn.injectOpen(injectionProfile))
  .assertions(forAll().failedRequests().percent().lte(5.0));

// Assert that the percentage of failed requests named "MyRequest" in the group "MyGroup"
// is exactly 0 %
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup", "MyRequest").failedRequests().percent().is(0.0));

// Assert that the rate of requests per seconds for the group "MyGroup"
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup").requestsPerSec().between(100.0, 1000.0));

// Assert that the max response time of all requests is less than 100 ms
setUp(scn.injectOpen(injectionProfile))
  .assertions(global().responseTime().max().lt(100));

// Assert that every request has no more than 5% of failing requests
setUp(scn.injectOpen(injectionProfile))
  .assertions(forAll().failedRequests().percent().lte(5.0));

// Assert that the percentage of failed requests named "MyRequest" in the group "MyGroup"
// is exactly 0 %
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup", "MyRequest").failedRequests().percent().is(0.0));

// Assert that the rate of requests per seconds for the group "MyGroup"
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup").requestsPerSec().between(100.0, 1000.0));

// Assert that the max response time of all requests is less than 100 ms
setUp(scn.injectOpen(injectionProfile))
  .assertions(global().responseTime().max().lt(100))

// Assert that every request has no more than 5% of failing requests
setUp(scn.injectOpen(injectionProfile))
  .assertions(forAll().failedRequests().percent().lte(5.0))

// Assert that the percentage of failed requests named "MyRequest" in the group "MyGroup"
// is exactly 0 %
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup", "MyRequest").failedRequests().percent().shouldBe(0.0))

// Assert that the rate of requests per seconds for the group "MyGroup"
setUp(scn.injectOpen(injectionProfile))
  .assertions(details("MyGroup").requestsPerSec().between(100.0, 1000.0))

// Assert that the max response time of all requests is less than 100 ms
setUp(scn.inject(injectionProfile))
  .assertions(global.responseTime.max.lt(100))

// Assert that every request has no more than 5% of failing requests
setUp(scn.inject(injectionProfile))
  .assertions(forAll.failedRequests.percent.lte(5))

// Assert that the percentage of failed requests named "MyRequest" in the group "MyGroup"
// is exactly 0 %
setUp(scn.inject(injectionProfile))
  .assertions(details("MyGroup" / "MyRequest").failedRequests.percent.is(0))

// Assert that the rate of requests per seconds for the group "MyGroup"
setUp(scn.inject(injectionProfile))
  .assertions(details("MyGroup").requestsPerSec.between(100, 1000))

Edit this page on GitHub