Actions

Learn how to produce records, wait for their replies, and measure the end-to-end latency of a pipeline.

Understand Kafka actions

Kafka actions start with kafka(requestName), where the request name is a Gatling Expression Language String or a function. Then pick what the action does:

Action Does Response time
send(topic) Produces a record. Until the broker acknowledges the record.
requestReply(requestTopic, replyTopic) Produces a record, then waits for its reply. Until the reply is read.
consume(topic) Waits for a record that another system produces. From when the action starts, or from a timestamp you provide, until the record is read.
subscribe(topic) Starts waiting for a record that another system produces, without blocking the virtual user, which collects it later with kafka.poll. From when the subscription starts, or from a timestamp you provide, until the record is read.

Topics are Gatling Expression Language Strings or functions.

The records still in flight when the run ends, sent but not acknowledged, or waiting for their reply or their record, are neither OK nor KO.

Produce records

send produces a record on a topic. Its value is required:

     
kafka("Place order")
  .send("orders")
  .value(StringBody("{\"id\": \"#{orderId}\", \"amount\": #{amount}}"));
kafka("Place order")
  .send("orders")
  .value(StringBody("""{"id": "#{orderId}", "amount": #{amount}}"""))
kafka("Place order")
  .send("orders")
  .value(StringBody("""{"id": "#{orderId}", "amount": #{amount}}"""))

The record also accepts:

Option Description
key The record’s key: a Gatling Expression Language String or a function, encoded as UTF-8, a Gatling body, or a schema registry body.
header(name, value) A header. The name and the value are Gatling Expression Language Strings, the value can also be a function. The value is encoded as UTF-8. Call it once per header.
partition The record’s partition: an integer, a Gatling Expression Language String, or a function. Without it, the producer’s partitioner picks the partition, typically from the key.
subscribe A subscription to a record that another system produces because of this one, tracked before the record is sent, see attach subscriptions. Call it once per subscription.
     
kafka("Place order")
  .send("orders")
  .value(ElFileBody("order.json"))
  // the key, encoded as UTF-8
  .key("#{orderId}")
  // headers, whose values are encoded as UTF-8
  .header("source", "gatling")
  .header("traceId", "#{traceId}")
  // the partition, else the producer's partitioner picks it
  .partition(0);
kafka("Place order")
  .send("orders")
  .value(ElFileBody("order.json"))
  // the key, encoded as UTF-8
  .key("#{orderId}")
  // headers, whose values are encoded as UTF-8
  .header("source", "gatling")
  .header("traceId", "#{traceId}")
  // the partition, else the producer's partitioner picks it
  .partition(0)
kafka("Place order")
  .send("orders")
  .value(ElFileBody("order.json"))
  // the key, encoded as UTF-8
  .key("#{orderId}")
  // headers, whose values are encoded as UTF-8
  .header("source", "gatling")
  .header("traceId", "#{traceId}")
  // the partition, else the producer's partitioner picks it
  .partition(0)

Set the value

The value is any Gatling body, as with HTTP request bodies. Text bodies are encoded with their charset, and byte bodies are sent as is:

     
// text, with the Gatling Expression Language or Pebble templates
kafka("Text").send("orders").value(StringBody("#{order}"));
kafka("Template").send("orders").value(ElFileBody("order.json"));
kafka("Pebble template").send("orders").value(PebbleFileBody("order.peb"));
// bytes, sent as is
kafka("File").send("payments").value(RawFileBody("payment.bin"));
kafka("Bytes").send("payments").value(ByteArrayBody("#{paymentBytes}"));
// a key from a body
kafka("Bytes key")
  .send("payments")
  .value(ByteArrayBody("#{paymentBytes}"))
  .key(ByteArrayBody("#{keyBytes}"));
// text, with the Gatling Expression Language or Pebble templates
kafka("Text").send("orders").value(StringBody("#{order}"))
kafka("Template").send("orders").value(ElFileBody("order.json"))
kafka("Pebble template").send("orders").value(PebbleFileBody("order.peb"))
// bytes, sent as is
kafka("File").send("payments").value(RawFileBody("payment.bin"))
kafka("Bytes").send("payments").value(ByteArrayBody("#{paymentBytes}"))
// a key from a body
kafka("Bytes key")
  .send("payments")
  .value(ByteArrayBody("#{paymentBytes}"))
  .key(ByteArrayBody("#{keyBytes}"))
// text, with the Gatling Expression Language or Pebble templates
kafka("Text").send("orders").value(StringBody("#{order}"))
kafka("Template").send("orders").value(ElFileBody("order.json"))
kafka("Pebble template").send("orders").value(PebbleFileBody("order.peb"))
// bytes, sent as is
kafka("File").send("payments").value(RawFileBody("payment.bin"))
kafka("Bytes").send("payments").value(ByteArrayBody("#{paymentBytes}"))
// a key from a body
kafka("Bytes key")
  .send("payments")
  .value(ByteArrayBody("#{paymentBytes}"))
  .key(ByteArrayBody("#{keyBytes}"))

The value can also be an Avro, Protobuf, or JSON Schema value, serialized with a schema registry.

Use functions

The request name, the topic, the key, the header values, and the partition can be functions of the session:

     
kafka(session -> "Send to " + session.getString("topic"))
  .send(session -> session.getString("topic"))
  .value(StringBody("#{payload}"))
  .key(session -> session.getString("id"))
  .header("type", session -> session.getString("type"))
  .partition(session -> session.getInt("partition"));
kafka { session -> "Send to " + session.getString("topic") }
  .send { session -> session.getString("topic") }
  .value(StringBody("#{payload}"))
  .key { session -> session.getString("id") }
  .header("type") { session -> session.getString("type") }
  .partition { session -> session.getInt("partition") }
kafka(session => "Send to " + session("topic").as[String])
  .send(session => session("topic").as[String])
  .value(StringBody("#{payload}"))
  .key(session => session("id").as[String])
  .header("type", session => session("type").as[String])
  .partition(session => session("partition").as[Int])

Understand what Gatling measures

The response time of send lasts from when the virtual user sends the record until the broker acknowledges it, as the producer’s acks property defines. It includes:

  • waiting for a sender thread
  • the serialization with the schema registry, if any: the first record of a schema also registers or looks up the schema, which shows as an outlier in the max response time
  • the producer’s batching, see linger.ms
  • sending the record to the broker and waiting for the acknowledgement

A record that the producer fails to send fails the request, with the cause.

Send a request and wait for its reply

requestReply produces a request on the request topic, then waits for the reply that the system under test produces on the reply topic. It accepts the same value, key, header, and partition as send.

     
kafka("Check order")
  .requestReply("order-requests", "order-replies")
  .value(StringBody("{\"id\": \"#{orderId}\"}"))
  .key("#{orderId}")
  .check(jsonPath("$.status").is("ACCEPTED"))
  // overrides the protocol's replyTimeout
  .replyTimeout(Duration.ofSeconds(5));
kafka("Check order")
  .requestReply("order-requests", "order-replies")
  .value(StringBody("""{"id": "#{orderId}"}"""))
  .key("#{orderId}")
  .check(jsonPath("$.status").shouldBe("ACCEPTED"))
  // overrides the protocol's replyTimeout
  .replyTimeout(Duration.ofSeconds(5))
kafka("Check order")
  .requestReply("order-requests", "order-replies")
  .value(StringBody("""{"id": "#{orderId}"}"""))
  .key("#{orderId}")
  .check(jsonPath("$.status").is("ACCEPTED"))
  // overrides the protocol's replyTimeout
  .replyTimeout(5.seconds)

The request-reply also accepts:

Option Description
check Checks on the reply.
replyTimeout How long to wait for the reply, overriding the protocol’s. The Java and Kotlin replyTimeout also takes a number of seconds.
decodeAvroReply, decodeProtobufReply, decodeJsonSchemaReply Decode the reply with the schema registry before the checks.
subscribe A subscription to another record that the system produces because of this request, besides its reply, tracked before the request is sent, see attach subscriptions.

Gatling matches the reply to its request with the correlation id that the protocol defines: by default, the reply has the request’s key. See how to match replies to their requests.

The response time of a request-reply lasts from when the request is sent until its reply is read. Checks then run on the reply, outside of the response time. A reply that doesn’t come within the reply timeout fails the request.

Before sending the first request on a reply topic, Gatling waits for its consumer to be ready to read the replies on this topic. This wait isn’t part of the response time.

Wait for a record from another system

consume doesn’t produce anything: it blocks the virtual user until a record that another system produces on a topic is read, typically a pipeline that processes a record sent earlier, and measures the end-to-end latency. correlationId provides the correlation id of the record to wait for, a Gatling Expression Language String or a function, matched as the protocol defines, like a reply.

     
ChainBuilder placeOrder =
  // the start of the end-to-end latency, in epoch milliseconds
  exec(session -> session.set("sentAt", System.currentTimeMillis()))
    .exec(
      kafka("Place order")
        .send("orders")
        .value(StringBody("{\"id\": \"#{orderId}\"}"))
        .key("#{orderId}")
    )
    .exec(
      kafka("Order enriched")
        .consume("enriched-orders")
        // matched with the protocol's matcher, here the record's key
        .correlationId("#{orderId}")
        .check(jsonPath("$.status").is("ENRICHED"))
        // overrides the protocol's replyTimeout
        .timeout(Duration.ofSeconds(30))
        .startTimestampFrom("sentAt")
    );
val placeOrder =
  // the start of the end-to-end latency, in epoch milliseconds
  exec { session -> session.set("sentAt", System.currentTimeMillis()) }
    .exec(
      kafka("Place order")
        .send("orders")
        .value(StringBody("""{"id": "#{orderId}"}"""))
        .key("#{orderId}")
    )
    .exec(
      kafka("Order enriched")
        .consume("enriched-orders")
        // matched with the protocol's matcher, here the record's key
        .correlationId("#{orderId}")
        .check(jsonPath("$.status").shouldBe("ENRICHED"))
        // overrides the protocol's replyTimeout
        .timeout(Duration.ofSeconds(30))
        .startTimestampFrom("sentAt")
    )
val placeOrder =
  // the start of the end-to-end latency, in epoch milliseconds
  exec(session => session.set("sentAt", System.currentTimeMillis()))
    .exec(
      kafka("Place order")
        .send("orders")
        .value(StringBody("""{"id": "#{orderId}"}"""))
        .key("#{orderId}")
    )
    .exec(
      kafka("Order enriched")
        .consume("enriched-orders")
        // matched with the protocol's matcher, here the record's key
        .correlationId("#{orderId}")
        .check(jsonPath("$.status").is("ENRICHED"))
        // overrides the protocol's replyTimeout
        .timeout(30.seconds)
        .startTimestampFrom("sentAt")
    )

The consume also accepts:

Option Description
check Checks on the record.
timeout How long to wait for the record, overriding the protocol’s replyTimeout. The Java and Kotlin timeout also takes a number of seconds.
startTimestampFrom(attributeName) The session attribute holding the start of the response time, in epoch milliseconds.

By default, the response time lasts from when the action starts until the record is read. With startTimestampFrom, it starts at the timestamp in the session attribute instead, typically saved right before sending the record that the pipeline processes. A record read before this timestamp is ignored, as it’s an older record with the same correlation id.

The timeout counts from when the virtual user starts waiting. A record that doesn’t come within the timeout fails the request.

The record may be read before the virtual user starts waiting for it, while it runs other actions: Gatling keeps the records that no virtual user waits for yet, by default for up to 60 seconds and up to 10,000 records, the oldest being dropped first. On a busy topic, 10,000 records may cover less time than the virtual user takes to start waiting, and the record it waits for is dropped before: the consume times out although the record came. Gatling then logs a warning naming the protocol’s consumeBuffer option, which raises these limits.

When the topic is a static String, Gatling reads it from when the simulation starts. When it’s a Gatling Expression Language String or a function, Gatling reads it from when a virtual user first waits on it.

For a record that comes from a record or a request that Gatling sends, prefer a subscription attached to the send: it doesn’t depend on the kept records, and it doesn’t need startTimestampFrom.

Subscribe to records without blocking

subscribe starts waiting for a record that another system produces on a topic, and lets the virtual user continue right away, for example with the HTTP request that makes the system produce the record. Later, kafka.poll (kafka.poll() in Java and Kotlin) waits for the virtual user’s subscriptions to complete. correlationId provides the correlation id of the record, a Gatling Expression Language String or a function, matched as the protocol defines, like a reply.

     
ChainBuilder checkout =
  exec(
    // before the HTTP request that makes the system produce the record
    kafka("Order shipped")
      .subscribe("order-events")
      // matched with the protocol's matcher, here the record's key
      .correlationId("#{orderId}")
      .check(jsonPath("$.trackingNumber").saveAs("trackingNumber"))
      // overrides the protocol's replyTimeout
      .timeout(Duration.ofSeconds(30)),
    http("Checkout").post("/orders/#{orderId}/checkout"),
    http("Browse").get("/products"),
    // waits for the subscriptions, then applies what their checks saved
    kafka.poll(),
    http("Track").get("/tracking/#{trackingNumber}")
  );
val checkout =
  exec(
    // before the HTTP request that makes the system produce the record
    kafka("Order shipped")
      .subscribe("order-events")
      // matched with the protocol's matcher, here the record's key
      .correlationId("#{orderId}")
      .check(jsonPath("$.trackingNumber").saveAs("trackingNumber"))
      // overrides the protocol's replyTimeout
      .timeout(Duration.ofSeconds(30)),
    http("Checkout").post("/orders/#{orderId}/checkout"),
    http("Browse").get("/products"),
    // waits for the subscriptions, then applies what their checks saved
    kafka.poll(),
    http("Track").get("/tracking/#{trackingNumber}")
  )
val checkout =
  exec(
    // before the HTTP request that makes the system produce the record
    kafka("Order shipped")
      .subscribe("order-events")
      // matched with the protocol's matcher, here the record's key
      .correlationId("#{orderId}")
      .check(jsonPath("$.trackingNumber").saveAs("trackingNumber"))
      // overrides the protocol's replyTimeout
      .timeout(30.seconds),
    http("Checkout").post("/orders/#{orderId}/checkout"),
    http("Browse").get("/products"),
    // waits for the subscriptions, then applies what their checks saved
    kafka.poll,
    http("Track").get("/tracking/#{trackingNumber}")
  )

The subscription accepts the same options as consume: check, timeout, and startTimestampFrom.

Place a subscription before what makes its record

A subscription is tracked before the virtual user continues, once Gatling’s consumer reads the topic, so the record can’t come before the subscription. Place it before the action that makes the system produce the record, such as an HTTP request.

A subscription placed after this action may miss its record, as for a consume: it only finds the record among the records Gatling keeps, see consumeBuffer.

Attach subscriptions to a send or a request-reply

send and requestReply also accept subscribe, with a request name, a topic, and a correlation id, or with a subscription built with kafka(requestName).subscribe(topic), to set checks and a timeout. Gatling tracks the attached subscriptions before sending anything, so the record can’t come first. Call subscribe once per subscription.

     
ChainBuilder pipeline =
  exec(
    kafka("Place order")
      .send("orders")
      .value(StringBody("{\"id\": \"#{orderId}\"}"))
      .key("#{orderId}")
      // the request name, the topic, and the correlation id
      .subscribe("Order enriched", "enriched-orders", "#{orderId}")
      // or a subscription with checks and a timeout
      .subscribe(
        kafka("Order billed")
          .subscribe("billing-events")
          .correlationId("#{orderId}")
          .check(jsonPath("$.status").is("BILLED"))
          .timeout(Duration.ofSeconds(30))
      ),
    kafka("Check order")
      .requestReply("order-requests", "order-replies")
      .value(StringBody("{\"id\": \"#{orderId}\"}"))
      .key("#{orderId}")
      .subscribe("Order shipped", "order-events", "#{orderId}"),
    kafka.poll()
  );
val pipeline =
  exec(
    kafka("Place order")
      .send("orders")
      .value(StringBody("""{"id": "#{orderId}"}"""))
      .key("#{orderId}")
      // the request name, the topic, and the correlation id
      .subscribe("Order enriched", "enriched-orders", "#{orderId}")
      // or a subscription with checks and a timeout
      .subscribe(
        kafka("Order billed")
          .subscribe("billing-events")
          .correlationId("#{orderId}")
          .check(jsonPath("$.status").shouldBe("BILLED"))
          .timeout(Duration.ofSeconds(30))
      ),
    kafka("Check order")
      .requestReply("order-requests", "order-replies")
      .value(StringBody("""{"id": "#{orderId}"}"""))
      .key("#{orderId}")
      .subscribe("Order shipped", "order-events", "#{orderId}"),
    kafka.poll()
  )
val pipeline =
  exec(
    kafka("Place order")
      .send("orders")
      .value(StringBody("""{"id": "#{orderId}"}"""))
      .key("#{orderId}")
      // the request name, the topic, and the correlation id
      .subscribe("Order enriched", "enriched-orders", "#{orderId}")
      // or a subscription with checks and a timeout
      .subscribe(
        kafka("Order billed")
          .subscribe("billing-events")
          .correlationId("#{orderId}")
          .check(jsonPath("$.status").is("BILLED"))
          .timeout(30.seconds)
      ),
    kafka("Check order")
      .requestReply("order-requests", "order-replies")
      .value(StringBody("""{"id": "#{orderId}"}"""))
      .key("#{orderId}")
      .subscribe("Order shipped", "order-events", "#{orderId}"),
    kafka.poll
  )

The response time of an attached subscription lasts from when it starts, right before the record or the request is sent, until the subscription’s record is read: the end-to-end latency of the pipeline, without startTimestampFrom. Waiting for Gatling’s consumer to read the subscription’s topic isn’t part of the response time of the send or the requestReply. Prefer this form to measure Kafka-to-Kafka pipelines.

Understand what Gatling measures and when the session changes

The response time of a subscription lasts from when it starts, or from the timestamp of startTimestampFrom, until its record is read. Gatling logs it then, or when the subscription times out, whether or not the virtual user polls: polling doesn’t change the response time. The timeout counts from when the subscription starts waiting.

A subscription’s checks run when its record is read, on the session as it was when the subscription started. What they save, and their failures, only apply to the virtual user’s session at kafka.poll, which waits for every pending subscription to complete, by its record or by its timeout. A virtual user that doesn’t poll keeps its session as is, but its subscriptions are still logged.

Gatling logs a warning, once per run, when a virtual user ends with completed subscriptions it never polled, or holds more than 1,000 of them in memory. Add kafka.poll after the subscriptions, unless you only want their response times.

A virtual user that ends with pending subscriptions logs them as failed, with Virtual user ended before its record.

Edit this page on GitHub