Kafka Protocol
Learn how to configure the Kafka protocol, secure the connections to the brokers, and match replies to their requests.
Add the imports
The Gatling Kafka SDK isn’t imported by default. Add the following imports:
import static io.gatling.javaapi.kafka.KafkaDsl.*;import io.gatling.javaapi.kafka.KafkaDsl.*import io.gatling.kafka.Predef._Configure the protocol
Use the kafka object to create a Kafka protocol, then register it like any other protocol:
KafkaProtocolBuilder kafkaProtocol = kafka
.bootstrapServers("broker1:9092", "broker2:9092");
ScenarioBuilder scn = scenario("Kafka")
.exec(
kafka("Place order")
.send("orders")
.value(StringBody("{\"id\": \"#{orderId}\"}"))
.key("#{orderId}")
);
setUp(
scn.injectOpen(constantUsersPerSec(10).during(60))
).protocols(kafkaProtocol);val kafkaProtocol = kafka
.bootstrapServers("broker1:9092", "broker2:9092")
val scn = scenario("Kafka")
.exec(
kafka("Place order")
.send("orders")
.value(StringBody("""{"id": "#{orderId}"}"""))
.key("#{orderId}")
)
setUp(
scn.injectOpen(constantUsersPerSec(10.0).during(60))
).protocols(kafkaProtocol)val kafkaProtocol = kafka
.bootstrapServers("broker1:9092", "broker2:9092")
val scn = scenario("Kafka")
.exec(
kafka("Place order")
.send("orders")
.value(StringBody("""{"id": "#{orderId}"}"""))
.key("#{orderId}")
)
setUp(
scn.inject(constantUsersPerSec(10).during(60))
).protocols(kafkaProtocol)The protocol accepts the following options:
| Option | Default | Description |
|---|---|---|
bootstrapServers |
The brokers to bootstrap from, as host:port. Required. |
|
property, properties |
none | Kafka client properties for the producers and the reply consumer. |
producerProperty, producerProperties |
none | Kafka producer properties, overriding the shared ones. |
consumerProperty, consumerProperties |
none | Kafka consumer properties for the reply consumer, overriding the shared ones. |
saslPlain, saslScramSha256, saslScramSha512, saslOauthBearer, saslAwsMskIam, saslAwsMskIamProfile, saslAwsMskIamRole, saslGssapi |
none | The SASL authentication. |
useTls, trustStore, keyStore |
no TLS | The TLS configuration. |
matchByKey, matchByHeader, matchByValue |
matchByKey |
How a reply is matched to its request. |
replyTimeout |
60 seconds | How long a request waits for its reply, and a consume or a subscription for its record, unless they override it. |
consumeBuffer |
10,000 records, 60 seconds | The records kept for a consume or a subscription that starts after its record is read. |
schemaRegistry, schemaRegistryBasicAuth, schemaRegistryProperty, schemaRegistryProperties |
none | The schema registry of the Avro, Protobuf, and JSON Schema keys and values. |
Set Kafka client properties
The protocol passes Kafka client properties to the producers and to the reply consumer. Refer to the producer and consumer configurations of Kafka for the available properties.
kafka
.bootstrapServers("broker1:9092")
// for the producers and the reply consumer
.property("client.id", "gatling")
.properties(Map.of("request.timeout.ms", "10000"))
// for the producers only, overriding the shared properties
.producerProperty("acks", "all")
.producerProperties(Map.of("compression.type", "lz4"))
// for the reply consumer only, overriding the shared properties
.consumerProperty("fetch.max.wait.ms", "100")
.consumerProperties(Map.of("max.poll.records", "1000"));kafka
.bootstrapServers("broker1:9092")
// for the producers and the reply consumer
.property("client.id", "gatling")
.properties(mapOf("request.timeout.ms" to "10000"))
// for the producers only, overriding the shared properties
.producerProperty("acks", "all")
.producerProperties(mapOf("compression.type" to "lz4"))
// for the reply consumer only, overriding the shared properties
.consumerProperty("fetch.max.wait.ms", "100")
.consumerProperties(mapOf("max.poll.records" to "1000"))kafka
.bootstrapServers("broker1:9092")
// for the producers and the reply consumer
.property("client.id", "gatling")
.properties(Map("request.timeout.ms" -> "10000"))
// for the producers only, overriding the shared properties
.producerProperty("acks", "all")
.producerProperties(Map("compression.type" -> "lz4"))
// for the reply consumer only, overriding the shared properties
.consumerProperty("fetch.max.wait.ms", "100")
.consumerProperties(Map("max.poll.records" -> "1000"))The plugin sets some properties itself, and refuses them right away:
bootstrap.servers: usebootstrapServersinsteadkey.serializer,value.serializer,key.deserializer, andvalue.deserializer: the plugin turns keys and values into bytes itself, from Gatling bodies or with a schema registrygroup.id,group.instance.id, andgroup.protocol: the reply consumer doesn’t join a consumer group and commits no offsets, see how replies are read
Secure the connections
The typed security options below generate the Kafka client properties for SASL and TLS: security.protocol, sasl.mechanism, sasl.jaas.config, and the sasl.* and ssl.* properties of each mechanism.
They apply to the producers and to the reply consumer alike.
The plugin picks security.protocol from the options you set:
| Options | security.protocol |
|---|---|
| A SASL mechanism, without TLS | SASL_PLAINTEXT |
| A SASL mechanism, with TLS | SASL_SSL |
saslAwsMskIam, saslAwsMskIamProfile, saslAwsMskIamRole |
SASL_SSL, always |
| TLS only | SSL |
The protocol uses a single SASL mechanism, a single trust store, and a single key store: setting a second one fails right away.
The Kafka clients are shared by all the virtual users, so the credentials are plain values, resolved once for the whole run, not Gatling Expression Language Strings.
security.protocol, sasl.mechanism, or sasl.jaas.config: the simulation fails at startup.
Use either the typed option or the raw properties for a given key.Raw properties for keys the typed options don’t generate complete them:
kafka.bootstrapServers("broker1:9093")
.saslScramSha512("user", "password")
.trustStore("/path/to/truststore.jks", "changeit")
// a key that the typed options don't generate
.property("ssl.enabled.protocols", "TLSv1.3");kafka.bootstrapServers("broker1:9093")
.saslScramSha512("user", "password")
.trustStore("/path/to/truststore.jks", "changeit")
// a key that the typed options don't generate
.property("ssl.enabled.protocols", "TLSv1.3")kafka.bootstrapServers("broker1:9093")
.saslScramSha512("user", "password")
.trustStore("/path/to/truststore.jks", "changeit")
// a key that the typed options don't generate
.property("ssl.enabled.protocols", "TLSv1.3")Authenticate with SASL
saslPlain, saslScramSha256, and saslScramSha512 authenticate with a username and a password, with the PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 mechanisms:
// SASL PLAIN over TLS: security.protocol is SASL_SSL
kafka.bootstrapServers("broker1:9093")
.saslPlain("user", "password")
.useTls();
// SCRAM without TLS: security.protocol is SASL_PLAINTEXT
kafka.bootstrapServers("broker1:9092")
.saslScramSha256("user", "password");
kafka.bootstrapServers("broker1:9092")
.saslScramSha512("user", "password");// SASL PLAIN over TLS: security.protocol is SASL_SSL
kafka.bootstrapServers("broker1:9093")
.saslPlain("user", "password")
.useTls()
// SCRAM without TLS: security.protocol is SASL_PLAINTEXT
kafka.bootstrapServers("broker1:9092")
.saslScramSha256("user", "password")
kafka.bootstrapServers("broker1:9092")
.saslScramSha512("user", "password")// SASL PLAIN over TLS: security.protocol is SASL_SSL
kafka.bootstrapServers("broker1:9093")
.saslPlain("user", "password")
.useTls
// SCRAM without TLS: security.protocol is SASL_PLAINTEXT
kafka.bootstrapServers("broker1:9092")
.saslScramSha256("user", "password")
kafka.bootstrapServers("broker1:9092")
.saslScramSha512("user", "password")Authenticate with OAuth
saslOauthBearer authenticates with the OAUTHBEARER mechanism.
The Kafka client gets its tokens from the OAuth token endpoint with the client credentials grant, with an optional scope.
kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret")
.useTls();
// with a scope
kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret", "kafka")
.useTls();kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret")
.useTls()
// with a scope
kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret", "kafka")
.useTls()kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret")
.useTls
// with a scope
kafka.bootstrapServers("broker1:9093")
.saslOauthBearer("https://idp.example.com/oauth2/token", "client-id", "client-secret", "kafka")
.useTlsorg.apache.kafka.sasl.oauthbearer.allowed.urls.
The plugin adds the protocol’s token endpoint to this property, keeping the endpoints already listed.
As it’s a system property, it applies to the whole JVM.Authenticate to Amazon MSK with IAM
The plugin authenticates to Amazon MSK with IAM with one of these options:
saslAwsMskIam: with the credentials of the AWS default credentials provider chainsaslAwsMskIamProfile(profileName): with the credentials of a named profile of the AWS credentials filessaslAwsMskIamRole(roleArn): by assuming a role through AWS STS, with the credentials of the default credentials provider chain, typically for a cluster in another AWS account. It optionally takes the role session’s name, the external id that the role’s trust policy requires, and the region of the STS endpoint, in this order.
MSK only serves IAM authentication over TLS, so the plugin always uses TLS with it, trusting the JVM’s default certificates unless you set a trust store.
// the AWS default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIam();
// a named profile of the AWS credentials files
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamProfile("load-tests");
// a role assumed with the default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole("arn:aws:iam::123456789012:role/gatling-msk");
// with a session name, an external id, and the STS endpoint's region
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole(
"arn:aws:iam::123456789012:role/gatling-msk", "gatling", "my-external-id", "eu-west-1");// the AWS default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIam()
// a named profile of the AWS credentials files
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamProfile("load-tests")
// a role assumed with the default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole("arn:aws:iam::123456789012:role/gatling-msk")
// with a session name, an external id, and the STS endpoint's region
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole(
"arn:aws:iam::123456789012:role/gatling-msk", "gatling", "my-external-id", "eu-west-1")// the AWS default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIam
// a named profile of the AWS credentials files
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamProfile("load-tests")
// a role assumed with the default credentials provider chain
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole("arn:aws:iam::123456789012:role/gatling-msk")
// with a session name, an external id, and the STS endpoint's region
kafka.bootstrapServers("b-1.mycluster.abc123.c2.kafka.eu-west-1.amazonaws.com:9098")
.saslAwsMskIamRole(
"arn:aws:iam::123456789012:role/gatling-msk", "gatling", "my-external-id", "eu-west-1")It needs the Amazon MSK Library for AWS IAM, software.amazon.msk:aws-msk-iam-auth, which the plugin doesn’t bring:
add it in the same scope as the Gatling dependencies. Without it, the simulation fails at startup.
Authenticate with Kerberos
saslGssapi authenticates with the GSSAPI mechanism, logging in to Kerberos from a keytab.
It takes the Kerberos principal name the brokers run as, the path of the client’s keytab, and the client’s principal.
kafka.bootstrapServers("broker1:9093")
// the brokers' service name, the client's keytab and principal
.saslGssapi("kafka", "/etc/security/gatling.keytab", "gatling@EXAMPLE.COM")
.useTls();kafka.bootstrapServers("broker1:9093")
// the brokers' service name, the client's keytab and principal
.saslGssapi("kafka", "/etc/security/gatling.keytab", "gatling@EXAMPLE.COM")
.useTls()kafka.bootstrapServers("broker1:9093")
// the brokers' service name, the client's keytab and principal
.saslGssapi("kafka", "/etc/security/gatling.keytab", "gatling@EXAMPLE.COM")
.useTlsUse TLS
useTls connects to the brokers over TLS, trusting the JVM’s default certificates.
trustStore trusts the certificates of a store instead, and keyStore authenticates the client with the certificate and key of a store, for mutual TLS. Both imply TLS.
// trust the JVM's default certificates
kafka.bootstrapServers("broker1:9093")
.useTls();
// trust the certificates of a trust store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/truststore.jks", "changeit");
// mutual TLS, with the type of each store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/ca.pem", "", "PEM")
.keyStore("/path/to/client.p12", "changeit", "PKCS12");// trust the JVM's default certificates
kafka.bootstrapServers("broker1:9093")
.useTls()
// trust the certificates of a trust store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/truststore.jks", "changeit")
// mutual TLS, with the type of each store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/ca.pem", "", "PEM")
.keyStore("/path/to/client.p12", "changeit", "PKCS12")// trust the JVM's default certificates
kafka.bootstrapServers("broker1:9093")
.useTls
// trust the certificates of a trust store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/truststore.jks", "changeit")
// mutual TLS, with the type of each store
kafka.bootstrapServers("broker1:9093")
.trustStore("/path/to/ca.pem", "", "PEM")
.keyStore("/path/to/client.p12", "changeit", "PKCS12")Stores take a path, a password, empty for a store without one such as PEM, and an optional type: JKS, PKCS12, or PEM.
Without a type, Kafka uses its default type, JKS.
The key store’s password is also the key’s.
Match replies to their requests
A request-reply sends a request, then waits for its reply, which the system under test produces on a reply topic. The request and its reply carry the same correlation id, so that Gatling can tell which request a reply answers. The protocol defines where this id is:
// the reply has the request's key (default)
kafka.bootstrapServers("broker1:9092")
.matchByKey();
// the reply has the request's header with this name
kafka.bootstrapServers("broker1:9092")
.matchByHeader("correlationId");
// the reply's value has the request's id where the check finds it
kafka.bootstrapServers("broker1:9092")
.matchByValue(jsonPath("$.orderId"));// the reply has the request's key (default)
kafka.bootstrapServers("broker1:9092")
.matchByKey()
// the reply has the request's header with this name
kafka.bootstrapServers("broker1:9092")
.matchByHeader("correlationId")
// the reply's value has the request's id where the check finds it
kafka.bootstrapServers("broker1:9092")
.matchByValue(jsonPath("$.orderId"))// the reply has the request's key (default)
kafka.bootstrapServers("broker1:9092")
.matchByKey
// the reply has the request's header with this name
kafka.bootstrapServers("broker1:9092")
.matchByHeader("correlationId")
// the reply's value has the request's id where the check finds it
kafka.bootstrapServers("broker1:9092")
.matchByValue(jsonPath("$.orderId"))matchByKey, the default: the reply has the request’s key. A request without a key fails.matchByHeader(name): the reply has the request’s header with this name. When the request doesn’t set this header, Gatling sets it to a random UUID.matchByValue(check): the request and the reply carry the id in their value, where the check finds it, typically ajsonPathor aregexcheck. The ids are compared as text. A request whose value has no id fails, and a reply whose value has none is ignored.
The system under test must copy the correlation id from the request to its reply.
The same matching applies to the records that a consume or a subscription waits for, with the correlation id it provides.
matchByValue check runs without the virtual user’s session, so it can’t use the virtual user’s attributes, such as #{orderId}.
It can’t correlate values serialized with a schema registry either: the simulation fails at startup if you combine them.Two requests, consumes, or subscriptions waiting at the same time on the same topic must have different correlation ids: the second one fails.
Understand how replies are read
Gatling reads the reply topics, and the topics of the consumes and subscriptions, with a consumer shared by all the virtual users of the protocol. This consumer doesn’t join a consumer group: it reads every partition of a reply topic, starting from its end, from before the first request on this topic is sent. It ignores the replies no virtual user waits for. The plugin doesn’t load test consumer groups.
As a result, when you run a test on several load generators, each one reads every reply, including the replies to the requests of the other load generators, which it ignores.
With matchByValue, each load generator runs the check on every record of these topics: at high rates, prefer matchByKey or matchByHeader.
A single thread runs this consumer for each protocol configuration.
On a 10-core laptop against a local broker, with 1 KB records and the default consumer settings, it matches about 105,000 records per second by header, 95,000 with a jsonPath check, and reads about 140,000 records per second that no virtual user waits for.
Past these rates, the replies queue up and the response times grow.
Gatling reads the partitions that a topic has when it starts reading it. The partitions added to a topic during the run aren’t read, so the requests, consumes, and subscriptions whose record lands on them time out. The timeout’s error message mentions it.
Keep the records nobody waits for yet
On the topics of the consumes and subscriptions, a record may be read before a virtual user waits for it, for example when a consume comes after other actions.
Gatling keeps these records, at most 10,000, for at most 60 seconds, the oldest being dropped first, so that a consume or a subscription that starts later still finds its record.
On a busy topic, 10,000 records may cover a short time, and a consume that starts later than that after its record was read times out although the record came.
Gatling then logs a warning naming consumeBuffer, which sets the number of records kept and how long they’re kept:
kafka.bootstrapServers("broker1:9092")
// at most 50,000 records, each kept up to 30 seconds
.consumeBuffer(50000, Duration.ofSeconds(30));kafka.bootstrapServers("broker1:9092")
// at most 50,000 records, each kept up to 30 seconds
.consumeBuffer(50000, Duration.ofSeconds(30))kafka.bootstrapServers("broker1:9092")
// at most 50,000 records, each kept up to 30 seconds
.consumeBuffer(50000, 30.seconds)Raise the number of records as the memory of the load generator allows.
The kept records only matter for a consume, and for a subscription placed after the action that makes its record.
A subscription placed before this action, or attached to a send or a requestReply, waits before its record can come.
Configure timeouts
replyTimeout limits how long a request-reply waits for its reply, and a consume or a subscription for its record, 60 seconds by default.
Each action can override it.
kafka.bootstrapServers("broker1:9092")
.replyTimeout(Duration.ofSeconds(10));kafka.bootstrapServers("broker1:9092")
.replyTimeout(Duration.ofSeconds(10))kafka.bootstrapServers("broker1:9092")
.replyTimeout(10.seconds)The Java and Kotlin replyTimeout also takes a number of seconds.
Understand producer performance
Each protocol configuration has a single Kafka producer, shared by all the virtual users of all the scenarios using it, and closed when the run ends.
Configure batching like the system under test
A Kafka producer waits up to linger.ms milliseconds to batch records together.
Since Kafka 4, linger.ms defaults to 5 milliseconds instead of 0.
With few virtual users, there are few records to batch, so this wait drives the response time of send.
Set the producer properties that drive latency, such as linger.ms, batch.size, acks, and compression.type, to the same values as the producers of the system under test, so that Gatling measures what the system’s own producers would experience:
kafka.bootstrapServers("broker1:9092")
// the same as the producers of the system under test
.producerProperty("linger.ms", "0");kafka.bootstrapServers("broker1:9092")
// the same as the producers of the system under test
.producerProperty("linger.ms", "0")kafka.bootstrapServers("broker1:9092")
// the same as the producers of the system under test
.producerProperty("linger.ms", "0")Understand how records are sent
A Kafka producer may block while it waits for the metadata of a topic, or for room in its buffer. So that virtual users never block, Gatling hands the records to 8 dedicated sender threads per producer, which send them. The records of a topic without an explicit partition are sent from the same thread, in order.
Each of the first 8 topics of a producer gets a sender thread of its own, so a topic waiting for its metadata doesn’t delay the records of another topic. The records with an explicit partition are sent from another thread than their topic’s. Past these, topics share the sender threads.
The virtual users serialize the records themselves, so that serialization spreads over their threads, except the keys and values of a schema registry until their schema is cached.
Each sender thread holds a bounded number of records waiting to be sent.
When it’s full, because the brokers or the producer can’t keep up with the load, the send fails right away with a Kafka producer saturated error, instead of piling up records without bound.