Figure II · Partitions, keys, and consumer groups

A Kafka topic, partitioned.

Kafka does not promise ordering across a topic — it promises ordering inside a partition. Everything else about how Kafka scales follows from that one decision.

Why this matters

The bus that carries every modern data pipeline

LinkedIn built Kafka in 2010 because their activity-feed ingest had outgrown every traditional message queue. Today Kafka is the de-facto central nervous system for most of the internet:

Uber — every ride request, GPS ping, and surge price decision flows through Kafka
Netflix — billions of events per day from every device watching
The New York Times — every article, every edit, every day, archived back to 1851
Pinterest — pin events drive the ML ranking pipelines downstream

The pattern: a durable log that decouples producers from consumers, lets many independent applications consume the same stream, and lets you replay the past without coordinating with the writers.

A topic is split into partitions. A producer sends a message; if the message has a key, Kafka picks the partition by hashing the key (hash(key) mod n); if it has no key, Kafka round-robins. The producer cannot pick a partition cleverly — only the key controls placement.

Each consumer group reads the topic independently. Inside a group, each partition is assigned to exactly one consumer at a time. Add more groups and every group sees every message; add more consumers within a group and you spread the work — up to the number of partitions, after which the extras sit idle.

Type a key below, send some messages, and watch where they land. Toggle the keyed switch off to see round-robin. Add a second consumer group and watch fan-out. Add another consumer to group α and watch the partitions get reassigned.

Figure II.1 A topic with four partitions and two consumer groups

Message key

Use key? keyed

Partitions 4

Group α consumers 2

Group β consumers 1

Action

Read this: the producer flings each message into whichever partition the key hashes to. Inside a consumer group, each partition pipes to exactly one consumer; idle consumers (more consumers than partitions) sit empty. Different groups consume the same messages independently.

Last key —

Hash mod n —

Landed in partition —

Total messages 0

§ II.1What just happened

Same key always lands in the same partition

Type the same key twice and watch — the message lands in the same partition both times. That is the foundation of Kafka’s ordering guarantee: within a single partition, messages are strictly ordered. So all messages with the same key are guaranteed to be delivered in order. Across partitions there is no ordering at all.

No key means round-robin

Turn off the keyed toggle and the producer round-robins messages across partitions. This maximises load balancing but throws ordering away — you cannot say which message arrived first if they ended up in different partitions.

One consumer per partition, per group

Move the group α consumers slider. With fewer consumers than partitions, some consumers are assigned multiple partitions; with more, the extras are idle. The maximum useful parallelism in a group is the number of partitions.

Adding a group is fan-out, not load balancing

Group β does not steal messages from group α. Both groups see every message in the topic. This is how Kafka acts as a durable bus: many independent applications consume the same stream.

Adding partitions later breaks per-key order

Increase the partition count after sending some messages. New messages with the same key may now hash to a different partition than before — the per-key ordering guarantee was tied to the partition assignment, and the assignment changed. This is why production Kafka topics rarely add partitions.

§ II.2The rules to remember

partition       : hash(key) mod n        (or round-robin if no key)
order           : guaranteed within a partition only
group           : one consumer per partition, extras idle
fan-out         : every group sees every message
add a partition : breaks per-key ordering for affected keys
durable triple  : acks=all + min.insync.replicas=2 + RF=3

← previous figure: the Cassandra ring · next figure: HDFS, by the byte →