📜Kafka Offsets & Commit Failures: The Hidden Mechanics Behind Reliable Messaging
If you’ve worked with Apache Kafka for more than five minutes, you’ve probably heard terms like offset, consumer group, and commit.
They sound simple — but under the hood, they define how reliable your system actually is.
Let’s break it down in a way that’s both practical and production-ready.
📍 What is a Kafka Offset?
The Kafka introduction page fabulously illustrates what an offset is.
An offset is the unique position of a message within a Kafka partition.
Each partition is an ordered, immutable log:
- Messages are appended
- Each gets a sequential number (offset)
Example:
Partition 0:
[0] OrderCreated
[1] PaymentProcessed
[2] OrderShipped
[3] OrderDelivered
👉 Offset = your position in this log
In turn, this Medium article excellently explains how topics relate to partitions in Kafka:
Partitions are parts of a topic and each of them has offsets that indicate pr
🧠 Why Offsets Matter
Kafka does not track what each consumer has processed internally.
Instead:
The consumer is responsible for tracking its own progress.
That progress = offset
📌 What is an Offset Commit?
An offset commit is when a consumer tells Kafka:
“I have successfully processed messages up to offset X.”
Kafka stores this in a special internal topic:
__consumer_offsets
This enables:
- restarting consumers
- scaling consumer groups
- fault tolerance
⚙️ Auto Commit vs Manual Commit
🟢 Auto Commit
- Kafka periodically commits offsets
- Easy to use
- ❌ Risk: commits may happen before processing completes
🔵 Manual Commit
- You explicitly commit offsets
- More control
- ✅ Safer for production systems
💥 What Are Offset Commit Failures?
This article illustrates the situation when a message is failed to be committed:
An offset commit failure occurs when Kafka fails to persist the consumer’s progress.
In simple terms:
👉 Kafka forgets where you left off.
🐉 Common Causes of Commit Failures
🧨 1. Consumer crashes
- message processed
- commit never happens
🧨 2. Network issues
- commit request never reaches broker
🧨 3. Broker unavailability
- leader changes
- temporary outages
🧨 4. Timeouts / rebalances
- consumer group rebalances mid-processing
⚠️ The Real Consequence: Duplicate Processing
This is the key idea:
If a commit fails, Kafka assumes the message was not processed
So it will:
- deliver the message again
🔁 This is Why Kafka Is “At-Least-Once”
Offset commit failures are not bugs — they are by design.
Kafka guarantees:
- ✔️ no message loss (in most configurations)
- ❗ but possible duplicates
🧩 Real-World Example
- Consumer processes:
ChargeCustomer - Payment succeeds ✅
- Offset commit fails ❌
- Consumer restarts
- Same message reprocessed
💥 Customer may get charged twice
🛡️ How to Handle Commit Failures Safely
🥇 1. Idempotent Consumers
Use a unique event ID:
eventId = UUID
Store processed IDs:
- DB table with unique constraint
- Redis cache
👉 If seen before → skip
🥈 2. Commit After Processing
Correct pattern:
process message
↓
commit offset
NOT:
commit
↓
process
🥉 3. Use Transactions (Advanced)
Kafka supports:
- transactional producers
- exactly-once semantics (EOS)
But:
- complex
- not always needed
🧱 4. Handle Rebalances Properly
During consumer group rebalance:
- in-flight processing may be interrupted
- ensure safe shutdown + commit
🧠 Mental Model
Offset = bookmark
Commit = saving your place
Failure = losing your bookmark and rereading pages
🎯 Key Takeaways
- Offsets track position in a partition
- Consumers are responsible for committing progress
- Commit failures lead to duplicate processing
- This is why Kafka provides at-least-once delivery
- Idempotency is not optional in real systems
🧘 Final Thought
Kafka doesn’t promise perfection.
It promises durability — and leaves correctness to you.
Design your consumers like they will see the same message twice.
Because eventually… they will.
