Kafka Tiered Storage Explained
Fraud models need 90 days of replay, but most reads are the last hour. Keeping it all on broker SSD is costly. The fix: Hot local, cold remote, same topic.
In this chapter
Hot data on disk, old data in cheap storage
What breaks
Fraud models need 90 days of replay, but most reads are the last hour. Keeping it all on broker SSD is costly.
Three ways to respond
- Delete after an hour, restore backups: A backup isn’t a readable log.
- Tiered storage: Recent data local, older data remote. (the fix)
- Object storage makes brokers stateless: Brokers still own leaders and hot data.
The fix: Tiered storage
Tiered storage moves finished log segments to object storage, so retention is no longer limited by local disk.
Remember: Hot local, cold remote, same topic.
More on Storage problems
- Kafka Simulator: Producers, Partitions and Consumer Groups
- How Uber, Netflix and Zomato Use Kafka: Real Architectures
- Kafka Interview Questions: 26 Real Production Problems
- Kafka vs a Message Queue: Why Events Stay After Reading
- Kafka Topics vs Partitions and Message Keys Explained
- Kafka Producer Batching: Async Send, linger.ms, batch.size
- Event-Driven Architecture with Kafka: Drop Sync Calls
- Kafka Fan-Out: One Event, Many Consumer Groups
Keep learning
More courses
- Jobs, linked to lessons — Browse jobs where every requirement links to the chapter that teaches it. See how ready you are, how long the gap takes to close, and which listings look stale.
- Match a job post to courses — Paste a job post. Each skill is matched to a track here and put in learning order, foundations first.
- React from open source with patterns — Patterns extracted from Material UI, Radix, React Aria, React Hook Form, Redux Toolkit, TanStack, and Next.js — not slides about hooks.
- Database in depth — How a query becomes work. Indexes, isolation, locks, and the plans that explain why a page is slow — not an ORM tour.
- System Design — Core concepts first (SQL vs NoSQL, CAP, indexing, replication, sharding), then designs grouped by what carries them: the database, Redis, Kafka, or many services.
- Microservices — Microservice patterns one diagram at a time: how to split a monolith, how services talk, and how data stays consistent without distributed transactions.
- Frontend system design — How a UI is sliced when it is a product, not a page: ownership of state, cache, routes, and what happens when the network lies.
- TypeScript in depth — The type system as a design tool. Inference, narrowing, generics, and the errors that mean the model is wrong — not a catalog of utility types.
- CSS in depth — See the box, then the page. Visual chapters from the box model through Flexbox surprises, Grid, selectors, and units, then backgrounds, shapes, and visual effects, each with a playground.
- Accessibility — Foundations first (WCAG in plain words, semantic HTML, lists, alt text), then hands-on HTML labs, a full pass of WAI-ARIA 1.3, and a final fix-the-page challenge.
- Docker — What an image is, why layers cache, and how containers find each other on a network.
- Redis — Fifteen Redis patterns, one picture each: cache, session, lock, counter, rate limiter, queue, and when Redis is the wrong tool.