Denis Washington & Olli Salonen

Kafka Streams Microservices

Can you build a microservices platform entirely on Kafka Streams? Discover the power and pitfalls of using Kafka as your single source of truth.

Kafka Streams Microservices
#1about 3 minutes

Core concepts of Kafka and Kafka Streams

Kafka is a distributed event streaming platform using topics, partitions, producers, and consumers for scalable data processing.

#2about 6 minutes

Evolving from classic microservices to event-driven design

The architecture evolved from traditional request-response microservices to an event-driven model using Kafka as the single source of truth to improve decoupling and extensibility.

#3about 3 minutes

Understanding the system topology and failure scenarios

The system uses an API service with materialized views for robust reads and command processing topologies that can recover from failures by replaying input topics.

#4about 4 minutes

Building a searchable product catalog pipeline

A data pipeline cleans, deduplicates, and amends product data from various sources, then streams it to Elasticsearch to create a searchable materialized view.

#5about 2 minutes

Implementing inventory management using a CQRS pattern

A command processing pipeline implements the CQRS pattern by separating write operations from read models, using an event topic as the source of truth for inventory data.

#6about 7 minutes

Solving uniqueness constraints and race conditions

Race conditions caused by eventual consistency are solved by using manually updated state stores and repartitioning command streams to ensure data locality for validation.

#7about 3 minutes

Opportunistic data consumption for new features

New features like automatic warranty extensions can be added by deploying new services that consume existing data streams without modifying the original producers.

#8about 5 minutes

Key challenges and lessons from a pure Kafka approach

A pure Kafka Streams architecture presents challenges in development complexity, stateful operations, careful configuration for transactions, and operational tooling.

#9about 12 minutes

Evolving the architecture with a hybrid database approach

The architecture can be evolved by integrating traditional databases to simplify complex stateful logic, while using connectors to publish all state changes back to Kafka.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

See all videos
The Rise of Reactive Microservices46:24

The Rise of Reactive Microservices

David Leitner

Why and when should we consider Stream Processing frameworks in our solutions30:51

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Tips, Techniques, and Common Pitfalls Debugging Kafka54:29

Tips, Techniques, and Common Pitfalls Debugging Kafka

DeveloperSteve

Let's Get Started With Apache Kafka® for Python Developers38:50

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Advanced Caching Patterns used by 2000 microservices44:37

Advanced Caching Patterns used by 2000 microservices

Natan Silnitsky

What is a Message Queue and when and why would I use it?45:36

What is a Message Queue and when and why would I use it?

Clemens Vasters

Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

Don't Change the Partition Count for Kafka Topics!24:53

Don't Change the Partition Count for Kafka Topics!

Dainius Jocas

From learning to earning

Jobs that call for the skills explored in this talk.

Rust and GoLang

Rust and GoLang

NHe4a GmbH
Karlsruhe, Germany

Remote
€55-65K
Intermediate
Senior
Go
Rust

Kafka DevOps

REWE digital
Municipality of Madrid, Spain

Kafka
DevOps