WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Sep 30, 2021

Distributed search under the hood

Alexander Reelsen

How do distributed search engines survive network fallacies? Peek under the hood to see how trading exact precision for probabilistic data structures unlocks massive query performance.

Pause
Mute Enter Fullscreen
#1 about 5 min

Reasons for adopting distributed systems in modern architectures

Scaling capabilities beyond a single machine enables load balancing, better reliability, and parallel workload execution.

#2 about 3 min

Complexity increases when widening system processing boundaries

Operating across networked machines requires coordination, state synchronization, and handling latency over typical single-node designs.

#3 about 3 min

Addressing the fallacies of distributed computing networks

Assuming reliable networks with zero latency and infinite bandwidth leads to systemic failures during implementations.

#4 about 3 min

Achieving consensus and handling cluster participant failure

Distributed algorithms depend on consensus for leader election, membership tracking, and avoiding destructive split-brain configurations.

#5 about 5 min

Core search and resiliency capabilities of Elasticsearch

Built on Apache Lucene, Elasticsearch processes fast full-text search patterns using deep relevancy scoring and distributed data structures.

#6 about 5 min

Managing cluster state updates through master nodes

Dedicated master nodes track topology and handle structural decisions without blocking regular read or write data operations.

#7 about 3 min

Distributing workload data using primary and replica shards

Splitting active indices into multiple logical shards supports write parallelization while copies prevent immediate data loss.

#8 about 2 min

Executing two-phase distributed search strategies across clusters

Distributed queries utilize a scatter-gather sequence where local node results independently process before retrieving the final document matches.

#9 about 2 min

Selecting nodes intelligently using adaptive replica selection

Tracking past query delays and thread pool capacity ensures optimized node targeting and prevents performance bottlenecks.

#10 about 3 min

Skipping non-competitive search hits in scoring algorithms

Halting execution on documents mathematically unable to reach top ranks speeds up queries at the cost of exact hit totals.

#11 about 7 min

Using probabilistic data structures for distributed aggregations

Calculating cardinality and percentiles efficiently requires embracing approximate accuracy through constant-memory algorithms like HyperLogLog++ and T-Digest.

#12 about 5 min

Evaluating algorithmic trade-offs across distributed system implementations

Studying alternative storage models uncovers how varied database systems prioritize consensus, replication routing, and fault repairs differently.

#13 about 6 min

Audience questions on consensus, hashing, and tokenizers

Technical inquiries cover Elasticsearch coordination formats, targeted murmur hashing for data placement, and custom Unicode tokenization rules.

Matching moments

3:55 min

Core architecture and federated execution of SearchOLAP

Andrey Abramov Andrey Abramov · World Congress 2026 Europe

2:19 min

Core OpenSearch cluster architecture and terminology

Olena Kutsenko · World Congress 2022

1:56 min

Keeping systems straightforward to minimize performance bottlenecks at scale

Josip Stuhli Josip Stuhli · Coffee With Developers

3:16 min

Q&A on database vendor lock-in and alternative architectural choices

George Asafev · World Congress 2023

1:32 min

Understanding fast, accurate, and big data store trade-offs

Philipp Krenn · World Congress 2022

1:03 min

Design principles for optimizing distributed application structures

Wei Hu Wei Hu · World Congress 2025

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
4 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
5 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
12 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
17 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
17 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
24 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 12:50–13:20

Stage 3

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Senior Vice President of Research and Development

Wei Hu
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 25, 2026 · 12:55–13:25

Stage 3

Replay-Safe Architecture: Building Event-Driven Systems That Can Recover With Confidence

Ishan Shah

PayPal, Software Engineer | Distributed Systems, AI, and Platform Engineering

Ishan Shah
Open session

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 4

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Garvit Kataria, Sahil Sabharwal

Garvit Kataria
Sahil Sabharwal
Open session

World Congress 2026 North America

September 24, 2026 · 14:50–15:20

Stage 2

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Stage 2

KV Cache Is Not About Speed: It's About Surviving Inference Costs

David vonThenen

AI/ML Leader | Keynote Speaker | OSS Engineer & Developer Advocate | Agentic AI, Deep Learning, Production AI | Python, Go, C++

David vonThenen

Related videos

Make Your Data FABulous thumbnail
28:39
World Congress 2022 • about 4 years ago

Make Your Data FABulous

Philipp Krenn

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
32:46
World Congress 2024 • about 2 years ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
28:41
World Congress 2025 • about 1 year ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Scaling Databases thumbnail
21:09
World Congress 2024 • about 2 years ago

Scaling Databases

Tobias Petry

Add Location-based Searching to Site with ElasticSearch thumbnail
45:13
LIVE • almost 6 years ago

Add Location-based Searching to Site with ElasticSearch

Derek Binkley

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.