Photo

Designing YDB: Constructing a Distributed DBMS for OLTP and OLAP from the Ground Up

Evgenii Ivanov

from Yandex Infrastructure (Serbia)

About speaker

Principal software developer at Yandex Infrastructure.

I've had a passion for programming and computer systems since my teenage years. Now, holding a Master's degree in Computer Science, I have been fortunate throughout my career to amass a breadth of experience.

About speakers company

At Yandex, thousands of team members generate hundreds of products that entail tens of thousands of comments and pool requests every day. Building user-friendly infrastructure for developing and operating products at that scale is a serious challenge. We build the systems, services, and tools Yandex developers rely on. Our solutions are aimed at ensuring that every product Yandex delivers has ready-to-use infrastructure at every stage. We have our own version control system for storing source code; C++, Java, Python, and Go systems for distributed builds and seamless integration capable of processing hundreds of builds a minute; a distributed task management system; rollout systems; and app monitoring systems. We also develop products for supporting development processes, resource planning, and much more.

Abstracts

broad

Distributed systems are great in multiple aspects: they are built to be fault tolerant and reliable, can scale almost infinitely, provide low latency in geo-distributed scenarios and, finally, they are geeky and fun to explore. YDB is a distributed SQL database which has been running production for years. There are installations with thousands of servers storing petabytes of data. To provide these capabilities, any distributed DBMS must achieve consistency and consensus while tolerating unreliable networks, faulty hardware and absence of a global clock.

In this session, we will provide a gentle introduction to problems, challenges and fallacies of distributed computing, explaining why sharded systems like Citus are not always ACID and differ from truly distributed systems. Then, we will dive deep into the design decisions made by YDB to address these difficulties and sketch YDB's architecture layer by layer - from the bare metal disks and distributed storage up to OLTP and OLAP functionalities. At the end, we will briefly compare our approach with that of Calvin, which originally inspired YDB, and Spanner.

The talk was revoked

other talks of this topic

Photo
Putting the asm in Wasm: from bytecode to native

Edoardo Vacchi

Tetrate

specific
Photo
Migrations at Scale

Karen Tovmasyan

Uber

broad
Photo
Achieving scalability with custom locks

Ivan Savu

VeloxDB

specific