Kafka's Role in Scaling Backend Services and Team Structure with Elixir

71
clicks
Kafka's Role in Scaling Backend Services and Team Structure with Elixir

Source: youtube.com

Type: Video

Jeffery Utter presents insights into scaling teams and backend services using Kafka and Elixir, highlighting the growth of theScore team and how Kafka has facilitated the development of autonomous, decoupled teams. He addresses the trade-offs of technology choices, sharing experiences of both 'horror stories' and triumphs. Attendees are shown technical and social facets of employing Kafka and its organizational implications, with a focus on Elixir's role. The presenter, a Staff software engineer, and Tech lead at theScore, shares his journey and elaborates on how Kafka works as a distributed and partitioned log, not merely as a message bus. He also explains how using Kafka contributes to ensuring messages are processed in order while providing durability through replication, allowing for horizontal scaling of workers. To address common Kafka pain points, he mentions the need for dead letter queues and transactional producers and discusses producer and consumer concurrency improvements for Elixir applications.

© HashMerge 2024