Transforming Transactional Messaging with Kafka and GenStage in Elixir

Ryan Trontz from Dollar Shave Club delivered a talk on the interplay between Apache Kafka, GenStage, and Elixir in the development of transactional messaging systems. He illustrated how the company transitioned from a monolithic architecture to microservices, highlighting the importance of asynchronous messaging in distributed systems. Trontz described the Transactional Messaging Service (TMS), an Elixir microservice built with Kafka and GenStage, which is highly scalable, fault-tolerant, and designed to send thousands of transactional messages to members globally. He compared and contrasted Kafka and GenStage roles, demonstrated the flexibility of TMS beyond a Kafka Consumer, and explained how GenStage enables the management of traffic, message transformation, message routing, and fault-tolerance through republishing to Kafka. Ryan later delved into the system's extensibility and the ease of adding, subtracting, and swapping upstream and downstream service dependencies. A key focus was placed on the demand-driven nature of GenStage, known as back pressure, and its role in preventing bottlenecks by allowing consumers to control the flow of events. Ryan concluded by discussing Dollar Shave Club's future plans to optimize the TMS by reducing node counts, implementing retry logic, leveraging distributed Elixir with libraries like LibCluster and Horde for state management, and potentially switching to a more capable email service provider.