Developing a Custom Kafka Client for Elixir

In this talk, Saša Jurić outlines the process of developing a bespoke Kafka client due to the inadequacies of existing libraries within the BEAM ecosystem. The client was necessary for a team with numerous Elixir-based microservices communicating heavily through Kafka. Jurić explains that while established libraries provide a solid foundation, they fall short for advanced requirements. This led to the idea of creating a thin Java layer that interacts with Kafka while keeping most of the functionality in Elixir. The talk delves into how to connect Elixir and Java using ports for better integration, including considerations for message processing, concurrency, and handling issues like backpressure. Jurić shares insights into the technical complexities and trade-offs made during the development process, emphasizing the importance of flexibility and critical thinking when facing unique project requirements.