Elixir for Large-Scale Data Processing in Smart Cities

Jeff Grunewald presented at the EMPEX Conference about the capabilities of Elixir in processing large amounts of data from smart and connected vehicles. He explained how his team at SmartColumbus employed Elixir for handling data ingestions from thousands of vehicles to support analyses of vehicle-to-vehicle interactions and traffic patterns. The talk covered their strategies for concurrent processing, data batching, and validation across multiple data streams, as well as the open source libraries they developed for these tasks. Grunewald also shed light on the design principles which emphasized scalability, composability, and the ability to process streaming data effectively. He shared insights into leveraging Erlang and Elixir for pattern matching against raw binaries and utilizing powerful tools like Kafka for data throughput. The lessons learned included keeping it simple, using the right tools for the job, parallelizing wherever possible, and the value of performing certain data processes internally to reduce network hops and increase efficiency. Grunewald's talk demonstrated Elixir's utility in building robust systems capable of real-time, high-throughput, and scalable data processing.