Enhancing Complex Data Processing with DAGs in Elixir

97
clicks
Enhancing Complex Data Processing with DAGs in Elixir

Source: youtube.com

Type: Video

Zack Kayzer and Stephanie Lane from cars.com discuss the challenges associated with complex modules in Elixir that involve multiple private functions with intertwined data and logic. They propose using Directed Acyclic Graphs (DAGs) to sort out dependencies and maintain clear relationships between data points. They introduce an abstraction called 'Pacer.Workflow' built on top of graphing libraries like Erlang's :digraph and the Hex package LibGraph. Pacer.Workflow is designed to enhance the developer experience by providing compile-time checks, Telemetry events, and focus on documentation and example usage. The talk covers how Pacer.Workflow allows for collapsing multiple graph nodes into single representations, allowing parallel execution of functions where possible. The solution aims to simplify testing, provide granular instrumentation and metrics, and facilitate single responsibility modules. The speakers also delve into the technical details of DAGs, such as topological sorts and graph visualization, and explain how these concepts are applied in the Pacer.Workflow abstraction.

© HashMerge 2024