Advanced Broadway Use for Recursive Workflows

In this ElixirConf presentation, Anthony Accomazzo from Sequin explains how to leverage the Elixir Broadway library to handle complex processing streams not ideally suited for traditional message queues. For their specific case at Sequin, they created recursive workflows tailored for API scraping, where work generates more work iteratively. These recursive workflows do not follow the usual queue-based data processing pipelines because they need to process and create jobs simultaneously. Incorporating Broadway into their system offered benefits such as streamlined error handling, graceful shutdowns, and utilizing 'acknowledgments' to manage job successions. To optimize the workflow, they focused on keeping messages light to avoid overhead during process communication and implemented an intelligent scheduler to manage API rate limits. While Broadway provided a strong foundation, Accomazzo also discussed the flexibility of Elixir's OTP, suggesting that if needed, developers could revert to GenServers or other OTP constructs, highlighting the practicality and accessibility of Elixir's concurrency tools.