Creating Custom Producers in Elixir's Broadway Framework

This article focuses on the process of creating custom producers within Elixir's Broadway framework, particularly in the context of building a web crawler. The author, having previously used Broadway for handling concurrent data streams, shares their experience while implementing a custom producer for their side project. They highlight initial challenges faced with demand handling in the producer, referencing the documentation by the Dashbit team. The implementation details include utilizing GenStage behavior and how the producer's function needs to continuously check for available data rather than responding to a single demand request. The article also discusses the importance of maintaining state and using `Process.send_after` for continuous polling of data. It concludes with remarks on the flexibility and efficiency of Broadway as a tool for concurrent processing in Elixir.