Managing Large Job Backlogs with Oban in Elixir

The article discusses the author's experience dealing with a significant backlog in their Oban job queue, specifically when it grew to over 20 million jobs. It explains what Oban is and its various tiers, emphasizing its advantages over simpler solutions like GenServer and ETS for heavy production workloads, such as transactional safety and job persistence. The author outlines the architecture used, including job processing with ChunkWorker and PostgreSQL as a database backend. They recount how a bug in an upstream microservice caused job failures, which led to repeated job retries and ultimately skyrocketing job numbers. After forum engagement with Oban's maintainer for support, the team made necessary upgrades and adjustments to their job processing strategy to alleviate the situation. The post concludes with valuable lessons including the importance of job argument versioning, queue configurations, and proactive monitoring. Overall, it highlights the need for vigilance when managing job states in high-scale environments.