Avoiding Common Pitfalls when Retrying Elixir Tests in CI

The article explains a problem with the Elixir testing command 'mix test || mix test --failed', used in Continuous Integration (CI) when dealing with flaky tests. This approach is common for retrying failed tests, particularly with browser-based testing tools like Hound and Wallaby. However, the author points out a trap: if any `_test.exs` files don't compile during the initial run, the next run with `--failed` flag will report success without actually rerunning the failed tests. The issue arises because `mix compile` does not affect `.exs` files and thus can omit compiling necessary test scripts. To address the issue, the author suggests checking the exit codes of `mix test` carefully: an exit code of `1` indicates a compilation failure, while `2` signifies test failures. Therefore, the safe way to retry is to conditionally run `mix test --failed` only if the exit code is `2`, ensuring that tests are actually retried only when necessary.