For example, Mac CMake builds were reported as "Failed" with 6 build errors (e.g. https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-arm-sonoma-clang-cmake-nightly-everything-release/85/). These errors were actually warnings such as SyntaxWarning: invalid escape sequence... that CMake interpreted as failures.

The CI fix was to update CTEST_CUSTOM_ERROR_EXCEPTION in #21102.

A few things to note:

The equivalent Bazel jobs also show this warning in the output (e.g. https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-arm-sonoma-clang-bazel-nightly-everything-release/85/consoleFull) but are shown as passing in CI

Both CMake and Bazel jobs use CTestCustom.cmake.in. CMake from CMakeLists.txt and Bazel from step-build.cmake.

This issue was originally investigated in #2817 and as a result of that issue, the CAPTURE_CMAKE_ERROR option was added to CMake 3.7. See RobotLocomotion/drake-ci#63

Later, in #9268, we attempted to use launchers instead of scraping logs. That change was reverted in #9320.