In 2020, RCS collaborated with LAS to develop the Brittleness Audit, a systematic approach to diagnosing and mitigating latent flaws in complex Joint Cognitive Systems (JCS). Over years of experience in analyzing and re-engineering JCS, RCS has identified a set of generalized vulnerabilities, termed brittlenesses, which significantly impair decision-making processes during critical, high-pressure scenarios. These brittlenesses represent latent flaws in the JCS that lead to breakdowns in decision-making under real-world conditions, thereby undermining the rigor and reliability of analytic endeavors. Such vulnerabilities impact a wide array of stakeholders, including commanders, staff, analysts, researchers, and students, exposing them to flawed analytic methodologies and biased decision outcomes.

What Are Brittlenesses?

Brittlenesses are pervasive in human-computer teams and are a primary factor in the failure of 85% of current AI deployments. RCS's analyses typically uncover 6–12 such faults within a given system, with at least one brittleness present in every system examined. Addressing and mitigating these brittlenesses unlocks the potential of human-AI teams, with empirical evidence demonstrating up to a 400% improvement in system performance and decision-making outcomes.

The Brittleness Audit Service

To address these challenges, RCS offers the Brittleness Audit, a professional evaluation service conducted by seasoned experts. This service involves a comprehensive analysis of human-machine teams to identify and assess latent vulnerabilities that may precipitate catastrophic failures. The audit provides scientifically grounded solutions to enhance system performance, improve JCS resilience under critical pressures, and ensure robust decision-making capabilities. The Brittleness Audit distills decades of expertise into an accessible and actionable report, empowering organizations to rectify systemic flaws effectively. Additionally, RCS provides training for product development teams, equipping them with the skills to conduct Brittleness Audits internally across their entire portfolio of JCS tools.

Dual-Use Applications

Commercially, RCS offers the Brittleness Audit in two forms, 1) As a turnkey service provided by RCS engineers, producing an assessment and redesign recommendations or, 2) as a product offering for companies to conduct their own internal audits, producing an automated assessment report along with proven design patterns to mitigate the identified brittleneses.

The Pilot

The Brittleness Audit Y3 pilot was successfully executed in support of the TMOD program across three workflows, demonstrating the capability of analysts and developers to accurately detect and identify brittleness within enterprise workflows and tools as an intrinsic capability that does not require specialized expertise. Furthermore, the pilot validated the ability to effectively integrate these findings into the program's agile software development processes, ensuring continuous improvement and resilience.

Brittleness Audit is now available to analyst-product leads, developers, and project managers across the enterprise from their desktop.

Read more about the Brittleness Audit here