How should I threat-model an AI system?

Answered

Use a holistic process covering impacts on the system, users, organisations and wider society if an AI component is compromised or behaves unexpectedly. Assess AI-specific threats, document decisions, and note that sensitive data and rising target value increase attacker interest. Secure by Design also expects threat assessment and modelling as part of a risk-driven approach.

From the guidance

Primary (how) NCSC: Secure design (AI systems)

As part of your risk management process, you apply a holistic process to assess the threats to your system, which includes understanding the potential impacts to the system, users, organisations, and wider society if an AI component is compromised or behaves unexpectedly. This process involves assessing the impact of AI-specific threats and documenting your decision making.

Section: Model the threats to your system

Read this in NCSC: Secure design (AI systems) (opens in new tab)

Secondary (normative) Secure by Design Principles

Establish the project’s risk appetite and maintain an assessment of cyber security risks to build protections appropriate to the evolving threat landscape.

Section: 3. Adopt a risk-driven approach

Read this in Secure by Design Principles (opens in new tab)

Related questions