Should we red-team automated decision-making systems before go-live?

Answered

Yes. The ADM framework recommends red-team testing on the presumption that algorithmic systems can inflict some degree of harm. Pair with rigorous staged testing, impact/risk assessments (DPIA/EIA where appropriate), and NCSC-style security evaluation before release.

From the guidance

Primary (how) Ethics, Transparency and Accountability Framework for Automated Decision-Making

Do ‘red team testing’, with the presumption that all algorithms systems are capable of inflicting some degree of harm.

Section: 1. Test to avoid any unintended outcomes or consequences

Read this in Ethics, Transparency and Accountability Framework for Automated Decision-Making (opens in new tab)

Secondary (normative) NCSC: Secure deployment (AI systems)

You release models, applications or systems only after subjecting them to appropriate and effective security evaluation such as benchmarking and red teaming

Section: Release AI responsibly

Read this in NCSC: Secure deployment (AI systems) (opens in new tab)

Related questions

• Should we red-team AI before release? Delivery, assurance and operations
• Do automated decision-making systems need legal sign-off?
• Should we publish algorithms used for automated decision-making?
• Should automated decision-making be the default solution for complex policy problems?
• When should I use the Ethics, Transparency and Accountability Framework for Automated Decision-Making?