What is data poisoning in an AI system?

Answered

Data poisoning deliberately corrupts training data or user feedback so the model behaves as an attacker intends. NCSC lists it alongside prompt injection as a way adversarial ML causes unintended behaviours. Sanitise inputs and feedback used for continuous learning.

From the guidance

Primary (how) Guidelines for secure AI system development

There are many ways to achieve these effects, such as prompt injection attacks in the large language model (LLM) domain, or deliberately corrupting the training data or user feedback (known as ‘data poisoning’).

Section: Why is AI security different?

Read this in Guidelines for secure AI system development (opens in new tab)

Secondary (normative) NCSC: Secure design (AI systems)

you apply appropriate checks and sanitisation of data and inputs; this includes when incorporating user feedback or continuous learning data into your model, recognising that training data defines system behaviour

Section: Design your system for security as well as functionality and performance

Read this in NCSC: Secure design (AI systems) (opens in new tab)

Related questions