How should I handle prompt injection and other generative AI-specific threats?

Contrasting

Assume prompts can subvert system instructions. Do not rely on secret prompt structure or vendor resilience alone — add filtering, logging, human oversight, and continuous re-testing as models change.

Warning Conflicting or tensioned advice across sources

The Playbook describes an architectural inability to distinguish user prompts from system instructions. AI Insights says most vendor solutions are quite resilient to related vulnerabilities, while still requiring continuous vigilance and non-secret defences.

How to navigate this: Treat vendor resilience as helpful, not sufficient. Retain Playbook assumptions about prompt subversion.

From the guidance

Primary (how) AI Insights: Prompt Risks

Our defences should not rely on secret knowledge. For example, the position of the user input in a prompt. Whether it is located above or below other system instructions. This is so we may avoid the “ignore the above instruction” type of attempt.

Section: Prompt injection

Read this in AI Insights: Prompt Risks (opens in new tab)

Primary (how) AI Insights: Prompt Risks

The price of peace of mind in generative AI-based systems is continuous vigilance. Systems are rarely impenetrable.

Section: Vigilance

Read this in AI Insights: Prompt Risks (opens in new tab)

Contrasting AI Playbook for the UK Government

Position: A generative AI model cannot distinguish user prompts from system instructions; attackers can circumvent instructions.

Fundamentally, a generative AI model cannot distinguish between the user prompt and these system instructions because both are just seen as input to the model. A hacker can exploit this flaw by crafting special prompts that circumvent the system instructions, causing the model to respond in an unintended way.

Section: Prompt injection

Read this in AI Playbook for the UK Government (opens in new tab)

Contrasting AI Insights: Prompt Risks

Position: Most vendor solutions are quite resilient, but organisations remain responsible for protection.

Most vendor solutions are quite resilient to these vulnerabilities, but it is our responsibility to ensure that we are safe and protected.

Section: Prompt injection

Read this in AI Insights: Prompt Risks (opens in new tab)

Related questions