What is the difference between prompt injection and jailbreaking?
Answered
Both manipulate LLMs, but at different levels: jailbreaking aims at the model’s internal safety constraints; prompt injection aims at unauthorised data or behaviour via crafted inputs.
From the guidance
Primary (how) AI Insights: Prompt Risks
While both jailbreaking and prompt injection can be used to manipulate LLMs, they operate at different levels and have distinct goals: jailbreaking is focused on gaining access to the model’s internal workings; whereas prompt injection is focused on manipulating the model’s output through cleverly designed input prompts.
Section: Jailbreaking
Secondary (normative) AI Insights: Prompt Risks
This is a mechanism which manipulates LLM inputs to return unintended responses by crafting specific prompts that exploit the language model’s response mechanisms. It is an attempt to gain unauthorised access to data or behaviour, either returning information to which the user is not entitled or invoking methods or instructions that the user is not authorised to execute.
Section: Prompt injection
Related questions
- How should I handle prompt injection and other generative AI-specific threats?
- What is data poisoning in an AI system?
- What are the main security risks of using AI in government?
- What is adversarial machine learning and why does it matter?
- Should I use a public AI API, a privately hosted model, or a managed platform?