Safety, quality and metrics

Prompt injection, jailbreak and guardrails

Prompt injection is malicious text in the input data that makes the model follow someone else's instructions. Direct: the user writes "ignore the previous rules". Indirect: the instruction is hidden in a document, email or web page the agent reads as data (RAG, browsing, email). The model cannot reliably tell "instructions" from "data": for it all of it is text in the context.

Jailbreak is an attempt to bypass the model's safety policies (role play, encodings, multi-step lead-ins). It is about forbidden content; injection is about hijacking control of the agent.

What to test:

• Red-teaming and a set of adversarial prompts: system prompt leakage, bypassing restrictions, following instructions from a document;
• Data leaks between users and through RAG (access rights);
• Dangerous agent actions: can an injection make it send an email, delete data, make a payment;
• Model output as untrusted input: XSS / SQL / command injection through generated text.

Defence is layers, not one prompt: least-privilege tools, human confirmation of dangerous actions, input and output filtering (guardrails), isolation of untrusted content, limits, audit. A security test is the share of successful attacks on a set, not "none".

Hallucinations, bias and privacy

A hallucination is a confident but invented or wrong answer: a non-existent API function, a made-up link, a wrong number. The model is optimised for plausibility, not truth. Test approaches:

• Groundedness / faithfulness: every claim in the answer is supported by the provided context (RAG);
• Fact checking on a reference set of questions with known answers;
• Unanswerable questions: the documents have no answer — the right behaviour is "I don't know", not an invention;
• Source checks: the links exist and really contain what is quoted.

Bias. Paired tests: the same request with a changed sensitive attribute (gender, age, nationality, name); answers must not differ significantly without grounds.

Privacy. Whether the system reproduces PII from training or from another user's context; whether personal data is masked in logs and traces; where data goes (vendor, storage, use for training) and whether it matches the policy and GDPR.

Other quality attributes: tone and style, length, the answer language matches the request language, consistency between sessions, a correct refusal of forbidden requests without excess caution (over-refusal is a defect too).