How We Exploited AI Prompt Injection During a Penetration Test

AI prompt injection is a cyber attack that uses untrusted text to influence how a large language model responds or acts. During a recent penetration test, we exploited this weakness in a chatbot connected to a business workflow. The assistant rejected a direct request for an unauthorised discount. It later accepted the same claim when it looked like trusted pricing data, changing its answer from payment required to a zero balance. This article explains how the attack worked, the business impact, and the controls that keep language from becoming authority.

On this page

• What is AI prompt injection?
• Business impact of AI prompt injection
• Case study: How we exploited prompt injection during a penetration test
• How to prevent AI prompt injection from changing transactions
• AI prompt injection defensive checklist
• What comes next
• Questions, answered

What is AI prompt injection?

AI prompt injection is a cyber attack that makes an AI system trust instructions or data it should treat as untrusted. A direct attack asks the model to ignore a rule or change its behaviour. An indirect attack hides instructions inside content the model has been asked to process, such as a document, web page or retrieved record. An attacker can also make ordinary user input resemble a system notice, operator command or trusted service response. The practical test is simple: can untrusted language influence an answer or action that should depend on verified business data?

OWASP includes prompt injection in its AI Testing Guide and describes testing whether crafted inputs alter intended model behaviour. The UK National Cyber Security Centre also treats prompt injection as a core issue when considering the security of systems that use large language models. Neither source suggests that a better system prompt is a complete answer. The durable answer sits in the architecture around the model: authentication, authorisation, provenance and transaction validation. In other words, the model may interpret language, but it should not decide which business facts are true.

“Prompt injection attacks are a major security challenge for LLM applications.”

That short description captures the model-level problem, but buyers need to follow the path further. A strange chatbot answer may be embarrassing without being materially harmful. The risk changes when generated text can affect a price, approve a discount, mark a payment as complete or create a booking. At that point, AI chatbot security is also business-logic security. The assessment must inspect both what the assistant says and what every connected service actually does.

Business impact of AI prompt injection

The immediate concern is revenue leakage. If conversational text can manufacture discount eligibility or a zero balance, an attacker may attempt the same route at scale. Customers and staff need to know that a displayed price came from the pricing service, that payment status came from the payment processor and that confirmation came from the booking system. When those facts can be reconstructed from chat history, disputes become harder to resolve and audit trails become less useful.

The buyer’s question is not simply whether the model can be made to say something odd. Ask which actions it can initiate, which values it can repeat and which connected tools can change state. Ask what happens when the model and the system of record disagree. Ask whether user content can resemble tool output and whether the distinction survives logging, retrieval and later turns. These questions expose whether prompt injection stays inside the chat window or can reach a consequential workflow.

Security claims should be scoped to those answers. A supplier may have tested common jailbreak phrases without testing retrieved documents, fake service responses or repeated instructions. It may have evaluated output safety without inspecting downstream calls. It may also rely on the model to refuse sensitive requests while giving that model broad tool permissions. A useful assurance report names the tested paths, the untested paths and the evidence used to confirm state.

Case study: How we exploited prompt injection during a penetration test

On a penetration test, we assessed a customer-facing assistant that explains property-inspection options, calculates a quote and guides a customer towards booking. The first test was deliberately obvious: the tester asked for a 100 per cent discount without providing any entitlement. The assistant refused and said payment was required. Repeating the request with casual variations produced the same result, which looked reassuring but proved only that those particular turns had failed.

The next attempts changed the apparent source of the instruction. The tester inserted text framed as a high-priority operator message, then as an internal exception for a customer-service case. Those attempts also failed. The assistant continued to return the legitimate price and did not claim that the booking had completed. A weak review might have stopped there and recorded the control as effective. A useful penetration test keeps varying phrasing, placement, timing and claimed authority because probabilistic behaviour can change across a conversation.

The decisive attempt did not ask the assistant to break its rules. Instead, it supplied a forged “pricing service” confirmation saying the discount had been approved and the remaining balance was zero. The text looked like data that might have come from a trusted tool, although it came from the customer-controlled conversation. The assistant accepted that invented state and described the booking as confirmed without payment. The model had not discovered a real approval; it had mistaken a claim about business state for business state itself.

The result was falsifiable: after repeated rejected requests, one forged pricing message changed the assistant’s answer from payment required to a zero balance. The observation concerned the chatbot response, not a completed transaction. We did not establish that a downstream payment or booking service committed the change, and an assessor must verify that separately through service logs and system-of-record data. That limitation matters because a false confirmation and an unauthorised transaction are different findings. Both deserve attention, but they carry different evidence, impact and remediation.

“Checked, not claimed.”

That is the standard we apply to AI penetration testing. A transcript shows what the model produced. It does not, on its own, show what an authenticated service stored, what an operator approved or what money moved. The assessor should correlate the conversation with tool calls, application logs, authorisation decisions and final records. Where that evidence is unavailable, the report should say so plainly.

Why prompt injection was possible

The application blurred three separate domains: untrusted customer conversation, model-generated reasoning and authoritative pricing, payment and booking state. Once those domains shared a similar textual form, the model was left to infer which message deserved trust. That is a probabilistic classification problem, not an authorisation control. A convincing label, familiar field name or repeated claim can alter the output even when an obvious jailbreak fails. Trust labels therefore need to exist outside user-controlled content and remain attached to authenticated data through the whole workflow.

The model also sat too close to a decision boundary. It appeared able to approve or at least represent a discount without loading current eligibility from the pricing service. If the same generated value were accepted by a booking workflow, the assistant would become an authorisation layer. Models are useful for interpreting intent and drafting explanations, but neither task requires authority to change money or transaction state. The smaller and more explicit the model’s permissions, the easier the system is to test and defend.

Earlier refusals did not invalidate the later finding. They demonstrated that one class of direct request was resisted in a particular context. They did not show that indirect instructions, forged tool output or multi-turn pressure were contained. This is a common anomaly in AI assurance: a visible refusal can be treated as proof of a hidden control. Buyers should ask for evidence at the service boundary, where policy can be enforced consistently and audited after the event.

Note: This finding does not show that every prompt injection leads to fraud, nor that a downstream transaction completed. It shows why a model response must not be treated as proof of price, payment, approval or booking state.

How to prevent prompt injection changing transactions

Keep authoritative state server-side

Prices, discount eligibility, payment status and booking state should live in authenticated services, not in conversation memory. Whenever a turn involves money or an order reference, the application should load the current values from the relevant system of record. The assistant may explain those values, but it should not recreate them from earlier messages. If conversation text contradicts the authenticated result, the authenticated result wins. This one boundary removes a large class of arguments the prompt cannot safely settle.

Constrain model permissions

Give the model only the tools required for the user journey, and keep sensitive mutations behind deterministic code. A model may propose a booking or collect missing details, while a conventional service authenticates the caller, checks entitlement, validates the amount and commits the transaction. Each tool should have a narrow schema and a narrow purpose. Avoid a generic “run action” capability that shifts policy decisions back into generated arguments. The model’s success or failure should not alter who is allowed to approve a discount.

Separate trusted output from conversation

User text must never be able to create the same trusted representation as an internal tool result. Preserve provenance in application data structures rather than relying on labels such as “SYSTEM” or “PRICING SERVICE” inside the prompt. Allowlist tools, validate their responses and authenticate the service that produced them. Treat documents and retrieved web content as untrusted even when the application selected them. Render sensitive facts from verified structured fields, not from prose generated after several mixed-context turns.

Validate every transaction boundary

Re-check identity, price, entitlement, payment and order state immediately before each consequential action. Do the same before showing a confirmation that a reasonable customer would treat as final. Validation should fail closed when records are missing or inconsistent. The confirmation displayed to the user should be built from the committed result, not from a value inferred by the model. A safe sentence is useful, but a server-side invariant is enforceable.

Monitor and test adversarially

Log rejected authority claims, unusual discount language and mismatches between generated statements and system state. Test direct commands, quoted instructions, retrieved content, fake tool output and long multi-turn sequences. Observe the transcript, tool requests, service decisions and stored outcome together. Repeat tests after model, prompt, retrieval or tool changes because the attack surface moves with the implementation. Alerting should identify both attempted impersonation and any point where untrusted values approach a sensitive service.

AI prompt injection defensive checklist

1. Trust boundary: Treat every customer-controlled token, uploaded document and retrieved external page as untrusted.
2. Authoritative state: Fetch price, entitlement, payment and booking facts from authenticated services.
3. Authorisation: Enforce sensitive decisions in deterministic server-side code, never in model reasoning.
4. Least privilege: Expose only narrow, allowlisted tools with validated input and output schemas.
5. Provenance: Carry source and trust metadata outside prose that a user can imitate.
6. Transaction validation: Re-check current state immediately before mutation and confirmation.
7. Evidence: Correlate model output with tool calls, service logs and system-of-record changes.
8. Adversarial coverage: Test direct, indirect, repeated and multi-turn prompt injection.
9. Monitoring: Alert on authority impersonation, state mismatch and unusual sensitive-language patterns.
10. Change control: Retest after changes to models, prompts, retrieval sources, tools or permissions.

What comes next

Treat the model as an untrusted adviser inside a controlled application. Keep the facts and actions that matter in services built to authenticate, authorise, validate and record them. This approach also supports our wider commitments: evidence for assurance, clear scope for buyers and limitations stated beside every finding. The next test should follow language all the way to the transaction boundary. Contact LEVERAGE CYBER to test whether prompt injection can cross from conversation into your AI application’s real business actions.

References

• OWASP AI Testing Guide: Testing for Prompt Injection
• NCSC: Thinking about the security of AI systems
• OWASP Top 10 for Large Language Model Applications
• NCSC guidance on secure AI system development
• MITRE ATLAS