AI Security Basics: Prompt Injection, Data Leakage and Practical Defenses

Applications built on large language models introduce security problems that differ from those in traditional software. A language model reads instructions and data in the same stream of text, and it cannot reliably tell which is which. This guide explains the core ideas: how prompt injection works, how sensitive data can leak, why giving models tools raises the stakes, and which layered defenses security engineers commonly apply. It is written for developers, product teams and managers who need a working mental model rather than a deep research background.

Why Language Models Change the Threat Model

In conventional programs, code and data are kept separate. A database query uses parameters so that user input cannot be mistaken for commands. Language models blur that line. A system prompt, a user question, a retrieved document and a web page may all be concatenated into one block of text, and the model predicts a response based on everything it sees. There is no hard boundary that forces it to treat a sentence inside a document as mere content.

This creates a new category of weakness. Because behavior is shaped by natural language, an attacker does not always need to find a software bug; sometimes they only need to write convincing text. It also means that testing is probabilistic. A defense that works on a hundred attempts may fail on the next, so security engineers generally design for containment rather than assume any single filter is perfect.

Prompt Injection: Direct and Indirect

Direct prompt injection occurs when a user types instructions designed to override the developer's intended behavior, for example telling a customer-support bot to ignore its rules or reveal its hidden instructions. This is often called jailbreaking when the goal is to bypass safety policies.

Indirect prompt injection is generally considered the more serious problem for deployed systems. Here the malicious instructions are hidden in content the model processes on a user's behalf: a web page, an email, a shared document, a code comment or a search result. The user never sees the attack, yet the model may follow the embedded instructions.

An illustrative scenario, described hypothetically: an assistant is asked to summarize incoming email. One message contains text, perhaps invisible to a human reader, instructing the assistant to forward the user's recent messages to an external address. If the assistant has permission to send email and nothing checks its actions, the instruction may be carried out. No software vulnerability was exploited; the system simply obeyed text it should have treated as untrusted data.

The OWASP project listed in Further Reading ranks prompt injection among the leading risks for language-model applications, and no known technique removes the problem entirely. Practical security therefore concentrates on limiting what a successful injection can accomplish.

Data Leakage and Sensitive Information Exposure

Data can leak from language-model systems through several different paths, and it helps to separate them.

• Prompt and context exposure. System prompts, retrieved documents and conversation history may contain confidential details. A user, or an injected instruction, may coax the model into repeating them. Secrets such as API keys should never be placed in prompts.
• Over-broad retrieval. In retrieval-augmented systems, a search index may contain documents that the current user is not permitted to see. If the retrieval step ignores access controls, the model can summarize restricted material for anyone who asks.
• Training and fine-tuning data. Models can sometimes reproduce fragments of their training data, and fine-tuning on sensitive records raises the chance that such content resurfaces. Teams should avoid training on data that users should not be able to extract.
• Third-party handling. Prompts sent to an external service may be logged or retained according to that provider's policies. Employees pasting confidential material into unapproved tools is a common, non-technical source of exposure.
• Output channels. Model output rendered as markdown or HTML may include links or images that send data to an attacker-controlled address when a browser loads them, a technique that can exfiltrate conversation content without any explicit network access by the model.

The common principle is that anything the model can see, a determined user may be able to extract. Data minimization, therefore, is a core defense: supply the model only what the task requires.

Excessive Agency: When Models Can Act

Risk grows sharply when a model is connected to tools: sending messages, running code, querying databases, making purchases or changing records. OWASP describes the problem of granting a system too much functionality, permission or autonomy as excessive agency. The danger is a combination: a model that can be influenced by untrusted text, plus the ability to take consequential actions, plus the user's own credentials.

Security engineers sometimes summarize the worst case as three conditions occurring together: access to private data, exposure to untrusted content, and a way to communicate externally. If all three are present in the same workflow, an attacker may be able to steal data through injected instructions. Removing any one of the three breaks the chain, which is a useful design lever.

Layered Defenses That Work in Practice

Because no single control is reliable, teams combine several. The following measures are widely recommended.

1. Apply least privilege. Give the model and its tools only the permissions needed for the task. A summarization assistant does not need the ability to send email or delete files. Use scoped, short-lived credentials rather than broad ones.
2. Enforce authorization outside the model. Access decisions should be made by ordinary code that checks the user's identity and permissions, not by instructions in a prompt. Retrieval should filter documents by the requesting user's rights before anything reaches the model.
3. Require human approval for consequential actions. Sending external messages, spending money, modifying production data and similar steps should present the proposed action to a person for confirmation, with clear display of what will occur.
4. Separate and label untrusted content. Mark external text as data, use structured formats, and consider architectures in which one component reads untrusted material but has no tools, while a separate component with tools only receives sanitized, constrained outputs.
5. Validate and constrain outputs. Check model output against expected schemas, allow only approved functions and parameters, and treat output as untrusted input to downstream systems to avoid injection into databases, shells or web pages.
6. Limit data exposure. Keep secrets out of prompts, redact personal data where possible, and restrict rendering of external links and images in model responses.
7. Filter, but do not rely on filters. Classifiers that flag likely injection or sensitive content can reduce noise and catch obvious attempts, but they can be bypassed and should be one layer among many.

Testing, Monitoring and Incident Response

Security for these systems is an ongoing process. Before launch, teams commonly run red-team exercises in which testers deliberately attempt injection, extraction and tool misuse, using both hand-written and automated attack prompts. Findings should feed into regression tests that run whenever prompts, models or tools change, because an update can quietly reopen a closed gap.

In production, logging prompts, retrieved sources, tool calls and outcomes supports investigation, although logs themselves may contain sensitive data and need protection and retention limits. Monitoring can look for unusual patterns such as repeated attempts to reveal instructions, unexpected tool usage or large volumes of data leaving the system. An incident response plan should specify how to disable a tool or the whole feature quickly, how to revoke credentials, and who decides when to notify affected parties.

Governance frameworks help organize this work. The NIST AI Risk Management Framework offers a vocabulary for mapping, measuring and managing risks across the life cycle of an AI system, and complements traditional security programs rather than replacing them.

Frequently Asked Questions

Can prompt injection be fully prevented?

Not with currently known techniques. Because models process instructions and data together, the realistic goal is to reduce the likelihood and limit the impact through least privilege, human approval, output validation and isolation of untrusted content.

Is a self-hosted model immune to data leakage?

No. Self-hosting keeps data inside the organization's infrastructure, but leakage through prompt exposure, poor access controls on retrieval, or unsafe output rendering can still occur. Self-hosting changes who is responsible for the controls rather than removing the need for them.

What is the most useful first step for a small team?

Inventory what the model can access and what actions it can take, then remove anything unnecessary. Reducing permissions and requiring confirmation for sensitive actions delivers substantial protection before any specialized tooling is added.

Further Reading

• OWASP Top 10 for Large Language Model Applications — a community-maintained list of the main risks, including prompt injection and excessive agency.
• NIST AI Risk Management Framework — guidance for managing AI risks across the system life cycle.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — background on the retrieval architecture whose access-control pitfalls are discussed above.