Pentest AI/LLM Security

Purpose

Prerequisites

Authorization Requirements

• Written authorization with AI/LLM testing scope explicitly included
• Model access details — API endpoints, model versions, tool/function access
• Data sensitivity classification — what data the LLM can access
• Rate limit awareness — LLM API costs can escalate quickly

Environment Setup

• Garak for automated LLM vulnerability scanning
• Burp Suite for API interception of LLM requests/responses
• Python scripts for custom prompt injection payloads
• Local proxy to capture full request/response chains

Core Workflow

1. Integration Point Discovery: Identify all LLM integration points — chat interfaces, content generation, RAG pipelines, AI search, code completion, summarization.
2. Direct Prompt Injection: Override system prompts, extract system prompt content, inject instructions that change model behavior.
3. Indirect Prompt Injection: Embed malicious instructions in documents/emails/web pages the LLM processes, poisoned RAG context.
4. Data Exfiltration: Extract training data, PII from context windows, other users' conversation history, system config details.
5. Insecure Output Handling: LLM output rendered as HTML (XSS via LLM), used in SQL queries (SQLi via LLM), used in system commands.
6. Excessive Agency: LLM with tool access performing unauthorized actions, privilege escalation through tool chains, resource abuse.
7. Classification: Document findings with OWASP LLM Top 10 (2025) classification and remediation guidance.

OWASP LLM Top 10 (2025) Coverage

Tool Categories

References

• references/tools.md

  - Tool function signatures and parameters

• references/workflows.md

  - Attack pattern definitions and test vectors