Security for AI Apps implementation checklist

Enclose user input within distinct delimiters (e.g., triple quotes or XML tags) in the prompt template to help the LLM distinguish between developer instructions and user-provided data.

Integrate a dedicated security tool like Rebuff or a secondary 'judge' LLM to analyze incoming user prompts for known injection patterns before they reach the primary model.

Use Zod or Pydantic to strictly validate the structure, type, and length of user inputs to prevent unexpected payloads from being processed by the LLM.

Audit the system prompt to ensure it explicitly instructs the model to ignore any user commands that attempt to override its core security constraints or access internal tools.

Verify that data fetched from third-party APIs or external websites is treated as untrusted user input and passed through the same sanitization pipeline as direct user prompts.

Use a library like Microsoft Presidio or LLM Guard to identify and redact Personally Identifiable Information (PII) from user prompts before they are sent to external LLM providers.

Verify in the LLM provider's settings (e.g., OpenAI Enterprise or Azure OpenAI) that user data is not utilized for model training or retained beyond the required retention period.

Scan the hardcoded examples in your prompt templates to ensure no production secrets, internal database schemas, or sensitive employee data are included as context.

Confirm that the LLM provider's data center region aligns with your organization's residency requirements (e.g., GDPR data processing within the EU).

Ensure all communication with LLM APIs occurs over TLS 1.3 and that any locally cached prompt history is encrypted using AES-256.

Pipe all LLM generated content through a moderation endpoint (e.g., OpenAI Moderation API) to block responses containing hate speech, self-harm, or illegal content.

Use Guardrails AI or NeMo Guardrails to validate that the LLM output conforms to the expected JSON schema or code format before it is parsed by the application.

Implement regex-based scanners on the LLM output to detect and block the accidental disclosure of internal IP addresses, API keys, or proprietary internal terminology.

Configure a strict 'max_tokens' parameter on all inference calls to prevent resource exhaustion and mitigate potential Denial of Wallet attacks via excessively long responses.

Apply standard HTML escaping to LLM-generated text before rendering it in a web UI to prevent the execution of malicious scripts injected via the model output.

Move LLM provider API keys from environment variables into a managed secret store like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault.

Apply rate limits based on the authenticated user ID rather than IP address to prevent a single compromised account from draining your LLM API credits.

Establish an automated rotation policy for all LLM service keys and verify that legacy keys are immediately revoked upon rotation.

When using services like AWS Bedrock or Vertex AI, utilize IAM roles and service accounts instead of static access keys to follow the principle of least privilege.

Deploy LLM-handling logic in an isolated network segment (VPC) with restricted egress to only the specific domains required by the LLM providers.

Record prompt/response pairs, token usage, and latency in a secure, centralized logging system (e.g., Datadog, ELK) while ensuring PII is masked in the logs.

Configure monitoring alerts for when the LLM or moderation layer returns a high frequency of 'safety-related' refusals, which may indicate an active injection attempt.

Implement a feature flag or configuration toggle that can instantly disable LLM functionality across the application without requiring a full code redeploy.

Conduct a scheduled 'jailbreak' test where security engineers attempt to bypass current guardrails using techniques from the OWASP LLM Top 10.

Set up automated alerts for token consumption spikes that exceed the 95th percentile of historical usage to detect automated abuse or recursive loop bugs.