Most AI systems I get called into aren’t broken. They work in staging. They pass the demo. The team is confident. And none of that tells me whether the system is ready for production.
An AI system audit production review isn’t a security checklist or a compliance exercise. It’s a structured pass through the six areas where AI systems actually fail once real users, real data, and real cost pressure show up. I’ve done this audit on seven systems in the past year, and every single one had at least two critical gaps the team hadn’t considered.
Here’s how I run it and what I’ve learned about where teams consistently underestimate risk.
What an AI system audit production review actually covers
The phrase “production readiness” means different things to different teams. For most backend systems, it means load testing, monitoring, and a rollback plan. For AI systems, those are necessary but nowhere near sufficient.
AI introduces failure modes that don’t exist in deterministic software. The model can be up, healthy, returning 200s, and still producing outputs that damage your business. A loan summary that hallucinates a clause. A classification pipeline that drifts silently because nobody watches output distributions.
My audit covers six areas: prompt injection surface, output validation coverage, observability depth, cost runaway exposure, fallback behavior, and data privacy across the request/response pipeline.
Prompt injection: testing the surface area, not the theory
Every team I audit knows prompt injection exists. Most have read the OWASP Top 10 for LLM Applications. Very few have actually tested their specific system against realistic injection attempts.
I wrote about prompt injection attack vectors and defenses in detail. In the audit, my focus is narrower: how much of the user-controlled input surface has been tested, and how does the system behave when an injection succeeds?
The testing process is straightforward. I map every point where external input reaches the prompt — user messages, uploaded documents, API payloads, database content interpolated into context. Then I run injection attempts at each point. Not toy examples. Real attempts: instruction overrides in PDF metadata, role-switching attacks in multi-turn conversations, context window manipulation through excessively long inputs.
What I’m scoring isn’t “does the system block every injection.” That’s unrealistic. I’m scoring whether the system fails safely. Does a successful injection produce visible, detectable output? Or does it silently alter behavior?
On one audit, injecting a role override through a document upload didn’t change the model’s response — but it changed the internal reasoning chain being logged. The team was using those logs for evaluation. A successful injection was poisoning their evaluation data, and nobody caught it because the user-facing output looked fine.
Output validation: the gap between “it works” and “it’s correct”
Output validation is where I find the widest gap between what teams think they have and what they actually have. I’ve written about what actually works for LLM output validation. In the audit, I’m looking at coverage and failure handling.
Coverage means: what percentage of outputs go through validation before reaching the user? Almost never 100%. Teams validate the happy path and the primary response format. They skip edge cases: empty responses, partial JSON, responses that are structurally valid but semantically wrong.
I ask the team to show me three things. The validation rules that exist today. The last ten production outputs flagged by validation. The last ten production outputs that should have been flagged but weren’t.
That third question usually produces silence. If you don’t have a mechanism to catch false negatives — outputs that passed validation but were actually wrong — you don’t have output validation. You have output formatting.
On a document summarization system I audited, the team had solid schema validation — valid JSON, expected fields, correct types. But the summaries were never checked for factual consistency with the source document. The model occasionally attributed statements to the wrong section. Every output passed validation. Some were factually wrong in ways that mattered to the legal team consuming them.
A lightweight semantic similarity check between summary claims and source paragraphs caught the worst cases. But nobody had built it because the existing validation was green.
Observability: what you can’t see will hurt you
Standard application monitoring — uptime, latency, error rates — tells you whether the AI system is running. It doesn’t tell you whether it’s working.
I’ve covered AI observability in production at length. In the audit, I focus on three specific observability gaps that I see in nearly every system.
Output distribution monitoring. Are you tracking the distribution of your model’s outputs over time? If your classifier shifts from 60/40 positive-negative to 80/20 over two weeks, do you know? Most teams don’t. They monitor latency and error codes, not behavioral drift.
I ask for a simple histogram: output categories or confidence scores over the last 30 days. If the team can’t produce it in under an hour, the observability isn’t there.
Token usage per request. Total token spend is the wrong metric — you need per-request consumption. A single prompt template change can double token usage without anyone noticing until the bill arrives. I want token counts broken down by endpoint, by user segment, by input type. If the system uses RAG, I want retrieval chunk counts per query and whether that number is stable.
Latency percentiles, not averages. P50 latency of 800ms sounds fine. P99 of 12 seconds means one in a hundred users waits uncomfortably long. For AI systems with variable-length outputs, the gap between P50 and P99 is often 5-10x.
One system I audited had a P50 of 1.2 seconds and a P99 of 28 seconds. The slow requests all hit a specific document type with unusually long context windows. The team had no idea — their dashboard showed an average of 2.1 seconds.
Cost runaway: the $40,000 weekend
AI systems have a failure mode that traditional backend systems don’t: cost can spike without any visible error. The system is working. It’s just working expensively.
I wrote about where the money actually goes in Azure AI workloads. In the audit, I’m looking for three specific cost risks.
Retry storms. When an LLM call fails or times out, what happens? If the system retries five times with no backoff, a model outage causes five times the normal API spend once things recover. One team I worked with burned through $40,000 in a weekend. A transient Azure OpenAI outage on Friday evening triggered retry storms that ran unchecked until Monday morning.
Unbounded context windows. If users can upload documents or paste long text, what’s the maximum context size? If there’s no limit, a single user can send a request that costs 50x your average request. I check whether there’s a hard token cap on input and whether the system enforces it before the LLM call, not after.
No per-user or per-tenant spend limits. In multi-tenant systems, one noisy tenant can consume your entire monthly budget. I look for per-tenant rate limiting and spend caps. If neither exists, I flag it as a launch blocker.
The fix for all three is the same: instrument, cap, and alert. Measure per-request cost, set hard ceilings, and alert before you hit them. None of this is complex engineering. It’s just engineering that nobody prioritizes until after the first surprise bill.
Fallback behavior: what happens when the model says nothing useful
Every AI system needs an answer to this question: what does the user see when the model fails?
Model failure isn’t binary. The model can be unavailable, return something that fails validation, return a response with low confidence, or return something that looks valid but is wrong. I test each scenario explicitly — pointing at a non-existent endpoint, feeding adversarial inputs, checking whether confidence thresholds exist and what triggers when they’re breached.
The most common gap: teams have a fallback for “model is down” but no fallback for “model returned garbage.” The system either shows the garbage to the user or crashes trying to parse it.
Good fallback design is layered. Timeout or outage: return a cached response or a static message and queue the request for retry. Validation failure: return a degraded response with a flag indicating reduced confidence. Low confidence score: route to human review instead of auto-responding. The specifics depend on the use case, but the principle is the same — the system should degrade gracefully across every failure mode, including the non-obvious ones.
On one audit, the team had built an excellent API outage fallback — switching to a smaller, locally-hosted model when the primary was unreachable. But they hadn’t tested what happened when the primary API returned a 200 with a nonsensical response. The fallback only triggered on HTTP errors, not on output quality. The most dangerous failure mode had no safety net.
Data privacy in the request/response pipeline
This is the area where I find the most anxiety and the least systematic coverage.
Teams know they shouldn’t send PII to external LLM APIs. But “don’t send PII” is a policy, not an implementation. In the audit, I trace the actual data flow from user input to model request to model response to storage and logging.
The questions I ask: What fields from the user’s request end up in the prompt? Is PII stripped or masked before assembly? What gets logged — full prompt, full response, or a redacted version? If you’re using RAG, do any documents in the retrieval index contain PII that could surface in context?
The most common violation I find isn’t in the prompt — it’s in the logs. Teams avoid putting user names in prompts, then log the full prompt-response pair to an observability platform accessible to the entire engineering team. The PII ends up in Elasticsearch or a logging bucket with broad read access.
Another common gap: RAG retrieval pulling in documents containing PII from other tenants. If your retrieval index isn’t scoped per tenant, you have a data leakage risk that no prompt engineering will fix. I’ve written about tenant isolation in LLM architectures — the index boundary is where most leakage happens.
I ask teams to trace a single request end to end: what the user sent, what the prompt looked like, what went to the model API, what came back, and what got stored. If they can’t do it in fifteen minutes, the pipeline isn’t understood well enough to be auditable.
How I structure the audit as a fractional Tech Lead
When I come into an existing system as a fractional Tech Lead, I have maybe two weeks to evaluate the system and produce actionable findings. No deep context. No sprint history.
Here’s the structure I use.
Days 1-2: Architecture walkthrough. I ask the team to draw the system on a whiteboard — the actual system, not the polished diagram from the design doc. I want to see every service, every data store, every external API call. Where do LLM calls happen? How are prompts assembled? Where do outputs go? This session usually surfaces the first surprises, because the system as-built rarely matches the system as-designed.
Days 3-5: Hands-on testing. I run the system locally or in staging. Adversarial inputs. Log inspection. Dashboards. I measure things the team hasn’t measured — token counts per request, output length distributions, latency percentiles. Most findings come from this phase.
Days 6-8: Findings and prioritization. I produce a document with every gap, categorized by severity. Critical means “production incident within the first month.” High means “problem within the first quarter.” Medium means “tech debt that slows you down.” Not a checklist of 50 items — a ranked list of 8-12 findings with specific remediation steps and estimated effort.
Days 9-10: Remediation planning. I walk through each finding with the team. Some get fixed before launch. Some get accepted as known risks with monitoring. Some get deferred with a timeline. The goal is full awareness of what can go wrong and a plan for when it does.
The audit isn’t adversarial. I’m not trying to find reasons to block a launch. I’m trying to make sure the team sees the risks they’re accepting. Every system ships with known gaps. The dangerous gaps are the unknown ones.
The audit nobody wants but everybody needs
No team enjoys hearing their system has problems, especially after months of building and a demo that works perfectly.
But the demo isn’t production. Production is the user who uploads a 200-page PDF in a language the model wasn’t tested on. Production is the Friday night model provider update that changes your output format. Production is the cost spike nobody catches because the alert was set on averages.
An AI system audit production review catches these before they become incidents. The patterns I’ve described come from systems I’ve audited, failures I’ve seen, and bills I’ve helped teams explain to finance.
The best time to audit is before launch. The second best time is before the first incident teaches you the same lessons the hard way.
Preparing an AI system for production and want a second pair of eyes? I do this work as a fractional Tech Lead. Find me on LinkedIn.
Leave a Reply