Why our LLM Gateway filters before the model

"Let the model answer and filter out what shouldn't go out." That assumption sounds clean, but it's a dead end. Filter only at the output and you see the problem after the model has already burnt token budget. And if the answer contains something sensitive, it is already in the log by then.

Our LLM Gateway checks at three points instead of one.

Before the model. Prompts are checked for personal data, business secrets, credentials and policy violations before the API call goes out. An employee who accidentally pastes a customer name with bank details into a prompt gets the warning first. The model never sees the event.

In the tool call. When the model wants to trigger an action (read a file, close a ticket, send mail), the action is checked against policy. Write operations are human in the loop by default. Read operations are checked against their source: which tenant, which class.

After the model. The response is checked again for the same things as the input, as a safety net rather than the first line. If something slips through, it is blocked and flagged.

In practice that costs 15 to 25 milliseconds of latency per request, depending on check depth. Token budget does not increase. If anything, the opposite: nonsensical requests get caught before they reach the model.

Across six months of operation over three mandates, the gateway caught 148 prompts with personal data before they reached the model. Three times a model would have closed a production ticket without confirmation. Human in the loop prevented it. No business secret has slipped past the output filter.

Most of those 148 cases were not malicious. People paste data into prompts because it's fast. The check before the model is the reminder the model itself doesn't have.

If you're building your own gateway or deploying one, check four things: does it filter before the model, not just after? Does it see tool calls, not just chat text? Is every action audited, including the rejected ones? Can the rules be maintained separately per tenant, team and use case?