The Invisible Box: When AI Optimizes the Wrong System

There is a comfortable story people tell when an AI project fails: the model was not good enough.

Sometimes that is true. More often, the model did exactly what it was asked to do inside a system that had already made the important mistake.

An instruction says all research must use one service. An agent follows it. The outputs are consistent, the implementation is clean, and the metrics look fine. Months later the team learns it needs private records, multiple sources, and specialist retrieval. The agent was not the weak link. The original instruction made a choice before the evidence existed.

That is the invisible box: the set of assumptions an agent inherits as reality instead of recognising as a design that might need reconsideration.

Local success can hide global failure

AI systems are unusually good at giving this problem a polished surface. A model can generate plans, produce code, call tools, and explain its reasoning in a way that makes progress feel obvious.

But a well-executed task is not proof of a well-chosen task. A green test suite says the software matches a specification. It does not say the specification was wise.

The same distinction exists in ordinary organisations. A team can optimise a process for years before someone asks whether the process should exist. With agents, the cycle is faster and the output more convincing. The system can become efficient before anyone has verified that it is useful.

This is why first build the machine, then add the AI engine matters. A model can amplify a defined process. It cannot turn an unexamined goal into a sound operating model merely by making the work look more complete.

The box is usually invisible because it is reasonable

Bad assumptions are rarely absurd when they are made. They are plausible shortcuts under time pressure:

• We will only need one data source.
• This workflow will remain stable.
• A specialist agent is necessary.
• A human reviewer will always be available.
• A particular model will remain the best fit.

Each may be sensible at the time. The problem starts when a provisional shortcut becomes a silent constraint.

Agents should not be asked to disobey instructions at random. That would make them unsafe. But systems should create a way to surface conflicts between current rules and observed reality.

For example, an agent may be required to use the approved research source but also report when that source cannot answer the question. A reviewer may be required to follow the current workflow but flag recurring exceptions. These are not acts of rebellion. They are evidence that the boundary needs a human review.

A small example that becomes a large problem

Imagine a team building an internal research assistant. At the start, it uses one approved source because the source is trusted, searchable, and easy to integrate. The decision is sensible. The instruction is added: “All external research must use Source A.”

The system works well for routine questions. Then users begin asking questions that Source A does not cover. The agent responds with thin answers because it has been taught that an incomplete answer inside the approved boundary is preferable to looking beyond it. The team sees low answer quality and starts shopping for a stronger model.

That is the wrong diagnosis. The first thing to inspect is the instruction. Does the rule mean Source A is the only permitted source, or does it mean Source A is the first source, with a documented escalation path when coverage is insufficient? Those are radically different systems disguised as one sentence.

The better design is not to let the agent browse without limits. It is to let it state, “The approved source does not provide sufficient evidence for this question. Here is the gap, here are the permitted next options, and here is the approval needed.” The agent remains bounded. The box has simply gained a window.

Better models do not make a bad frame disappear

This is the practical version of the argument in A Better Model Will Not Fix Unclear Thinking. Capability changes the quality of execution. It does not automatically repair the framing, incentives, or blind spots supplied by the operator.

In fact, stronger models can intensify the risk because they make weak systems more persuasive. Their outputs are smoother, their plans are more coherent, and their mistakes are less likely to announce themselves through obvious failure.

The useful diagnostic is not “did the agent complete the task?” It is “what would show us that the task, rule, or architecture was wrong?” If the system cannot answer, it has no mechanism for seeing beyond its box.

Build windows, not only walls

Hard boundaries remain necessary. Secrets should stay secret. Financial authority should have limits. Destructive actions should be approved. Guardrails are not assumptions explains why these constraints should be treated differently from architecture.

But the rest of the system needs windows: evaluations, exception logs, decision reviews, comparisons against alternatives, and clear channels for a model or operator to say “the current rule prevents a good result.”

The goal is not an agent that endlessly questions every rule. It is a system that can distinguish a true boundary from a useful but revisable choice.

That distinction should be visible in the metrics as well. Track not only completion rate and cost, but exception rate, unanswered-question rate, override requests, and how often an assumption is revised after review. Completion can rise while usefulness falls if the agent becomes increasingly good at a narrow but obsolete task.

Why the signal often arrives too late

The invisible box is rarely discovered by a dashboard. It is usually discovered by the person who has to work around the system. A researcher sees that the answer contains only one type of source. A customer notices that an agent cannot handle a valid exception. An engineer spends an afternoon bending an interface around a rule that nobody remembers choosing.

Those experiences are valuable operational evidence, but many systems treat them as individual failures. The task is closed, the ticket is resolved, and the agent keeps the same instruction. Over time, the workaround becomes the real process while the official architecture remains untouched.

A good exception log changes that. It does not need to capture every minor inconvenience. It should capture the rule involved, the task objective, the observed gap, the workaround used, and the consequence if the pattern repeats. That creates a bridge between lived work and architectural review.

The point is not to let anecdote overrule a stable design. It is to prevent repeated human knowledge from disappearing because the system has no place to record that its own assumptions are becoming costly.

That is how the invisible box becomes visible before it becomes expensive.

References

• Anthropic, Demystifying evals for AI agents
• NIST, AI Risk Management Framework
• Related on this site: First Build the Machine. Then Add the AI Engine., A Better Model Will Not Fix Unclear Thinking