Deploying AI agents for warehouse-scale tasks such as bug detection, patch testing, and code review requires overcoming significant technical hurdles. One major bottleneck: the need to build dynamic execution sandboxes for each repository, which is expensive and computationally heavy.
Using a large language model (LLM) instead of executing code is gaining popularity to bypass this burden, but this often leads to unsupported guesses and hallucinations.
Meta researchers present to improve non-execution reasoning "semi-formal reasoning," structured referral technique. This method requires the AI agent to fill out a logical certificate by explicitly stating premises, following specific execution paths, and drawing formal conclusions before responding.
The structured format forces the agent to systematically gather evidence and follow function calls before drawing conclusions. This increases the accuracy of LLMs in coding tasks and significantly reduces errors in locating errors and answering codebase questions.
For developers using LLMs in code review tasks, semi-formal reasoning enables highly reliable, execution-free semantic code analysis while dramatically reducing the infrastructure costs of AI coding systems.
Agent code reasoning
Agent code reasoning is the ability of an AI agent to repeatedly gather context to navigate files, track dependencies, and perform deep semantic analysis on the code base without running the code. In enterprise AI applications, this capability is essential for scaling automated bug detection, comprehensive code review, and patch testing in complex repositories where the relevant context spans multiple files.
The industry currently addresses non-executable code inspection through two main approaches. The first involves unstructured LLM evaluators who attempt to validate code directly or by training specialized LLMs as a reward model to approximate test results. A major drawback is that they rely on unstructured reasoning, which allows models to make confident claims about code behavior without explicit justification. Without structured constraints, it is difficult to ensure that agents think comprehensively rather than guessing based on superficial patterns such as function names.
The second approach involves formal verification, which translates code or reasoning into formal mathematical languages such as Lean, Coq, or Datalog to enable automated proof checking. Although rigorous, formal methods require defining the semantics of the programming language. This is completely impossible for arbitrary enterprise codebases that span multiple frameworks and languages.
Existing approaches are also highly fragmented and task-specific, often requiring completely separate architectures or specialized training for each new problem domain. They lack the flexibility needed for large, multipurpose enterprise applications.
How semiformal reasoning works
To bridge the gap between unstructured guesses and overly rigorous mathematical proofs, Meta researchers propose a structured incentive methodology they call “semi-formal reasoning.” This approach equips LLM agents with task-specific, structured reasoning templates.
These templates act as mandatory logical certificates. To accomplish the task, the agent must clearly state premises, follow execution paths for specific tests, and draw formal conclusions based only on verifiable evidence.
The template forces the agent to gather evidence from the codebase before making a decision. An agent must actually follow function calls and data flows step by step, rather than guessing its behavior based on surface-level naming conventions. Gathering this systematic evidence helps the agent handle outliers, such as confusing function names, and avoid making unsupported claims.
Semi-formal reasoning in action
The researchers evaluated semi-formal reasoning on three software engineering tasks: patch equivalency checking to determine whether two patches produce the same test results without running them, bug localization to determine the exact lines of code causing the error, and code question answering to test nuanced semantic understanding of complex codebases. Used in experiments Close Opus-4.5 and Sonnet-4.5 models acting as autonomous verification agents.
The team compared their structured semi-formal approach to several benchmarks, including standard reasoning, where the agent model is given minimal instruction and allowed to freely explain its thinking in unstructured natural language. compared with traditional text similarity algorithms such as difflib.
Semi-formal reasoning in patch equivalence improved accuracy on difficult, selected samples from 78% to 88% using standard reasoning. When evaluating real-world agent-generated patches with the current test specification, the Opus-4.5 model using semi-formal reasoning achieved a verification accuracy of 93%, outperforming both the unstructured single shot database by 86% and the difflib database by 73%. Other tasks showed similar gains across the board.
The article highlights the value of semi-formal reasoning through real-world examples. In one case, the agent evaluates two patches in the Python Django repository that attempt to fix a bug with the 2-digit year format for years before 1000. One patch uses a custom format() function in the library that overrides the default function used in Python.
Standard reasoning patterns look at these patches, refer to Python’s standard built-in function format() , calculate that both approaches will produce the same string output, and declare the patches equivalent as false.
With semiformal reasoning, the agent follows the execution path and checks method definitions. After the structured template, the agent discovers that the name format() in one of the library’s files is actually overshadowed by a module-level custom function. The agent formally proves that, given the attributes of the input passed to the code, this patch will fail the system and another will succeed.
Based on their experience, the researchers suggest that “LLM agents can perform meaningful semantic code analysis without execution, potentially reducing verification costs in RL training pipelines by avoiding costly sandbox execution.”
Notices and Exchanges
Although semiformal reasoning offers significant improvements in reliability, enterprise developers should consider several practical caveats before adopting it. There is a clear computation and latency trade-off. Semi-formal reasoning requires more API calls and tokens. In evaluating patch equivalence, semiformal reasoning required approximately 2.8 times more execution steps than standard unstructured reasoning.
The technique also does not universally improve performance, especially if the model is already highly skilled at a particular task. When the researchers evaluated the Sonnet-4.5 model on a code question-answering benchmark, standard unstructured reasoning already achieved a high accuracy of nearly 85%. In this scenario, implementing a semi-official template did not provide any additional gains.
In addition, structured reasoning can produce high-confidence false answers. Because the agent is forced to create detailed, formal chains of evidence, he may become overconfident if his investigation is thorough but incomplete. In a Python evaluation, the agent carefully followed five different functions to open a valid edge case, but completely missed that a piece of downstream code already handled this exact scenario safely. Because he built a strong chain of evidence, he drew a false conclusion with extremely high confidence.
The system’s reliance on concrete evidence also breaks down when the limits of the code base are reached. When parsing third-party libraries where the underlying source code is not available, the agent will still resort to guessing behavior based on function names.
In some cases, despite strict operational guidelines, models sometimes fail to fully follow specific implementation paths.
Finally, while semi-formal reasoning dramatically reduces unstructured guesses and hallucinations, it does not eliminate them entirely.
What developers should take away
This technique can be used out of the box without requiring any model preparation or special packaging. This code execution is free, meaning you don’t need to add additional tools to your LLM environment. For higher accuracy in code review tasks, you pay more computation during inference.
The researchers suggest that structured agent reasoning can offer a “flexible alternative to classical static analysis tools”: instead of encoding analysis logic in specific algorithms, we can approach LLM agents with task-specific reasoning templates that are generalized across languages and frameworks."
The researchers have quickly made templates available, allowing you to easily apply them to your applications. While there’s a lot of talk about the death of agile engineering, this technique shows just how much performance you can get from well-constructed instructions.
