A Comparison of 11 LLMs on a Code Refactoring Task: An Experiment with God Node
Mikhail T. (Sh0ny)
6 июля 20261 min read
In short
A Habra author conducted an experiment: he asked 5 American and 6 Chinese language models to untangle the “god node” from a real LangGraph agent and then evaluate each other’s suggestions. The results were compared using three different methods to determine which model is most reliable for refactoring tasks.
What Is a God Node
When developing AI agents based on LangGraph, an antipattern known as a god node sometimes arises. This is a node in the state graph that takes on too much logic—iterations, checks, routing, LLM calls, and result correction. The experiment examines the plan node, which is about 350 lines long and contains the agent’s entire orchestration.
The problem with this approach is that the graph ceases to be a visual representation of the system: it becomes harder to explain, debug, test, and modify. The goal of refactoring is not simply to break the function into parts, but to move the control logic up to the graph level.
The Essence of the Experiment
The author asked 11 language models—5 American and 6 Chinese—to propose ways to reorganize the god node. The models then evaluated each other’s proposals. To determine which of them should be trusted, three approaches were used:
- Convergence of evaluations — checking how much the models agree with each other when selecting the best proposal.
- Comparison of analyses by thesis — identifying the best analyst based on the depth of analysis.
- Center of opinion and medoid — a statistical method for determining the most “typical” and representative analyst.
Why This Matters
The experiment helps us understand which LLMs are better suited for architectural refactoring and which are better suited for the role of a critic. This is particularly relevant when designing complex AI agents, where a clean and modular graph structure is critical for future maintenance and scalability.