How a 4B model beat 671B on a chip verification test

In chip verification the 4B CHORUS model scored 88.0% Pass@1 on the CVDP-ECov test — 13.5 percentage points more than DeepSeek-R1 with 671B. But this is not a story about small models suddenly becoming stronger than large ones in general: the gain was made on the specific task of generating stimuli for a testbench.

The point of the task is to write executable code that helps check hardware logic and uncover more of a circuit's behaviour. Here feedback from running the test is more useful than simply imitating examples: the code either passes the check or it does not.

CHORUS does not try to squeeze everything out of one general-purpose expert. First, staged SFT creates several behaviourally distinct checkpoints. Then reinforcement learning with a dense reward turns them into strong specialists: their overall results are similar, but their strengths differ at the level of individual tasks.

Those differences are then exploited in two ways — the models are merged without additional training, or post-training continues. In the end the specialists are consolidated into a single 4B model. The practical conclusion here matters more than the record itself: given a good executable check, it is sometimes better to assemble several narrow competencies than simply to increase the parameter count.

The limitation of the result is obvious: the figure of 88.0% relates to one benchmark and one domain — generating test stimuli for hardware verification. From the description given it is impossible to tell how far the method transfers to other tasks, what the training cost, or how it compares with other specialised models. So the comparison with DeepSeek-R1 is a strong signal for this task, but not a general ranking of models.

If you had to automate an engineering task, would you first enlarge the model or assemble several specialised experts? Source: cs.AI updates on arXiv.org