2 min read
In short
A model ensemble does not automatically become reliable: when labels are flipped in a coordinated manner, standard voting loses a significant amount of quality. Instead, the authors propose checking the consistency of predictions through the geometry of embeddings—without manual domain-specific rules.
A simple vote among detectors seems like a reasonable safeguard: if several models detect the same object, their collective response can be considered more reliable. But this only works as long as the errors are independent. If the models fail in a coordinated manner, the majority simply reinforces the incorrect answer with confidence.
This is precisely the scenario examined by the authors of the preprint Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models. They examine six ViT detectors and attempt to combine their predictions so that the system can detect not only discrepancies but also suspiciously consistent errors.
The key idea is a metacognitive layer that evaluates how closely each model’s prediction resembles its own training experience. To do this, the authors construct Label Vector Pools: sets of vector representations derived from a specific model’s training embeddings. The geometry of these vectors is used to derive rules capable of distinguishing normal detections from probable errors.
An important practical implication is that such rules do not require hard-coding knowledge such as “an object usually occupies this many pixels” or the use of domain-specific segmentation masks. At the same time, the approach remains neurosymbolic—geometric estimates can be combined with logical rules and additional domain-specific knowledge, if available.
Formally, the combination of predictions is defined as a consistency-based abduction problem and is solved during the testing phase. The paper presents two variants: an exact solution via an integer program and a heuristic with polynomial-time complexity. This is not simply an averaging of probabilities, nor is it just another voting classifier: the system attempts to find the most plausible explanation for a set of imperfect observations.
On clean data, the result was nearly identical to that of the strongest majority vote variant: the difference was no more than 0.005 F1. However, under an attack where labels are flipped in a coordinated manner, the gap changes. At a 90% flip rate, the proposed method achieved an average F1 score of 0.42 compared to 0.35 for MV-Plurality—a relative improvement of 22%. At attack levels above 0.4, it achieved the best F1 score on every test dataset.
This is a strong argument against the intuition that “adding a few more models will make the ensemble more robust.” If the underlying failure mechanism is not accounted for, a greater number of votes can only increase confidence in an error. Checking the geometric compatibility of the prediction with the training distribution provides the ensemble with a missing layer of self-monitoring.
However, it is too early to generalize the results to any system. The experiment was conducted on an aerial imagery benchmark with 15 test sets generated by Weather Shift and six ViT detectors. Therefore, the main conclusion at this point is not that majority voting is obsolete, but that a reliable ensemble requires modeling correlated errors—and this can be done without a complete set of domain-specific hand-crafted rules.
Source: cs.AI updates on arXiv.org