Why Do 3D Avatars "Mumble" Their Words, and How Does PD-GS Fix This?

With 3D avatars, the problem is often not with overall lip-sync but with specific sounds. The model may move its lips beautifully but fail to close them when pronouncing “p,” “b,” or “m.” The result is the “leaky mouth” effect—the mouth appears to be slightly ajar even at the moment of clear closure.

The reason is quite practical: audio is a continuous signal, while articulation consists of short, discrete events. If the model is trained to directly reconstruct lip movements based on the acoustic representation, it tends to choose an averaged configuration. While this is convenient for smooth animation, it distorts the meaning of the spoken consonants.

PD-GS adds temporally aligned phonemes to the audio. These are obtained through automatic speech recognition and forced alignment—a procedure that determines when each phonetic element is pronounced. The Linguistic Fusion Module then decides which to trust more: the continuous audio context or the discrete phoneme. In typical sections, smoothness is preserved, while in articulation-critical sections, the model receives a more rigid cue.

This is an important trade-off: the authors do not replace the audio with phonemes, but rather use phonemes as a corrective signal. As a result, PD-GS achieved the best lip geometry metric among the compared methods at HDTF—LMD 2.66—and significantly reduced closure errors in complex phoneme sequences.

But the solution isn’t magic. Quality depends on the specific ASR and forced-alignment pipeline, meaning that recognition or alignment errors can be directly carried over into the animation. Furthermore, the method is trained on monocular video with image reconstruction and lip keypoint annotation; the source provides no information on cost, availability of a ready-to-use implementation, or comparisons under a broader range of conditions. Therefore, the main takeaway here is not that “the avatar finally understands speech,” but a narrower and more useful conclusion: for realistic lip movements, acoustics alone are not enough—explicit linguistic constraints are needed.

If you had to choose between completely smooth, audio-responsive animation and a system that is sometimes subject to recognition errors, which would be more important for your project: visual smoothness or precise lip-syncing? Source: cs.AI updates on arXiv.org