In Persian speech, it is more important to condense than to elaborate when conveying emotions in Whisper

There is a troubling challenge in emotion recognition in Persian speech: a large pre-trained model does not guarantee that its language adaptation will translate well to emotions. In the Whisper study, the model was fine-tuned on Persian ASR, but the main improvement did not come from that.

The authors extracted frame-by-frame representations from the Whisper encoder and compressed them using PCA. This allowed them to eliminate trainable projection layers, reduce the number of parameters, and lower training latency and memory consumption. After compression, the features were combined using an attention mechanism and fed into a lightweight classification head.

On the ShEMO dataset, when tested on speakers independent of the training set, PCA consistently improved emotion recognition. However, fine-tuning Whisper for Persian speech recognition yielded only a modest improvement.

The practical takeaway here is more important than the result on a single dataset: before undertaking extensive language adaptation, it is worth checking whether the excessive dimensionality of the features hinders the task. For a small set of labeled data, a compact representation may prove more useful than yet another stage of fine-tuning a large model.

The limitations of this conclusion are significant. The study was tested on a single ShEMO dataset and under specific experimental conditions; numerical results are not provided in the available description. Therefore, it cannot be considered proven that PCA is always better than fine-tuning or that language adaptation is useless for other languages, models, and corpora.

If you were to build such a system with a small dataset, what would you test first: adapting Whisper to the language or reducing the dimensionality of its features?

Source: cs.CL updates on arXiv.org