Benchmarking Objective Quality Metrics for Audio Generation and Restoration: A Comprehensive Analysis
DOI:
https://doi.org/10.66525/ijictt88Keywords:
Audio Evaluation, Objective Metrics, Deep Learning, Generative Acoustics, Signal RestorationAbstract
The rapid advancement of deep learning technologies has fundamentally transformed the fields of audio generation and restoration, enabling the synthesis of highly realistic speech, music, and environmental sounds, as well as the sophisticated recovery of degraded audio signals. However, the evaluation of these advanced systems remains a significant bottleneck. Subjective evaluation, predominantly through Mean Opinion Score testing, is considered the gold standard for assessing perceptual audio quality. Yet, it is inherently expensive, time-consuming, and susceptible to various human biases, rendering it impractical for the rapid and iterative development cycles typical of modern machine learning research. Consequently, there is a critical need for robust, reliable, and scalable objective quality metrics. This paper presents a comprehensive benchmarking of existing objective quality metrics, spanning traditional digital signal processing approaches to contemporary neural embedding-based distances. By evaluating these metrics across a diverse and extensive dataset comprising both generated and restored audio under multiple degradation conditions, we systematically analyze their correlation with human subjective judgments. Our findings indicate a pronounced divergence in metric efficacy depending on the specific task. Traditional metrics often fail to capture the perceptual fidelity of generative models, heavily penalizing realistic but unaligned synthesized content. Conversely, neural metrics demonstrate superior alignment with human perception in generative tasks but exhibit vulnerabilities to specific spectral artifacts in restoration scenarios. The analysis highlights the necessity for task-specific metric selection and underscores the imperative for developing next-generation hybrid evaluation frameworks that integrate psychoacoustic principles with deep feature representations
Downloads
References
1. Meng, L., Shan, T., Li, K., & Gong, Q. (2021). Long-term tract-specific white matter microstructural changes after acute stress. Brain imaging and behavior, 15(4), 1868-1875.
2. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of ICML.
3. Xu, X., Tu, W., Yang, Y., Li, J., Zhang, Y., & Chen, H. (2026). Contribution-aware Dynamic Multi-modal Balance for Audio-Visual Speech Separation. IEEE Transactions on Multimedia.
4. Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., ... & Lu, Y. (2026, March). Sepprune: Structured pruning for efficient deep speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 38, pp. 31861-31869).
5. Wu, Y., He, Y., Liu, X., Wang, Y., & Dannenberg, R. B. (2023, June). Transplayer: Timbre style transfer with flexible timbre control. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
6. Xu, X. (2024). Improving monaural speech enhancement by mapping to fixed simulation space with knowledge distillation. IEEE Signal Processing Letters, 31, 386-390.
7. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems.
8. Xu, X., Tu, W., & Yang, Y. (2025). Efficient audio–visual information fusion using encoding pace synchronization for Audio–Visual Speech Separation. Information Fusion, 115, 102749.
9. Li, A., Yu, G., Zheng, C., Liu, W., & Li, X. (2023). A general unfolding speech enhancement method motivated by Taylor's theorem. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 3629-3646.
10. Shan, T., Wenner, C. E., Xu, C., Duan, Z., & Maddox, R. K. (2022). Speech-in-noise comprehension is improved when viewing a deep-neural-network-generated talking face. Trends in Hearing, 26, 23312165221136934.
11. Guo, J., Chen, X., Xia, Q., Wang, Z., Ou, J., Qin, L., ... & Tian, W. (2025, July). HASH-RAG: bridging deep hashing with retriever for efficient, fine retrieval and augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 26847-26858).
12. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.
13. Xu, X., Tu, W., & Yang, Y. (2024). Adaptive selection of local and non-local attention mechanisms for speech enhancement. Neural Networks, 174, 106236.
14. Cui, Z., Huang, T., Chiang, C. E., & Du, C. (2025, August). Toward verifiable misinformation detection: A multi-tool LLM agent framework. In Proceedings of the 2025 International Conference on Generative Artificial Intelligence for Business (pp. 179-185).
15. Huang, Z., Wang, J., Chen, L., Xiao, B., Cai, L., Zeng, Y., & Xu, J. (2025, October). MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 8797-8805).
16. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.
17. Shan, T., Cappelloni, M. S., & Maddox, R. K. (2024). Subcortical responses to music and speech are alike while cortical responses diverge. Scientific Reports, 14(1), 789.
18. Peng, Xueqing, et al. "Multifinben: a multilingual, multimodal, and difficulty-aware benchmark for financial LLM evaluation." arXiv e-prints (2025): arXiv-2506.
19. Li, J., Zeng, P., & Luo, P. (2026). CANAO: A Cloud-Aware Native Agentic AI Framework for Adaptive Task Orchestration in Cloud-Native Environments. Frontiers in Artificial Intelligence Research, 3(1), 187-198.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Estelle Mercier, Anna Clifford, Bertrand Dean, Doyun Choi (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
