Benchmarking Objective Quality Metrics for Audio Generation and Restoration: A Comprehensive Analysis

Authors

  • Estelle Mercier Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia Author
  • Anna Clifford Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia Author
  • Bertrand Dean Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia Author
  • Doyun Choi Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia Author

DOI:

https://doi.org/10.66525/ijictt88

Keywords:

Audio Evaluation, Objective Metrics, Deep Learning, Generative Acoustics, Signal Restoration

Abstract

The rapid advancement of deep learning technologies has fundamentally transformed the fields of audio generation and restoration, enabling the synthesis of highly realistic speech, music, and environmental sounds, as well as the sophisticated recovery of degraded audio signals. However, the evaluation of these advanced systems remains a significant bottleneck. Subjective evaluation, predominantly through Mean Opinion Score testing, is considered the gold standard for assessing perceptual audio quality. Yet, it is inherently expensive, time-consuming, and susceptible to various human biases, rendering it impractical for the rapid and iterative development cycles typical of modern machine learning research. Consequently, there is a critical need for robust, reliable, and scalable objective quality metrics. This paper presents a comprehensive benchmarking of existing objective quality metrics, spanning traditional digital signal processing approaches to contemporary neural embedding-based distances. By evaluating these metrics across a diverse and extensive dataset comprising both generated and restored audio under multiple degradation conditions, we systematically analyze their correlation with human subjective judgments. Our findings indicate a pronounced divergence in metric efficacy depending on the specific task. Traditional metrics often fail to capture the perceptual fidelity of generative models, heavily penalizing realistic but unaligned synthesized content. Conversely, neural metrics demonstrate superior alignment with human perception in generative tasks but exhibit vulnerabilities to specific spectral artifacts in restoration scenarios. The analysis highlights the necessity for task-specific metric selection and underscores the imperative for developing next-generation hybrid evaluation frameworks that integrate psychoacoustic principles with deep feature representations

 

Downloads

Download data is not yet available.

References

1. Meng, L., Shan, T., Li, K., & Gong, Q. (2021). Long-term tract-specific white matter microstructural changes after acute stress. Brain imaging and behavior, 15(4), 1868-1875.

2. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of ICML.

3. Xu, X., Tu, W., Yang, Y., Li, J., Zhang, Y., & Chen, H. (2026). Contribution-aware Dynamic Multi-modal Balance for Audio-Visual Speech Separation. IEEE Transactions on Multimedia.

4. Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., ... & Lu, Y. (2026, March). Sepprune: Structured pruning for efficient deep speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 38, pp. 31861-31869).

5. Wu, Y., He, Y., Liu, X., Wang, Y., & Dannenberg, R. B. (2023, June). Transplayer: Timbre style transfer with flexible timbre control. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

6. Xu, X. (2024). Improving monaural speech enhancement by mapping to fixed simulation space with knowledge distillation. IEEE Signal Processing Letters, 31, 386-390.

7. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems.

8. Xu, X., Tu, W., & Yang, Y. (2025). Efficient audio–visual information fusion using encoding pace synchronization for Audio–Visual Speech Separation. Information Fusion, 115, 102749.

9. Li, A., Yu, G., Zheng, C., Liu, W., & Li, X. (2023). A general unfolding speech enhancement method motivated by Taylor's theorem. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 3629-3646.

10. Shan, T., Wenner, C. E., Xu, C., Duan, Z., & Maddox, R. K. (2022). Speech-in-noise comprehension is improved when viewing a deep-neural-network-generated talking face. Trends in Hearing, 26, 23312165221136934.

11. Guo, J., Chen, X., Xia, Q., Wang, Z., Ou, J., Qin, L., ... & Tian, W. (2025, July). HASH-RAG: bridging deep hashing with retriever for efficient, fine retrieval and augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 26847-26858).

12. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.

13. Xu, X., Tu, W., & Yang, Y. (2024). Adaptive selection of local and non-local attention mechanisms for speech enhancement. Neural Networks, 174, 106236.

14. Cui, Z., Huang, T., Chiang, C. E., & Du, C. (2025, August). Toward verifiable misinformation detection: A multi-tool LLM agent framework. In Proceedings of the 2025 International Conference on Generative Artificial Intelligence for Business (pp. 179-185).

15. Huang, Z., Wang, J., Chen, L., Xiao, B., Cai, L., Zeng, Y., & Xu, J. (2025, October). MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 8797-8805).

16. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.

17. Shan, T., Cappelloni, M. S., & Maddox, R. K. (2024). Subcortical responses to music and speech are alike while cortical responses diverge. Scientific Reports, 14(1), 789.

18. Peng, Xueqing, et al. "Multifinben: a multilingual, multimodal, and difficulty-aware benchmark for financial LLM evaluation." arXiv e-prints (2025): arXiv-2506.

19. Li, J., Zeng, P., & Luo, P. (2026). CANAO: A Cloud-Aware Native Agentic AI Framework for Adaptive Task Orchestration in Cloud-Native Environments. Frontiers in Artificial Intelligence Research, 3(1), 187-198.

Downloads

Published

2026-05-05