Listener-Variance Modeling for Calibrating Synthetic Singing Quality Scores
DOI:
https://doi.org/10.66525/ijictt131Keywords:
Synthetic Singing, Listener Variance, Quality Calibration, Perceptual AssessmentAbstract
The evaluation of synthetic singing voices heavily relies on subjective human assessments, typically aggregated through the Mean Opinion Score paradigm. However, the perceptual evaluation of singing is notoriously complex, involving multi-dimensional cognitive judgments concerning pitch accuracy, timbral naturalness, musicality, and lyrical intelligibility. This complexity exacerbates the inherent variability among human evaluators, leading to listener-variance phenomena that obscure the true acoustic performance of underlying synthesis systems. Evaluators exhibit diverse baseline strictness, scale utilization, and internal consistency, all of which introduce substantial noise into the evaluation pipeline. To address these systemic inadequacies, this paper proposes a comprehensive listener-variance modeling framework designed to calibrate subjective quality scores in synthetic singing assessment. By conceptualizing the evaluation process through a latent variable paradigm, the proposed methodology disentangles the true systemic quality of the synthesized audio from the subjective biases and unreliability of individual listeners. The study conducts extensive perceptual experiments involving multiple generative architectures and a demographically diverse listener pool. Through sophisticated statistical calibration techniques, we demonstrate that modeling annotator behavior significantly reduces confidence intervals, alters systemic rankings to better reflect true acoustic fidelity, and enhances the statistical power of subjective evaluations. The findings underscore the critical necessity of transitioning from naive arithmetic averaging toward probabilistically calibrated scoring mechanisms in the domain of musical audio synthesis.
Downloads
References
1. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.
2. Zhang, B., Wang, S., Jiang, Y., Sui, D., Tu, Z., & Chu, D. (2025, July). Ask and Retrieve Knowledge: Towards Proactive Asking with Imperfect Information in Medical Multi-turn Dialogues. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1055-1065).
3. Yao, J., Jia, Y., Wang, X., & Khan, A. N. (2026). The digital gaze in retail encounters: AI face dialogues and consumer preference in the circular economy. Journal of Retailing and Consumer Services, 93, 104915.
4. Zhou, J., Xiong, H., Lu, J., Lin, Z., & Feng, B. (2025, September). Cgtgait: Collaborative graph and transformer for gait emotion recognition. In 2025 IEEE International Joint Conference on Biometrics (IJCB) (pp. 1-11). IEEE.
5. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: An ASR corpus based on public domain audio books. In Proceedings of ICASSP.
6. Zhao, R., Tang, J., Zeng, W., Guo, Y., & Zhao, X. (2025). Towards human-like questioning: Knowledge base question generation with bias-corrected reinforcement learning from human feedback. Information Processing & Management, 62(3), 104044.
7. Zhao, R., Zeng, W., Tang, J., Li, Y., Ye, G., Du, J., & Zhao, X. (2025, May). Towards unsupervised entity alignment for highly heterogeneous knowledge graphs. In 2025 IEEE 41St international conference on data engineering (ICDE) (pp. 3792-3806). IEEE.
8. Yao, Y., Zhang, J., Miao, P., Zhang, L., Chen, G., Shu, F., & Wong, K. K. (2025). Hybrid RIS-enhanced ISAC secure systems: Joint optimization in the presence of an extended target. IEEE Transactions on Communications.
9. Yu, X., Wei, Y., Zhou, S., Yang, Z., Sun, L., Peng, H., ... & Yu, P. S. (2025, April). Towards effective, efficient and unsupervised social event detection in the hyperbolic space. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 12, pp. 13106-13114).
10. Yao, Y., Zhao, J., Li, Z., Cheng, X., & Wu, L. (2023). Jamming and eavesdropping defense scheme based on deep reinforcement learning in autonomous vehicle networks. IEEE Transactions on Information Forensics and Security, 18, 1211-1224.
11. Zhang, Q., Gao, W., Liu, C., Yao, Y., Yan, S., Shu, F., & Jin, S. (2025). Covert transmission for active RIS-aided full-duplex UAV integrated sensing, communication, and computation systems. IEEE Journal on Selected Areas in Communications.
12. Wang, T., Li, L., Guo, H., Chen, Y., Li, Y., Wang, Y., ... & Chen, G. (2026). Anchored policy optimization: Mitigating exploration collapse via support-constrained rectification. arXiv preprint arXiv:2602.05717.
13. Li, Q., Ye, Q., Zhang, N., Zhang, W., & Hu, F. (2025). Digital-twin-enabled industrial IoT: Vision, framework, and future directions. IEEE Wireless Communications, 32(6), 173-181.
14. Li, X., Wang, P., Li, G., Ni, L., & Zhang, Y. (2023). Design of interface circuits and lightweight PUF for TMR sensors. IEEE Sensors Journal, 23(11), 11754-11761.
15. Liu, X., Yu, Z., Yao, Y., Yang, K., Zeng, H., & Chen, C. P. (2026). A Comprehensive Survey of Cross-Modal Retrieval: Breaking Through Modal Barriers. ACM Computing Surveys.
16. Pei, Z., Huang, Q., & Wang, S. (2026). When LLMs develop languages: Symbolic communication for efficient multi-agent reasoning. arXiv preprint arXiv:2606.29354.
17. Lin, Y., Yao, Y., Zhu, J., & He, C. (2025, July). Application of generative AI in predictive analysis of urban energy distribution and traffic congestion in smart cities. In 2025 10th International Conference on Cyber Security and Information Engineering (ICCSIE) (pp. 620-623). IEEE.
18. Lyu, M., Torii, R., Liang, C., Zhang, X., Wang, X., Li, Q., ... & Chen, D. (2025). Predictive computational framework to provide a digital twin for personalized cardiovascular medicine. Communications Medicine, 5(1), 370.
19. Emani, M., Xie, Z., Raskar, S., Sastry, V., Arnold, W., Wilson, B., ... & Boyd, M. (2022, November). A comprehensive evaluation of novel AI accelerators for deep learning workloads. In 2022 IEEE/ACM international workshop on performance modeling, benchmarking and simulation of high performance computer systems (PMBS) (pp. 13-25). IEEE.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Lars D. Dijkstra, Hannah Prins (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
