Toward Generalizable LLM Pipelines for Annotation and Knowledge Organization

Authors

  • Julian Kensington Department of Computing, Hong Kong Polytechnic University, Hong Kong, China Author
  • Beatrice Crawford Department of Computing, Hong Kong Polytechnic University, Hong Kong, China Author

DOI:

https://doi.org/10.66525/ijictt79

Keywords:

Large Language Models, Data Annotation, Knowledge Extraction, Pipeline Architecture, Generalization

Abstract

The rapid evolution of natural language processing has introduced unprecedented capabilities for information extraction, yet the construction of robust, generalizable pipelines for data annotation and knowledge organization remains a critical challenge. Traditional automated annotation systems often suffer from domain over-fitting, requiring extensive manual intervention and retraining when applied to novel schemas or unseen textual distributions. In this paper, we propose a comprehensive, domain-agnostic pipeline framework that leverages the reasoning and zero-shot capabilities of large language models to automate complex annotation workflows. Our methodology integrates dynamic prompt optimization, structural constraint satisfaction, and multi-agent verification to bridge the gap between unstructured text and highly organized knowledge graphs. We systematically evaluate the proposed architecture across multiple distinct domains, including biomedical literature, scientific publications, and general news corpora, comparing its performance against established supervised baselines and naive prompting strategies. The empirical results demonstrate that our generalized pipeline achieves substantial improvements in both annotation accuracy and structural consistency, yielding an average F1 score increase of up to fifteen percent over existing zero-shot approaches while drastically reducing the necessity for human-in-the-loop corrections. Furthermore, exhaustive ablation studies reveal the vital contributions of specific architectural components, particularly the dynamic few-shot retrieval module and the knowledge alignment validator. This research provides a foundational blueprint for deploying scalable, autonomous annotation systems, significantly reducing the computational and financial overhead associated with organizing unstructured human knowledge into machine-readable structures.

Downloads

Download data is not yet available.

References

Ma, Y., Qu, D., & Pyrozhenko, M. (2026). Bio-RegNet: A Meta-Homeostatic Bayesian Neural Network Framework Integrating Treg-Inspired Immunoregulation and Autophagic Optimization for Adaptive Community Detection and Stable Intelligence. Biomimetics, 11(1), 48.

Tu, P., Huang, Y., Zheng, F., He, Z., Cao, L., & Shao, L. (2022, June). Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 2379-2387).

Xie, C. (2026). Quantifying the Interplay Between Panic Propagation and Misinformation on Social Media Using Large Language Models. Frontiers in Artificial Intelligence Research, 3(1), 1-8.

Guo, Y., Sekiguchi, Y., Zeng, W., Ebihara, S., Owaki, D., & Hayashibe, M. (2025). Physics-informed learning framework for lower limb kinematic prediction with sparse sensors and its application in chronic stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering.

Ma, F., Liu, L., & Cheng, H. V. (2024). TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability. arXiv preprint arXiv:2405.17678.

Zhang, Y., Liu, J., Wang, J., Dai, L., Guo, F., & Cai, G. (2025, February). Federated learning for cross-domain data privacy: A distributed approach to secure collaboration. In 2025 8th International Symposium on Big Data and Applied Statistics (ISBDAS) (pp. 824-828). IEEE.

Zhang, Y., Valentino, M., Carvalho, D., Pratt-Hartmann, I., & Freitas, A. (2024, June). Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 474-489).

Jiang, A., Huang, J., & Yun, X. (2026, March). Design and empirical research of simulation algorithms for business decision support with BERT and ISAC integration. In International Conference on Cloud Computing, Performance Computing, and Deep Learning (CCPCDL 2025) (Vol. 14131, pp. 331-336). SPIE.

Zhang, G., Zeng, H., & Jiang, L. (2026). Uni-FinLLM: A Unified Multimodal Large Language Model with Modular Task Heads for Micro-Level Stock Prediction and Macro-Level Systemic Risk Assessment. arXiv preprint arXiv:2601.02677.

Ou, Y., de Bruijn, G. J., & Schulz, P. J. (2025). Social media as an emotional barometer: Bidirectional encoder representations from transformers–long short-term memory sentiment analysis on the evolution of public sentiments during Influenza A on Sina Weibo. Journal of Medical Internet Research, 27, e68205.

Li, M., Dong, Q., Zhang, D., Qin, K., & Luo, G. (2026). Efficient Industrial Dataset Distillation With Textual Trajectory Matching. IEEE Transactions on Industrial Informatics.

Wang, S., Yu, Y., Feldt, R., & Parthasarathy, D. (2025). Automating a complete software test process using llms: An automotive case study. arXiv preprint arXiv:2502.04008.

Ou, Y., Zhang, P., Yu, J., Li, M., Su, S., Zhang, M., ... & Wu, J. (2025, February). The application of the BERTopic model in natural language processing: In-depth text topic modeling. In 2025 5th International Conference on Consumer Electronics and Computer Engineering (ICCECE) (pp. 793-796). IEEE.

Zhang, H., Zhao, S., Zhou, Z., Zhang, W., & Meng, Y. (2025, September). Domain-Specific RAG with Semantic Normalization and Contrastive Feedback for Document Question Answering. In 2025 7th International Conference on Internet of Things, Automation and Artificial Intelligence (IoTAAI) (pp. 750-753). IEEE.

Chen, Y. (2025). The Lexical Bundles and Discourse Markers Between Bilingual and Monolingual Teachers’ Talk: A Corpus-Based Study. Florida Journal of Educational Research, 62(3), 19-31.

Zeng, D., Yang, Y., Tang, Y., Zhao, L., Wang, X., Yun, D., ... & Lin, H. (2025). Shaping school for childhood myopia: the association between floor area ratio of school environment and myopia in China. British Journal of Ophthalmology, 109(1), 146-151.

Tang, Y., Kojima, K., Gotoda, M., Nishikawa, S., Hayashi, S., Koike-Akino, T., ... & Klamkin, J. (2020). Design and Optimization of Shallow-Angle Grating Coupler for Vertical Emission from Indium Phosphide Devices.

Kong, R., Li, Y., Feng, Q., Wang, W., Ye, X., Ouyang, Y., ... & Liu, Y. (2024, August). SwapMoE: Serving off-the-shelf MoE-based large language models with tunable memory budget. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6710-6720).

Li, Z., Zhang, Y., Pan, T., Sun, Y., Duan, Z., Fang, J., ... & Wang, J. (2025, July). FocusLLM: Precise understanding of long context by dynamic condensing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 31087-31101).

Liang, H., Huang, C., Xu, Y., Tang, C., Ye, W., Zhang, J., ... & Xu, L. (2024). Llava-slt: Visual language tuning for sign language translation. arXiv preprint arXiv:2412.16524.

Zhang, W., Zhang, C., Gu, C., Kou, J., Yuan, H., Fang, X., ... & Fang, Y. (2024, October). Hallucination in Large Language Models: From Mechanistic Understanding to Novel Control Frameworks. In 2024 7th International Conference on Universal Village (UV) (pp. 1-36). IEEE.

Zhao, H., Qi, Z., Wang, C., Zheng, Q., Lu, G., Chen, F., ... & Wu, Z. (2025). Dynamictrl: Rethinking the basic structure and the role of text for high-quality human image animation. arXiv preprint arXiv:2503.21246.

Yifan, O. U. (2018). Participating in Chinese Social Question and Answer Communities: A Case Study of Zhihu. com.

Li, J., Shuang, K., Guo, J., Shi, Z., & Wang, H. (2023). Enhancing semantic relation classification with shortest dependency path reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 1550-1560.

Du, C., Chiang, C. E., Huang, T., & Cui, Z. (2025, September). Adaptive Graph Convolution and Semantic-Guided Attention for Multimodal Risk Detection in Social Networks. In 2025 5th International Conference on Artificial Intelligence, Automation and High Performance Computing (AIAHPC) (pp. 507-512). IEEE.

Ding, H., Fang, Y., Zhu, R., Jiang, X., Zhang, J., Xu, Y., ... & Wang, Y. (2024). 3ds: Decomposed difficulty data selection’s case study on llm medical domain adaptation.

Vuruma, S. K. R., Wu, D., Gupta, S. S., Aust, L., Lookingbill, V., Bellamy, W., ... & Huang, M. (2024). Can GPT-4 Help Detect Quit Vaping Intentions? An Exploration of Automatic Data Annotation Approach. arXiv preprint arXiv:2407.00167.

Li, B., Gu, B., & Ding, Z. (2025). LLM-based Personalized Portfolio Recommender: Integrating Large Language Models and Reinforcement Learning for Intelligent Investment Strategy Optimization. arXiv preprint arXiv:2512.12922.

Gao, Z., Qu, Y., & Han, Y. (2025). Cross-Lingual Sponsored Search via Dual-Encoder and Graph Neural Networks for Context-Aware Query Translation in Advertising Platforms. arXiv preprint arXiv:2510.22957.

Lou, Y., Liu, J., Sheng, Y., Wang, J., Zhang, Y., & Ren, Y. (2025, March). Addressing class imbalance with probabilistic graphical models and variational inference. In 2025 5th International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) (pp. 1238-1242). IEEE.

Zhang, L. (2025). MCP: Control-theoretic orchestration for multimodal large language models. arXiv preprint arXiv:2509.16597

Wang, Y., Song, R., Li, L., Zhang, R., & Liu, J. (2025). Dynamically allocated interval-based generative linguistic steganography with roulette wheel. Applied Soft Computing, 176, 113101.

Fan, D., Sun, B., Dui, H., Zhong, J., Wang, Z., Ren, Y., & Wang, Z. (2022). A modified connectivity link addition strategy to improve the resilience of multiplex networks against attacks. Reliability Engineering & System Safety, 221, 108294.

Ahmad, N. R. (2025). Exploring the role of digital technologies in enhancing supply chain efficiency: A case study of e-commerce companies. Indus Journal of Social Sciences, 3(1), 226–237. https://doi.org/10.59075/ijss.v3i1.618

Downloads

Published

2026-04-15