Embedding and Semantic Retrieval Based Evidence Traceability Model For LAM INFOKOM Accreditation Evidence Matching Using LLM RAG
DOI:
https://doi.org/10.58982/q7nz1b70Keywords:
Evidence traceability, Evidence matching, Semantic retrieval, Retrieval-Augmented Generation (RAG), LAM INFOKOM accreditationAbstract
This research addresses the critical need for auditable evidence matching in high-stakes LAM INFOKOM accreditation, where assessors face lengthy, heterogeneous documents, leading to inconsistent interpretation and weak audit trails. The study proposes a novel evidence traceability model based on embedding and semantic/hybrid retrieval within a Retrieval-Augmented Generation (RAG) framework. The model aims to map indicator-bound claims to relevant, faithful evidence at the span and multi-document bundle level, ensuring explicit citations. Methods include structured parsing and chunking to generate identifiable evidence spans, embedding and hybrid retrieval (dense + BM25) with reranking to form evidence bundles, and a GPT-4 based LLM-RAG system with mandatory span ID citation. Evaluation uses RAGAS metrics (faithfulness, answer relevance, context precision/recall) and a novel 5-level traceability rubric, complemented by ablation studies and practitioner validation. The key finding is a proof-of-concept prototype demonstrating end-to-end traceability from claims to evidence spans, producing an auditable traceability graph. The research confirms that integrating structured retrieval with mandatory citation significantly improves groundedness over standard RAG approaches. The proposed model offers a systematic solution for transparent, auditable accreditation evaluation, with potential applications in other formal audit domains.
References
[1] M. Zhu, A. Ahuja, D.-C. Juan, W. Wei, and C. K. Reddy, “Question Answering with Long Multiple-Span Answers,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 3840–3849. doi: 10.18653/v1/2020.findings-emnlp.342.
[2] Y.-M. Wang, H.-W. Shen, T.-J. Chen, S.-C. Chiang, and T.-G. Lin, “Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study,” JMIR Med Educ, vol. 11, pp. e56850–e56850, Jan. 2025, doi: 10.2196/56850.
[3] J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a Large-scale Dataset for Fact Extraction and VERification,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2018, pp. 809–819. doi: 10.18653/v1/N18-1074.
[4] M. Klesel and H. F. Wittmann, “Retrieval-Augmented Generation (RAG),” Business & Information Systems Engineering, vol. 67, no. 4, pp. 551–561, Aug. 2025, doi: 10.1007/s12599-025-00945-3.
[5] C. He, W. He, M. Liu, S. Leng, and S. Wei, “Enriched Construction Regulation Inquiry Responses: A Hybrid Search Approach for Large Language Models,” Journal of Management in Engineering, vol. 41, no. 3, May 2025, doi: 10.1061/JMENEA.MEENG-6444.
[6] M. Buffa, A. Ferrara, S. Picascia, D. Riva, and S. Castano, “Enhancing legal document building with Retrieval-Augmented Generation,” Computer Law & Security Review, vol. 59, p. 106229, Nov. 2025, doi: 10.1016/j.clsr.2025.106229.
[7] T. Yu et al., “EKRAG: Benchmark RAG for Enterprise Knowledge Question Answering,” in *Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing*, Stroudsburg, PA, USA: Association for Computational Linguistics, 2025, pp. 152–159. doi: 10.18653/v1/2025.knowledgenlp-1.13.
[8] M.-K. Ghali, A. Farrag, D. Won, and Y. Jin, “Enhancing knowledge retrieval with in-context learning and semantic search through generative AI,” Knowl Based Syst, vol. 311, p. 113047, Feb. 2025, doi: 10.1016/j.knosys.2025.113047.
[9] Q. Chen, W. Zhou, J. Cheng, and J. Yang, “An Enhanced Retrieval Scheme for a Large Language Model with a Joint Strategy of Probabilistic Relevance and Semantic Association in the Vertical Domain,” Applied Sciences, vol. 14, no. 24, p. 11529, Dec. 2024, doi: 10.3390/app142411529.
[10] I. O. William and M. Altamimi, “Text Embedding Implementation Using Retrieval Augmented Generation (RAG) Model Combined With Large Language Model,” International Conference on Engineering, Natural and Social Sciences, no. Llm, pp. 637–645, 2024.
[11] S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAs: Automated Evaluation of Retrieval Augmented Generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 150–158. doi: 10.18653/v1/2024.eacl-demo.16.
[12] E. Oro, F. M. Granata, A. Lanza, A. Bachir, L. De Grandis, and M. Ruffolo, “Evaluating Retrieval-Augmented Generation for Question Answering with Large Language Models,” CEUR Workshop Proc, vol. 3762, pp. 129–134, 2024.
[13] X. Jing, S. Billa, and D. Godbout, “On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation,” in Findings of the Association for Computational Linguistics: NAACL 2025, Stroudsburg, PA, USA: Association for Computational Linguistics, 2025, pp. 7765–7780. doi: 10.18653/v1/2025.findings-naacl.433.
[14] N. Dziri, H. Rashkin, T. Linzen, and D. Reitter, “Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark,” Trans Assoc Comput Linguist, vol. 10, pp. 1066–1083, Sep. 2022, doi: 10.1162/tacl_a_00506.
[15] J. Qi, G. Sarti, R. Fernández, and A. Bisazza, “Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 6037–6053. doi: 10.18653/v1/2024.emnlp-main.347.
[16] J. Vladika, L. Mülln, and F. Matthes, “Enhancing Answer Attribution for Faithful Text Generation with Large Language Models,” in Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, SCITEPRESS - Science and Technology Publications, 2024, pp. 147–158. doi: 10.5220/0013066600003838.
[17] N. Patel, S. Subramanian, S. Garg, P. Banerjee, and A. Misra, “Towards Improved Multi-Source Attribution for Long-Form Answer Generation,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 3906–3919. doi: 10.18653/v1/2024.naacl-long.216.
[18] N. W. Sumartini Saraswati, I. K. Gede Darma Putra, M. Sudarma, I. M. Sukarsa, C. P. Yanti, and N. K. Tri Juniartini, “Revealing the Potential of Hotel Improvements in Bali Based on Sentiment Analysis and Tourist Characteristics,” in 2024 11th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), IEEE, Sep. 2024, pp. 722–728. doi: 10.1109/EECSI63442.2024.10776092.
[19] N. W. S. Saraswati, I. Ketut Gede Darma Putra, M. Sudarma, and I. Made Sukarsa, “Enhance sentiment analysis in big data tourism using hybrid lexicon and active learning support vector machine,” Bulletin of Electrical Engineering and Informatics, vol. 13, no. 5, pp. 3663–3674, Oct. 2024, doi: 10.11591/eei.v13i5.7807.
[20] N. W. S. Saraswati and I. G. A. A. Diatri Indradewi, “Recognize The Polarity of Hotel Reviews using Support Vector Machine,” MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, vol. 22, no. 1, pp. 25–36, Nov. 2022, doi: 10.30812/matrik.v22i1.1848.
[21] N. W. S. Saraswati, I. P. K. S. Putra, I. D. M. K. Muku, and G. D. Pramitha, “Support Vector Machine For Hoax Detection,” SINTECH (Science and Information Technology) Journal, vol. 6, no. 2, pp. 107–117, Aug. 2023, doi: 10.31598/sintechjournal.v6i2.1366.
[22] N. W. Sumartini Saraswati, I. K. G. Darma Putra, M. Sudarma, and I. M. Sukarsa, “The Image of Tourist Attraction in Bali Based on Big Data Analytics and Sentiment Analysis,” in *2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS)*, IEEE, Nov. 2023, pp. 82–87. doi: 10.1109/ICSGTEIS60500.2023.10424322.
[23] I. W. Agus Surya Darma, P. R. Mahendra Putra, P. Sugiartawan, V. Waas, and N. P. Sutramiani, “A Fine-tuned BERT-based Approach for Sentiment Analysis of Indonesian Public Towards ChatGPT,” in *2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS)*, IEEE, Nov. 2023, pp. 88–93. doi: 10.1109/ICSGTEIS60500.2023.10424123.
[24] P. Sugiartawan and S. Hartati, “Time Series Data Prediction using Elman Recurrent Neural Network on Tourist Visits in Tanah Lot Tourism Object,” Int J Eng Adv Technol, vol. 9, no. 1, pp. 314–320, Oct. 2019, doi: 10.35940/ijeat.A1833.109119.
[25] I. M. B. H. Irawan, P. Sugiartawan, N. W. S. Saraswati, and I. M. D. P. Asana, “AI-Assisted Accreditation Evaluation Using a Web- Based GPT-RAG System within the LAM INFOKOM Framework,” in 2025 IEEE International Symposium on Consumer Technology (ISCT), IEEE, Sep. 2025, pp. 421–426. doi: 10.1109/ISCT66099.2025.11297403.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 I Made Bayu Hadi Irawan, Putu Sugiartawan, N. W. Sumartini Saraswati
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
