Modular Adaptation of Regulatory Medical Texts to Improve the Relevance of LLM-Based Outputs: A Pilot Study Using Clinical Guidelines for Arterial Hypertension

E.M. Frolov, D.N. Ermolaeva, K.Yu. Mokshin, D.D. Shemonaev, M.Yu. Frolov, M.G. Zhabitsky

Abstract


The objective of this study was to evaluate, in a pilot setting, the effect of modular adaptation of clinical guideline texts on the quality of medical responses generated by a large language model (LLM) when solving real-world clinical tasks related to the management of arterial hypertension in adults. A total of 45 clinical scenarios formulated by practicing physicians were randomized into three groups: (1) adapted modular version of the clinical guideline (intervention group), (2) full unadapted guideline text (control group 2), and (3) no document support (control group 1). Responses were generated by GPT-4 using a custom prompt and assessed independently by experts using three separate criteria: clinical adequacy, safety, and compliance with the official recommendations, rated on a 0–10 scale. The adapted guideline group demonstrated the highest mean scores across all criteria (clinical adequacy – 8.8; safety – 9.5; compliance – 9.2), with statistically significant differences confirmed by the Mann–Whitney U test. The results show that excluding structurally irrelevant sections of the guideline (e.g. definitions, epidemiology, methodological notes) significantly improves the relevance and regulatory accuracy of LLM-generated recommendations. This approach may be applicable in the design of AI-based clinical decision support systems and the broader context of digital transformation in healthcare.

Full Text:

PDF (Russian)

References


Morone, G., De Angelis, L., Martino Cinnera, A., Carbonetti, R., Bisirri, A., Ciancarelli, I., Iosa, M., Negrini, S., Kiekens, C., & Negrini, F. (2025). Artificial intelligence in clinical medicine: a state-of-the-art overview of systematic reviews with methodological recommendations for improved reporting. Frontiers in digital health, 7, 1550731. Available: https://doi.org/10.3389/fdgth.2025.1550731.

Khanov, A.M., Gusev, A.V., Tyurganov, A.G. Prospects for the application of artificial intelligence technologies for the digital transformation of healthcare. Russian Journal of Telemedicine and Electronic Health 2024;10(3):70-76; Available: https://doi.org/10.29188/2712-9217-2024-10-3-70-76.

Yu E, Chu X, Zhang W, Meng X, Yang Y, Ji X, Wu C. Large Language Models in Medicine: Applications, Challenges, and Future Directions. Int J Med Sci. 2025 May 31;22(11):2792-2801. doi: 10.7150/ijms.111780.

Maity, S., & Saikia, M. J. (2025). Large Language Models in Healthcare and Medical Applications: A Review. Bioengineering (Basel, Switzerland), 12(6), 631. https://doi.org/10.3390/bioengineering12060631.

Vankov VV, Artemova OR, Karpov OE, Matvienko AV, Gusev AV, Enikeev IM, Kostina EV. Results of the implementation of artificial intelligence in Russian healthcare. Doctor and information technology. 2024;(3):32-43. https://doi.org/10.25881/18110193_2024_3_32.

Teles AS, Abd-alrazaq A, Heston TF, Damseh R and Ruback L (2025) Editorial: Large Language Models for medical applications. Front. Med. 12:1625293. doi: 10.3389/fmed.2025.1625293.

Li H, Fu J, Python A Implementing Large Language Models in Health Care: Clinician-Focused Review With Interactive Guideline J Med Internet Res 2025;27:e71916 Available: https://www.jmir.org/2025/1/e71916. DOI: 10.2196/71916.

Andreev DA, Kamynina NN. Prospects for the use of Chat-GPT information and communication technology in organizing medical care for patients with diabetes mellitus: a brief review of foreign literature. Doctor and information technology. 2024;(2):6-11. Available: https://doi.org/10.25881/18110193_2024_2_6.

Bélisle-Pipon J. C. (2024). Why we need to be careful with LLMs in medicine. Frontiers in medicine, 11, 1495582. Available: https://doi.org/10.3389/fmed.2024.1495582.

Hager, P., Jungmann, F., Holland, R. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30, 2613–2622 (2024). Available: https://doi.org/10.1038/s41591-024-03097-1.

Zong H, Wu R, Cha J, Wang J, Wu E, Li J, Zhou Y, Zhang C, Feng W, Shen B Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis J Med Internet Res 2024;26:e66114. doi: 10.2196/66114.

Wang, L., Chen, X., Deng, X. et al. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. npj Digit. Med. 7, 41 (2024). https://doi.org/10.1038/s41746-024-01029-4.

Zaghir J, Naguib M, Bjelogrlic M, Névéol A, Tannier X, Lovis C. Prompt Engineering Paradigms for Medical Applications: Scoping Review J Med Internet Res 2024;26:e60501. doi:10.2196/60501.

Satvik Tripathi, Dana Alkhulaifat, Shawn Lyo, Rithvik Sukumaran, Bolin Li, Vedant Acharya, Rafe McBeth, Tessa S. Cook, A Hitchhiker's Guide to Good Prompting Practices for Large Language Models in Radiology, Journal of the American College of Radiology, Volume 22, Issue 7, 2025, Pages 841-847, ISSN 1546-1440, Available: https://doi.org/10.1016/j.jacr.2025.02.051

European Society of Cardiology. - Guidelines. - Clinical Practice Guidelines. - Guidelines Derivative. - Products Pocket Guidelines. Available: https://www.escardio.org/Guidelines/Clinical-Practice-Guidelines/Guidelines-derivative-products/Pocket-Guidelines (date of access: 11.07.2025).

Algorithms for managing a patient with arterial hypertension / compiled by Zh. D. Kobalava, A. O. Konradi, S. V. Nedogoda, E. A. Troitskaya, A. S. Salasyuk. — 2nd ed. — St. Petersburg: Russian Society of Cardiology, 2024. — 68 p. — Electronic resource: https://www.ahleague.ru/images/rekom/Algoritmy_AG.pdf (date of access: 11.07.2025).

American College of Obstetricians and Gynecologists. (2019). Clinical guidelines and standardization of practice to improve outcomes (ACOG Committee Opinion No. 792). Obstetrics & Gynecology, 134(4), e122–e125.

ASH Pocket Guides. – Электронный ресурс: https://www.hematology.org/education/clinicians/guidelines-and-quality-care/pocket-guides (дата обращения: 11.07.2025).

Joint Royal Colleges Ambulance Liaison Committee, Association of Ambulance Chief Executives (2019) JRCALC Clinical Guidelines 2019. Bridgwater: Class Professional Publishing. ISBN-13: 9781859599020.

Armando LG, Miglio G, Pierluigi de Cosmo, Cena C - Clinical decision support systems to improve drug prescription and therapy optimisation in clinical practice: a scoping review: BMJ Health & Care Informatics 2023;30:e100683. https://doi.org/10.1136/bmjhci-2022-100683

Chen Y., Lehmann C.U., Malin B. Digital Information Ecosystems in Modern Care Coordination and Patient Care Pathways and the Challenges and Opportunities for AI Solutions. - J Med Internet Res 2024;26:e60258. doi: 10.2196/60258.

Martinez-Costa C, Schulz S. HL7 FHIR: Ontological Reinterpretation of Medication Resources. Stud Health Technol Inform. 2017;235:451-455. PMID: 28423833.

Alowais, S.A., Alghamdi, S.S., Alsuhebany, N. et al. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ 23, 689 (2023). https://doi.org/10.1186/s12909-023-04698-z.

Alhejaily A. G. (2024). Artificial intelligence in healthcare (Review). Biomedical reports, 22(1), 11. https://doi.org/10.3892/br.2024.1889/

Giebel GD, Raszke P, Nowak H, Palmowski L, Adamzik M, Heinz P, Tokic M, Timmesfeld N, Brunkhorst F, Wasem J, Blase N. Problems and Barriers Related to the Use of AI-Based Clinical Decision Support Systems: Interview Study. J Med Internet Res 2025;27:e63377. doi: 10.2196/63377

Ethics and governance of artificial intelligence for health. Guidance on large multi-modal models. Geneva: World Health Organization; 2024. Licence: CC BY-NC-SA 3.0 IGO.

Sridharan K, Sivaramakrishnan G. Unlocking the potential of advanced large language models in medication review and reconciliation: A proof-of-concept investigation. Explor Res Clin Soc Pharm. 2024 Aug 17;15:100492. doi: 10.1016/j.rcsop.2024.100492.

Han, T., Nebelung, S., Khader, F. et al. Medical large language models are susceptible to targeted misinformation attacks. npj Digit. Med. 7, 288 (2024). https://doi.org/10.1038/s41746-024-01282-7.

Clinical guidelines "Arterial hypertension in adults": ID 62_3 / developers: Russian Society of Cardiology and Russian Scientific Medical Society of Therapists. - Moscow, 2024. - Approved on 03.10.2024. - Electronic resource: https://cr.minzdrav.gov.ru/preview-cr/62_3 (date of access: 11.07.2025).


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность ИБП для ЦОД СНЭ

ISSN: 2307-8162