Moscow scientists publish guide for evaluating AI quality in healthcare

RUSSIA—Scientists at the Center for Diagnostics and Telemedicine of the Moscow Department of Health have published a new study guide on evaluating the quality and performance of artificial intelligence (AI) technologies used in healthcare.

Titled Assessing the Quality and Performance Parameters of Artificial Intelligence Technologies in Healthcare, the guide draws on scientific and practical findings from the Moscow Experiment, conducted between 2020 and 2025.

The initiative focused on using computer vision to analyze medical images and supported the development of methods for evaluating AI technologies in clinical settings.

Based on the experiment, the authors developed a methodology for monitoring AI quality and performance throughout the technology lifecycle.

The guide also examines Russian national standards for AI systems used in clinical medicine and provides practical tools for assessing their performance.

Focus on safety, accuracy and clinical performance

The publication provides practical guidance on determining whether an AI system operates accurately, performs efficiently and delivers meaningful support to healthcare professionals.

It includes metrics, checklists, performance thresholds and procedures for assessing the safety and reliability of AI systems before and during clinical use.

“Over the past five years, as part of the Moscow Experiment, we have progressed from test implementations to the creation of a comprehensive quality-control system for medical AI,” said Yuri Vasiliev, Medical Director of the Center for Diagnostics and Telemedicine.

He added that the guide focuses on practical measures for determining whether AI systems can be trusted in real clinical environments.

The guide also addresses the evaluation of large language models (LLMs), including systems capable of analyzing patient records and generating summaries from medical documents.

In addition, it incorporates official Russian national standards, known as GOSTs, alongside research and methodologies developed by the authors.

Resource for healthcare and technology professionals

The publication targets students and residents studying General Medicine, Pediatrics and Medical Cybernetics, as well as programmers, testers and other professionals working with medical information systems.

According to Anton Vladzimirsky, Deputy Director for Research at the Center for Diagnostics and Telemedicine, the guide combines conventional AI performance metrics with tools for evaluating generative AI and introduces a “maturity matrix” for distinguishing established solutions from early-stage prototypes.

The authors said many of the methods and metrics presented in the guide emerged from the Moscow Experiment and the analysis of millions of medical images.

They aim to equip healthcare professionals and students with the skills to assess both the capabilities and limitations of AI while maintaining patient safety through continuous monitoring.

The textbook was recommended for use in higher education institutions offering Medical Informatics programs under Protocol No. 095, dated December 18, 2025.

Professor Nikolai Nudnov of the Russian Scientific Center of Roentgenology and Radiology and Professor Georgy Lebedev of Sechenov University served as reviewers.

The Center for Diagnostics and Telemedicine supports radiology and instrumental diagnostics, digital healthcare transformation, clinical AI implementation, scientific research and professional training under the Moscow Department of Health.

Sign up HERE to receive our email newsletters with the latest news and insights from Africa and around the world, and follow us on LinkedIn for updates.