Russia unveils standard for reliable AI evaluation in Healthcare

RUSSIA—The Center for Diagnostics and Telemedicine in Russia has unveiled a groundbreaking method for evaluating artificial intelligence (AI) systems used in healthcare.

This method promises to make the process of integrating AI into clinical practice faster and more reliable.

This new approach is expected to streamline how AI technologies are adopted across Moscow’s healthcare facilities, ultimately improving patient care.

Traditionally, one of the biggest challenges in introducing AI into medicine has been the lack of clear guidelines on the number of medical studies needed to test these systems accurately.

This often meant that researchers had to rely on very large and resource-intensive sample sizes, which slowed down the adoption of new technologies.

However, the team at the Center for Diagnostics and Telemedicine has now developed an empirical methodology to determine the optimal number of studies required for reliable AI validation.

According to Yuri Vasilev, Chief Consultant for Diagnostic Imaging at the Moscow Health Care Department, Moscow has long been a leader in applying AI to medicine.

He emphasized that this new method represents a significant step forward.

Previously, the absence of clear standards made it difficult to know how many cases were necessary for objective AI testing, often resulting in unnecessary time and resource use.

Now, researchers can pinpoint the precise number of medical studies needed to ensure that AI systems are accurate and safe for clinicians to use.

The new methodology is based on a detailed analysis of over two million test scenarios and 25,000 medical images.

The researchers discovered that for AI systems using binary classification, such as those that detect the presence or absence of disease in medical images, a minimum of 400 studies is sufficient for a stable and objective assessment.

Importantly, at least 10% of these cases should represent each class, meaning that at least 40 studies should include cases with clearly marked pathologies.

The team found that increasing the sample size beyond this threshold does not improve the results, making the process much more efficient.

This approach has already proven effective in radiology, where it has been used to validate AI systems that help diagnose diseases from medical images.

 However, the methodology is versatile and can be applied to any medical AI system that uses binary classification.

This universality is a major advantage, as it allows for the rapid and reliable testing of new AI tools across different areas of medicine.

Vasilev highlighted that traditional testing methods could not definitively determine the necessary sample size to verify AI performance.

The new technique, however, offers a stable and universal framework, regardless of the type of medical image or AI system being evaluated.

This is expected to speed up the integration of AI into clinical practice, making it easier for doctors to use these advanced tools in their daily work.

The findings from this research are detailed in the paper “Empirical Approach to Sample Size Estimation for Testing of AI Algorithms,” which has been well received by the Russian Academy of Sciences and won recognition at the AI Journey competition.

The methodology is based on extensive empirical data and offers a robust alternative to traditional sample size calculations, which often fall short when applied to binary classification AI models.

Since 2020, the Center for Diagnostics and Telemedicine has been leading the world’s largest clinical study on the use of computer vision for medical image analysis.

The Center continues to develop innovative methods for assessing AI and integrating it into Moscow’s healthcare system.

Founded in 1996, the Center is a key institution under the Moscow Healthcare Department, dedicated to advancing AI in medicine, developing radiology, conducting research, and providing medical education.

This latest initiative supports the Moscow Healthcare Development Strategy 2030, which aims to improve the quality and accessibility of healthcare for all city residents.

Sign up HERE to receive our email newsletters with the latest news and insights from Africa and beyond. Also, follow us on our WhatsApp channel for updates.