Novel products applying artificial intelligence (AI)-based methods to digital pathology images are touted to have many uses and benefits. However, publicly available information for products can be variable, with few sources of independent evidence. This review aimed to identify public evidence for AI-based products for digital pathology. Key features of products on the European Economic Area/Great Britain (EEA/GB) markets were examined, including their regulatory approval, intended use, and published validation studies. There were 26 AI-based products that met the inclusion criteria and, of these, 24 had received regulatory approval via the self-certification route as General in vitro diagnostic (IVD) medical devices. Only 10 of the products (38%) had peer-reviewed internal validation studies and 11 products (42%) had peer-reviewed external validation studies. To support transparency an online register was developed using identified public evidence (https://osf.io/gb84r/), which we anticipate will provide an accessible resource on novel devices and support decision making.
The increasing adoption of digital pathology, combined with the expansion of machine learning-based methodologies, has opened the door for novel products that generate predictions from the morphometric features within digital images. These artificial intelligence (AI)-based technologies have the potential to support the pathology workflow by improving diagnostic accuracy, assisting in case triage, and guiding treatment selection, which could alleviate pressure on pathology services, reduce costs, and improve patient outcomes. Realising these potential benefits, however, relies upon clinical deployment of well-validated algorithms, which have undergone thorough performance and usability testing.
High standards are expected of any medical device to be used in clinical practice, including software. Many organisations worldwide provide guidance on digital technology standards to be met for clinical adoption. In the UK, for example, the requirements for clinical performance, safety, security, and cost-effectiveness are defined in documentation including: the Evidence Standards Framework for Digital Health Technologies from the National Institute for Health and Care Excellence (NICE), the Digital Technology Assessment Criteria from NHSX, and the Guide to Good Practice for Digital and Data-Driven Health Technologies from the Department of Health and Social Care (DHSC). This guidance, alongside medical device legislation, is consistent in requiring new technologies to be thoroughly tested on representative datasets during clinical validation. For digital pathology in particular, rigorous validation of algorithms on diverse test datasets is essential to demonstrate accuracy and generalisability, as the process of digital image creation can introduce site-specific variation through differences in glass slide preparation, staining protocols/reagents, scanner platforms, image formats, and scan quality. However, as new AI-based products continue to appear on the market, it can be challenging to ascertain the level of testing a new product has undergone, including the source and composition of test datasets.
All novel medical devices must obtain regulatory approval prior to placement on the EEA/GB markets. This involves a conformity assessment process whereby manufacturers gather evidence to demonstrate compliance with medical device legislation. For software, this covers elements including safety and performance, risk management, and product verification and validation. In the EEA, AI-based image analysis software is typically considered an in vitro diagnostic (IVD) medical device, and is therefore subject to the In Vitro Diagnostic Medical Devices Regulation (2017/746) (IVDR), which applied from May 2022 (Fig. 1a). This replaced older legislation (the IVD Directive 98/79/EC; IVDD), which is currently mirrored in the existing equivalent UK legislation (UK MDR 2002). One crucial difference between the original IVDD/UK MDR and new IVDR is that, under IVDD, AI-based image analysis software for digital pathology was typically considered a 'General IVD', which is the lowest device class. This allows manufacturers to 'self-certify' their device as conforming with essential requirements (Fig. 1b). Contrastingly, under the new IVDR, similar software is more likely to be a higher-risk Class C device, and therefore must meet higher safety and performance standards, and requires evidence review by a conformity assessment body. At present, there is a transitional period which allows certain IVDD-compliant devices with a valid Conformité Européenne (CE)-mark to continue to be placed on the EEA market up to May 2027, depending on the device class (Fig. 1a). Following withdrawal from the EU, different rules now apply to the GB market (Fig. 1a, b). Currently, most IVDs registered with the MHRA can be placed on the GB market if they: (1) conform to the UK MDR 2002 requirements to receive the UK Conformity Assessed (UKCA) marking; or (2) if they have a valid certificate or declaration of conformity and CE-mark issued before May 2022 under the EU IVDD; or (3) if they have a valid certificate or declaration of conformity and CE-mark under the new EU IVDR (see Fig. 1a, b for further details). Consequently, AI-based medical devices on the EEA/GB market across this ongoing transition period may be compliant with different standards and requirements, which compels the need for independent evaluation of novel emerging products.
In the USA, a similar risk-based classification system is used for medical devices, with The Food and Drug Administration (FDA) being responsible for market approval. Devices must conform to the regulatory controls in the Federal Food, Drug, and Cosmetic Act and the Code of Federal Regulations Title 21. The three most common routes for medical device market approval are: (1) The Premarket Notification/510(k) approval pathway, which is for medium risk devices (Class I & II) and requires vendors to prove substantial equivalence to an already-marketed device; (2) the Premarket Approval pathway, which is for high risk (Class III) and novel devices; and (3) the De Novo Classification Request pathway, which is for novel Class I and II devices. In addition, Class I and some Class II products can be exempt from premarket submission as long as they comply with regulatory controls laid out in the relevant legislation. To date, the only AI-based product for digital pathology to receive approval for placement on the USA market was approved in September 2021 via the De Novo pathway as a Class II device. Furthermore, for machine learning-based medical devices, the FDA (alongside Health Canada and the MHRA) is considering a requirement for manufacturers to submit a Predetermined Change Control Plan (PCCP), which would address how they intend to manage future device modifications.
Although conformity assessment does not guarantee a product will make it into clinical use, it is a crucial step in making a product available on the market and commencing further evaluation and procurement activities. However, in the EEA/GB there is currently no requirement to make the evidence provided for conformity assessment public, even in summary form. The EU recently established the EU Database on Medical Devices (EUDAMED), alongside the transition to IVDR, which will provide public, searchable listings of all companies producing medical devices, as well as details of their products, certifications, approvals etc. However, this is under construction and currently contains limited information. This contrasts with the USA, where the FDA maintains databases of approved medical devices, and publishes detailed decision summaries on approved devices. In the UK, public evidence on medical devices can be released when products undergo a Health Technology Evaluation by NICE, which is associated with an independent, published evaluation summary. However, these evaluations are typically only conducted on select products which have received regulatory approval and have been recommended for a NICE evaluation. In radiology, NICE Guidance has so far been released for three forms of AI-based technology for medical image analysis. In digital pathology, there are no AI-based products associated with NICE Guidance, however, one product has undergone a NICE MedTech Innovation Briefing, which is a technology summary classed as 'NICE Advice' and is designed to support local decision-making rather than providing a recommendation or constituting formal NICE Guidance. Thus, for novel regulatory-approved products, it can be challenging to locate reliable public information about a product and its clinical evidence.
Maintaining transparency around novel medical devices, particularly those using AI-based methods, is crucial in promoting confidence, trust, and understanding. Details on novel technologies should be easily accessible for pathologists, patients, clinicians, commissioners, and other key stakeholders, to support informed decision-making, and promote the implementation of safe and effective technologies. In radiology, one initiative has created and maintained an index of all known AI-based products on the EU market. This currently details 210 commercially available tools for radiology, with details including their regulatory approval, intended use, integration, and associated publications. Similarly, the Royal College of Radiologists and NHS England have recently proposed to construct an open UK-based registry, to document where AI-based products are in use, and to facilitate comparison of their performance. In the USA, there is also an independent database dedicated to AI-based medical devices approved by the FDA, with details on how they were evaluated. However, we are not aware of any resources that currently provide a public registry of commercially available AI products on the EEA/GB market for digital pathology.
In light of these evolving initiatives, and to facilitate transparency, we sought to identify and analyse the public evidence on commercially available AI-based products on the EEA/GB market for digital pathology. Here, we describe our findings in relation to the current landscape of available products, their primary function, regulatory approvals, and clinical validation. We have compiled this information into a publicly accessible registry of products, including all available evidence relating to a product's features and performance: https://osf.io/gb84r/. It is hoped this will provide a resource to support pathologists, patients, and commissioners in understanding the tools available for diagnostic support and their associated evidence. To assemble this registry, we performed a very broad search of the grey literature to identify as many relevant products as possible. Given the extensive and detailed search required for this review, we prioritised identifying products for haematoxylin and eosin (H&E) staining as the most commonly used sample by histopathologists, therefore the scope of this work does not include immunohistochemistry images (IHC). However, it is important to acknowledge that AI for IHC staining is a growing area of interest, with many exciting developments (see e.g. refs. ). It is our intention to examine the public evidence for these products in more detail within future work.
