An optometrist’s checklist for AI use
Things to keep in mind when using AI and five questions to ask
Given the hype and excitement surrounding AI technologies, before using it professionally and/or clinically, it would be beneficial to ask yourself some key questions.
The questions to ask
1. Does the AI system have the regulatory approval (MHRA/UKCA)?
Systems with regulatory approval can be used in clinical practice as they have fulfilled the standard required and any/or rigorous testing performed by an adequate assessor.
Approved devices are searchable on the Medical and Healthcare Products Regulatory Agency.
You can also find out more information about the approval process on our Regulated AI and Software as a medical device page.
2. What is the system intended/designed for and how will we use it in practice?
AI software products are usually designed for a specific purpose. They generally cannot perform outside the intended scope. This is particularly relevant with anything diagnostic. A system designed to detect retinopathy is blind to other pathologies. MHRA approval of Software as a Medical Device (SaMD) and Artificial Intelligence as a Medical Device (AIaMD) requires the manufacturer to clearly display the intended purpose of the product, so please verify for yourself before using.
Further, the manufacturer should also provide the necessary information on a products efficacy for the intended purpose, look for scores of positive predictive value (PPV; also called precision) and the Specificity versus Sensitivity principle (see Glossary). To illustrate, algorithms with a high sensitivity and low specificity may be excellent for screening but create too many false positives. Always consider the purpose of the tool.
3. Has it been trained on the target population?
Access to good quality training data is imperative for developing effective AI tools. Images should be drawn from diverse patient groups, be demographically annotated, and have a full range of pathologies that the tool is being trained to monitor/diagnose. Results produced by an AI system have to be viewed with caution if the input data is too narrow. This is especially important for optometrists who work in a range of settings with diverse populations.
To further understand and trust that a product has been trained a sufficiently wide and/or appropriate dataset, you should look to the manufacturers notes for clear documentation - sometimes referred to as "model cards" or "model facts" labels. These that explain the AI algorithm's capabilities and limitations, including the characteristics of the training dataset, such as demographics.
The MHRA registration process for medical devices will have scrutinised the datasets used in the context of the product’s intended purpose. Manufacturers must provide documented evidence that they have identified, measured, and managed risks from bias, and ensured the generalisability of their data before MHRA will approve a product.
More information
Checklist: How to evaluate AI tools for healthcare – Health AI CPD
4. Should I be using ChatGPT and other readily available AI tools?
In short, widely available generative AI tools such as the group of Large Language Models (LLMs) including ChatGPT, Microsoft CoPilot and Gemini must never be used when seeking advice for patients without fact-checking the information first. ChatGPT is NOT a medical device. Furthermore, any patient-identifiable information should never be input into open-access LLMs.
The primary method for LLMs is to aggregate and regurgitate based on existing content. Parameters of accuracy are not necessarily in place. The more that these models are used, the less information will be curated for accuracy, and a new self-selected circular truth will be created.
Proceed with caution and always fact check anything that is generated by the available LLMs and never input patient identifiable information into open-access LLMs.
Further detail on the potential pitfalls are articulated in the Interim position on the use of artificial intelligence in eye care from the College of Optometrists.
5. Can you understand and explain how the tool arrived at its output?
Interpretability or explainability of the AI tool and how it reached its conclusion is important for understanding and trusting their use. While there is universal agreement that AI in healthcare should never be a “black box”, the MHRA (and equivalent bodies globally) are still working through how best to ensure that manufacturers comply.
Indeed, they note “the challenges and opportunities posed by AIaMD, ensuring that these devices are appropriately evidenced, as well as addressing wider issues of transparency of AI (both explainability and interpretability), and adaptivity (retraining of AI models)”.
Unfortunately, this means that it is not so easy for the busy frontline optometrist to always determine the “how” of an AI product. The MHRA will hopefully address this in the upcoming secondary legislation.
Look for tools that offer interpretable outputs or explanations for the AI’s reasoning until the regulators ensure that this is part of the registration requirements. The AOP will continue to monitor this and will update you.
