A multi-centre prospective evaluation of THEIA to detect diabetic retinopathy (DR) and diabetic macular oedema (DMO) in

While there has been a flurry of research designed to create artificial intelligence tools for screening diabetic retinopathy (DR) or diabetic macular oedema (DMO), few algorithms have been tested prospectively in a real-world clinical environment [20,21,22,23]. This study, was designed to test the efficacy of our previously published algorithm (THEIA™) in a real-world prospective setting of two DR screening programs in New Zealand [5, 6] (Supplementary Table 10); one an urban DHB tertiary hospital screening centre the other a provincial Optometrist led screening centre. In this multi-centre prospective trial of 900 patients, at the patient level when a binary Non referrable/referrable classification was used THEIA™ achieved 100% imageability, 100% sensitivity, 98% specificity, with an overall accuracy of 98% for identifying referable disease when compared to an adjudicated gold standard. When a more granular classification of none. Mild, mtmDR and sight threatening disease was used THEIA™ missed no patient who had referrable disease as defined by either “mtmDR” or Sight threatening disease. The few inconsistent grades between THEIA™ and the adjudicated gold standard were largely a result of drusen; both small and hard drusen, and large pachydrusen [24], being mistaken for exudates. The gold standard adjudicated dataset was derived from grades issued by the senior lead grader in each of the three metro Auckland DHB screening programs. In keeping with their experience, the level of agreement between the individual graders and the adjudicated gold standard (k value: 0.92–0.98) when the data was aggregated into Referrable vs Non referrable disease, was excellent. Although no cases of referrable disease were missed, all three of the human graders marginally under-graded compared to the adjudicated gold standard. This result was not statistically significant. There was a comparable level of agreement between THEIA™ and the adjudicated gold standard (k value: 0.95). In contrast to the human graders, THEIA™ marginally over-graded the images, a result which is in keeping with a tool which is designed with a high sensitivity and thus designed not to miss disease. Overall, these results demonstrate that THEIA™ is both reliable and is as consistent as experienced specialist graders in diagnosing and detecting referrable diabetic retinopathy and maculopathy in the New Zealand (or similar) screening program.

As expected, the accuracy of the level of agreement, for both the human graders and THEIA™ was reduced when the more granular grading system; None Detected, Mild, mtmDR, Sight threatening, was employed. The apparent drop off in performance of both the human graders and THEIA™, (k value: human graders 0.96–0.75; THEIA 0.79), is a function of a number of compounding factors; the imposition of an ordinal scale onto a what is disease continuum leading to an increased probability of a mismatch at what is an artificial boundary of two disease states, and the increased numbers of boundaries that a more granular grading system imposes. To reduce the likelihood of missing disease THEIA™ has therefore been designed with an inbuilt bias to over grade in situations where the disease sits at the boundary threshold of two disease states. Reassuringly THEIA™ accurately predicted the correct grade of retinopathy in 82% cases of retinopathy and 89% cases of maculopathy when these two conditions were considered as different entities. When retinopathy and maculopathy grades were aggregated THEIA™ under graded sight threatening disease in just 7 cases, but in all cases THEIA™ still correctly identified them as “referrable” disease labelling them instead as “mtmDR”.

Compared to other algorithms which have been assessed prospectively in a real world setting [7,8,9, 25], THEIA™ performed very favourably. These results suggest that THEIA™ is capable of providing a very high granularity in the diagnosis of both retinopathy and maculopathy. Furthermore, unlike other clinically tested AIs [26,27,28,29], THEIA™ provides these disease grades based on all images acquired per screening visit with the whole process from image acquisition through to grading being completely automated. While there has been significant interest in developing diabetic retinopathy grading AIs [30], few have been trained to specifically grade diabetic maculopathy as a separate entity [26,27,28,29], this despite diabetic maculopathy being the commonest reason for Ophthalmology referral [31]. The performance of the retinopathy classifier was better than the maculopathy classifier, with most false positives being a result of over grading maculopathy. Grading maculopathy is more challenging than grading retinopathy [32]; firstly, exudate is used as a surrogate marker for oedema, and secondly there are several mimics of exudate, such as drusen, pachydrusen, focal ERM, that are difficult to discern without OCT. To address this issue screening programs in the UK and New Zealand have now started to incorporate OCT into their screening pathways. However, as most DR screening programs still operate an asynchronous model of care, and small hard drusen and focal reflective ERM are easily over looked at the time the patient attends for screening, these pathologies are often not identified until the retinal images are reviewed after the screening event. One advantage of using an AI such as THEIA™, which is capable of grading in real time, is that it facilitates the transition to synchronous models of care where patients can be issued their results at the point of care. An additional benefit of this model is that those patients who the AI identifies as having “suspected” maculopathy can be immediately imaged with OCT. This image could be read on the spot if telehealth support is available, or later if not. In either case there is no requirement for the patient to return as all the data required to grade their disease has been acquired.

THEIA™ has been designed primarily as a clinician assist primary triage tool. As such, it has been designed with an ultra-high sensitivity to ensure that sight threatening disease is not missed. In its previous configuration, THEIA™ achieved this at the expense of a modest specificity [5]. With a modification to the algorithm, the current version of THEIA™ preserved its ultra-high sensitivity while achieving a specificity higher than 95%. Whilst there was still a tendency for THEIA™ to over grade the issue of false positives is not overly troublesome because being a primary grading support tool it simply means that borderline images need to be read by a member of the grading team. The trade-off for the tendency to over grade is an ultra-high sensitivity and negative predictive value. Consequently, if THEIA™ grades an image as having no significant disease those responsible for the diabetic eye screening program can be confident that no significant disease has been missed. As most patients undergoing screening have minimal or no disease, we believe that the trade-off between “no disease missed” and a small number of false positives is reasonable. In this trial, four different camera types were used in multiple clinical settings; these included an iCare Eidon camera (confocal scanning laser ophthalmoscopy technology), and a variety of Canon cameras (conventional flash photography technology). THEIA™’s performance was unaffected by the camera type used or the shape and size of the image (Supplementary Tables 8 & 9). It also coped well with a number of artifacts on the real-world images including a central bright halo that was generated by one camera, and a random assortment of dot artefacts that appeared in a consistent location from another camera (Supplementary Fig. 2).

Whilst an accurate Algorithm is clearly important, there are several diverse issues that need to be addressed before AI can be safely incorporated into diabetic eye screening programs. These include but are not limited to equity, consent data privacy and stakeholder acceptance [17]. We have recently explored the attitude of patients undergoing retinal screening to the concept of using AI to read the retinal images acquired at the time of screening [33]. We found that although there is low awareness of clinical AI applications among our participants, most (78%) were receptive towards the implementation of AI in diabetic eye screening. In line with other similar surveys [34] there was a strong preference towards continual involvement of clinicians in the screening process and it is likely there will need to remain an option for those who prefer the service to be delivered manually [33]. These findings suggest that if clinical algorithm’s like THEIA™ are to be acceptable to stakeholders they will need to be deployed as primary grading support tools that augment the clinical teams at the point of care. Although a separate cost analysis of implementing THEIA™ was not part of this project, a team from Singapore have estimated that the adoption of a primary grading AI system, similar to THEIA™, would reduce the costs of delivering their existing DRS program by 20% [35].

The principal limitation of THEIA™ is that it cannot reliably identify other eye diseases that can present at the time of diabetic screening, such as glaucomatous optic neuropathy and age-related macular degeneration. Three patients in the current study had a significant retinal vein occlusion that was flagged up as significant retinopathy. It would also be reasonable to expect that haemorrhagic neovascular macular degeneration to be similarly identified. The Auckland DR screening program systematically records all other pathologies that are detected during routine screening. A recent analysis of this data revealed that only severe hypertensive retinopathy, retinal vein occlusion and macular degeneration are sufficiently important to justify…