Smartphone applications (apps) that claim to detect skin cancer at an early stage, some through tracking moles, have rates of false negatives and false positives that mean they could be harmful by leading people to delay seeking advice, researchers have warned* in the BMJ. They argued that the current regulatory process for such apps is inadequate – and experts have called** for a cultural shift so that it becomes the norm for developers to support third party access to data to facilitate competition and error correction.
A research team led from the University of Birmingham conducted a systematic review of nine diagnostic accuracy studies to examine the validity and findings of six algorithm-based smartphone apps aimed at assessing the risk of skin cancer in suspicious skin lesions.
They reported that six of the studies verified results by using histology or follow-up (725 lesions), and three verified results by using expert recommendations (407 lesions). But they found that the studies were small and of poor methodological quality, with selective recruitment, high rates of unevaluable images, and differential verification. They noted that lesion selection and image acquisition were performed by clinicians rather than smartphone users.
They also explained that there are two CE (Conformit Europenne)- marked apps available for download. One, skinScan, was evaluated in a single study of only 15 lesions (five melanomas) with 0% sensitivity and 100% specificity for the detection of melanoma. The other, SkinVision was evaluated in two studies (252 lesions, including 61 malignant or premalignant lesions) and achieved a sensitivity of 80% and a specificity of 78% for the detection of malignant or premalignant lesions; the app only correctly picked up one of three melanomas as high risk. The authors noted: “When the interpretation of SkinVision was varied to consider high and moderate results from the app as test positive, sensitivity increased (between 17 and 57 percentage points) but at a considerable cost to specificity (falling by 11-45 percentage points).”
They commented: “Despite the limitations of the evidence base, two algorithm-based apps have obtained the CE marking and are currently being marketed with claims that they can ‘detect skin cancer at an early stage’ or ‘track moles over time with the aim of catching melanoma at an earlier stage of the disease’. Under the EU Medical Device Directive smartphone apps are class 1 devices. Manufacturers can apply CE marking to class 1 devices as long as they have shown compliance with the ‘essential requirements’ as outlined in the Directive, and without necessarily being subject to independent inspection by notified bodies such as the Medicines and Healthcare products Regulatory Agency in the United Kingdom. Under the new Medical Device Regulations, which come into full force by May 2020, smartphone apps could be in higher device classes and will be subject to inspection by notified bodies.”
They concluded: “Current algorithm-based smartphone apps cannot be relied on to detect all cases of melanoma or other skin cancers. Test performance is likely to be poorer than reported here when used in clinically relevant populations and by the intended users of the apps. The current regulatory process for awarding the CE marking for algorithm-based apps does not provide adequate protection to the public.”
The authors of the accompanying editorial said the current regimen, which allows both these apps to be approved and regulated as ‘class I medical devices’ and have a CE mark, which gives consumers the impression that they have been assessed as effective and safe, “is clearly unsatisfactory”. They said: “Collectively as a society we must decide what amounts to good evidence when evaluating health apps; who is responsible for generating, validating, and appraising this evidence; and how post-market monitoring of regularly updated software should be organised…
“Regulators most accustomed to managing medicines will need new skills to evaluate digital technology. But wherever the perimeter is drawn, they must avoid false reassurance: when regulators are not evaluating technology, they should clearly flag this to patients and policy makers.”
*Freeman K, Dinnes J, Chuchu N, et al. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. BMJ 2020; 368: m127
**Morley J, Floridi L, Goldacre B. Editorial: The poor performance of apps assessing skin cancer risk. BMJ 2020; 368: m428 doi: 10.1136/bmj.m428