Comparison analysis of AIbased smartphone applications for selfexamination of skin cancer risk

Stepan S. Korotkiy; Короткий Степан Сергеевич; Olga A. Saltykova; Салтыкова Ольга Александровна; Andrey O. Ukharov; Ухаров Андрей Олегович; Irena L. Shlivko; Шливко Ирена Леонидовна; Irina A. Klemenova; Клеменова Ирина Александровна; Oxana E. Garanina; Гаранина Оксана Евгеньевна; Kseniia A. Uskova; Ускова Ксения Александровна; Anna M. Myronycheva; Миронычева Анна Михайловна; Yana L. Stepanova; Степанова Яна Леонидовна

doi:10.22363/2312-8143-2023-24-3-262-270

Comparison analysis of AIbased smartphone applications for selfexamination of skin cancer risk

Authors: Korotkiy S.S.¹, Saltykova O.A.¹, Ukharov A.O.², Shlivko I.L.³, Klemenova I.A.³, Garanina O.E.³, Uskova K.A.³, Myronycheva A.M.³, Stepanova Y.L.³
Affiliations:
1. RUDN University
2. Moscow State Technical University n.a. Bauman
3. Privolzhsky Research Medical University
Issue: Vol 24, No 3 (2023)
Pages: 262-270
Section: Articles
URL: https://journals.rudn.ru/engineering-researches/article/view/37069
DOI: https://doi.org/10.22363/2312-8143-2023-24-3-262-270
EDN: https://elibrary.ru/TNDZIX

Cite item

Full Text

Abstract
Full Text
About the authors
References
Statistics

Abstract

A comparative analysis of 3 AI-based smartphone applications for self-service skin cancer risk assessment: ProRodinki, Skinive and Skin Vision. Analysis consists of description of applications and its ways of work, and results, such as sensitivity and specificity, done on the base of the practical experiment conducted with processing 516 images of the skin neoplasms and pathologies confirmed by histological research via each app. Every application is unique and differs from each other by its principles or work, algorithms, user experience and design, and of course AI model and the set of input data that is analyzed by neural networks. Current research and practical experiment were made with focus on images processing and the app risk assessment for each of the image, other details and mole prescription information were set neutral. This leads to a conclusion that there is a lack of methodology for testing and analysis of different AI-based applications and services. Having such methodology, the comparison analysis results can be more objective and transparent.

Keywords

Artificial Intelligence, AI, Neural network, Smartphone application, Skin cancer, Skin self-examination, ProRodinki, SkinVision, Skinive, Risk assessment

Full Text

Introduction According to [1-3], skin cancer prevalence rate is increasing globally, outpacing almost all other types of cancer. There are the following main types of skin cancer: Basal cell carcinoma (BCC) with 80 % share of total skin cancers, Squamous cell carcinoma (SCC) with about 19 % share and Melanoma, which accounts for approximately 1 % of total skin cancers. The most dangerous and aggressive one is melanoma, which causes a majority of skin cancer deaths [4]. Early detection and treatment significantly increase the chances for recovery and do require less financial investments for therapy compared to the case where detection was done on the latest stages [5]. It is a challenge to detect the skin cancer on early stages due to the fact that it usually starts to progress with a change in a skin and absence of any other noticeable symptoms [6]. Screening method using AI-based smartphone applications (“apps”) for risk assessment of skin cancer seems to have a great potential to provide earlier and more accurate guidance on a particular skin lesion or change [7-10]. The skin selfexamination can be conducted at home and doesn’t require special conditions. It’s affordable, convenient and instant way to get an immediate recommendation on next steps and decrease the risks of skin cancer. However, it’s crucial to remember that apps cannot be an alternative to dermatologists and oncologists. AI technology and apps aim to help and give an advice, but not to replace the professional doctors. There is a number of apps using machine learning algorithms to provide self-service skin cancer risk assessment. And this number is demonstrating double digit growth: between 2014 and 2017, more than 200 new apps in dermatology field were developed and available for download [11]. This trend is becoming even stronger, considering the increasing use of apps and AI technology [12]. Widespread usage of these apps must be supported by a robust evidence base [13; 14], which is the main objective of this work - to compare smartphone applications main principles of work and conduct the practical study in skin cancer detection and risk assessment for each of the app. Comparative analysis of three AI-based smartphone applications was conducted: ProRodinki, Skinive and SkinVision. The research consists of user experience overview and practical study to evaluate the level of quality of risk assessment. 1. Materials and methods It was decided to conduct this comparative analysis with the following applications: the leading skin cancer risk assessment application in Russian Federation - ProRodinki, the leading one in Europe - SkinVision, and Skinive - the application available in both, Russia and Europe. Highlevel overview on ProRodinki, Skinive and SkinVision solutions, country origin and monetization model are listed in Table 1 below. Comparison analysis study is conducted using ProRodinki app, Skinive app for home use and SkinVision app. All other solutions/products are out of scope of this work. Dataset for comparison study consists of 516 images of the skin neoplasms and pathologies confirmed by histological research. None of the dataset images are known to be a part of training dataset for any neural network scoped for current research. There are 328 (~64%) benign skin neoplasms and 188 (~36%) malignant cases. Benign group of images is represented by Nevus, Hemangioma and Seborrheic keratosis (SK). Malignant group consists of BCC, SCC and Melanoma images. Beside the description of screening flows and apps itself, the confusion matrix for binary classification [15] is used for each application study result. There are two class labels - malignant (positive) and benign (negative). The confusion matrix is listed in Figure 1. Based on the confusion matrix results (True Positive - TP, False Negative - FN, False Positive - FP, True Negative - TN), the sensitivity and specificity metrics are calculated respectively. Sensitivity means “out of all actual malignant neoplasms, how many malignant ones were predicted”, which is calculated using the following formula: = /( + ) Specificity means “out of all actual benign cases, how many benign moles were predicted”. This metric is calculated by the following formula: = /( + ) 2. Results Comparative analysis results are listed below - it contains the description of each app and its screening flows, study practical steps description and results. Table 1 Projects overview - products, monetization models Project Country origin Website Products / Solutions Free / Paid ProRodinki Russia https://www.prorodinki.ru/ 1. App Free, paid expert advice Skinive Belarus, Netherlands https://skinive.ru/ https://skinive.com/ 1. App for home use 2. App for clinicians 3. API for integration Paid subscription, Free trial 10 screenings SkinVision Netherlands https://www.skinvision.com/ 1. App No free trial, paid single check or subscription plan Вестник РУДН. Серия: Инженерные исследования. 2023. Т. 24. № 3. С. 262270 2.1. ProRodinki app Application developers are AIMED, LLC (Russian Federation, Moscow, 121205, Skolkovo Innovation Center, Noble st., 7, floor 2, sec. 58 RB 2) and Privolzhsky Research Medical University (Russian Federation, Nizhny Novgorod, 603005, 10/1 Minin and Pozharsky Square). The app was released in 2020, available for Russian Federation. Product localization is made in Russian and English languages. Can be installed on smartphones and tablets with iOS or Android. The application is updating approximately once a month, last version update made on 20th of February 2023. Roszdravnadzor of Russian Federation marking (№ KP-20-006 dated from 14th of April 2020) was applied to allow application usage as non-medical including the recommendation to doctors. ProRodinki website[16]: supervision.” the combination of artificial intelligence technology and professional doctors’ supervision, that provides a recommendation for further action and not make a diagnosis. Successful screening user experience flow is presented in Figure 2 and contains 6 steps. Every skin self-assessment is happening with mole localization and prescription data specification (Steps 1-3). The mole image can be taken using smartphone camera or uploaded from the gallery (Step 4). App checks the image quality and highlights the detected moles in case if the image quality is appropriate (Step 5). And on final stage (Step 6) user can see the neural network recommendation and the option to purchase a control opinion of real doctor. It is important to mention that mention that the results of neural network are monitored by real doctors and ProRodinki app can send an additional follow-up with updated recommendation later on. Figure 3. Skinive screening flow 2.2. Skinive app Application developers are Skinive Holding B.V. (The Netherlands, Amsterdam) and Wise AI, LLC (Belarus, Minsk). Company Skinive was founded in 2020. The app was released in October 2022, available worldwide except US and Canada. Product localization is made in 8 languages: Russian, English, French, German, Hindi, Bangla, Portuguese and Spanish. Can be installed to smartphones with iOS or Android. The application is updating once a month, last version update made on 22nd of March 2023. The following information is presented on Skinive website regarding certification[17]: “Skinive solutions are CE marked and meet all regulatory requirements for medical device class. Skinive is a Class 1 medical device that uses clinically proven technology. Moreover, Skinive company has an ISO13485 Certificate, which confirms compliance with the requirements of regulators for the quality management system of medical device manufac turers.” The following introduction is presented on Skinive website: “Skinive is a convenient way to detect and assess skin disease risks with your smartphone! Simply take a photo of the skin pathology and send it to the Skinive app for timely analysis. Skinive checkup: - Increases the detection rate of skin diseases with instant screening; - Fewer referrals to specialists thanks to primary screening at home; - Reduce the cost of identifying skin conditions when not necessary; - Reduces the cost of treatment by detecting diseases at earlier stages; - Skinive also lets you store photos to track changes over time, helping you closely monitor your health over the long term.” Noticeable that Skinive provides several smartphone applications - app for home use and app for medical professionals. The app for medical professionals is out of scope of current work and mainly different from the app for home use with capabilities of patient management (ability to create patients with its skin profiles and screen moles for the particular patient). Figure 4. Take mole using Skinive AI camera capability Successful screening user experience flow is presented in Figure 3 and contains 5 steps. Every skin check requires a mole localization (Steps 1-2). The mole image can be taken using smartphone camera or be uploaded from the gallery (Step 3). Taking the mole via camera can be done with AI mode on, Вестник РУДН. Серия: Инженерные исследования. 2023. Т. 24. № 3. С. 262-270 which allows to take a mole only if Skinive app detects the mole area highlighting it with a red square. 2 examples of app screens are shown in the Figure 4 to demonstrate the AI mode for Skinive mole detection. Steps 4 and 5 in Figure 3 are showing the recommendation with risk assessment level and top-3 diseases with formation classes and sign percentage for each disease. No additional follow-up recommendations or updates are received from Skinive app after check was done. 2.3. SkinVision app Application developer is SkinVision B.V. (Barbara Strozzilaan 201, NL-1083 HN Amsterdam, Netherlands). Company SkinVision was founded in 2011. The app was released in 2014, available worldwide. Product localization is made in 7 languages: English, Dutch, French, German, Italian, Polish, Spanish. Can Can be installed on smartphones with iOS or Android. The application is updating every 1-3 months, last version update made on 27th of February 2023. The following information is presented on SkinVision website regarding certification[18]: “We are a regulated medical device with European CE marking. We are ISO certified for medical device and information security management.” The following introduction is presented on SkinVision website4: “SkinVision is a regulated medical service that takes control over your skin health to a new level. It expands your ability to selfexamine your skin and elevates your knowledge when to act, how and why. It is designed to provide accurate and timely skin cancer detection, along with the most reliable personalised skin health advice and health path recommendation. At the centre of the service is the SkinVision app, which is a regulated medical device that merges AI technology with the expertise of skin health professionals and dermatologists. SkinVision is a service of choice whether you want to address your most immediate concerns, learn what steps you should take next, understand your skin risk profile and introduce the most intelligent skin health regime to your seasonal rhythm.” Successful screening user experience flow is presented in Figure 5 and contains 6 steps. Every skin self-assessment starts with mole localization (Steps 1-2). The mole image can be taken using smartphone camera only (Step 3). There is no option to upload the mole image from smartphone gallery and the mole detection is processed via smartphone camera considering 3 main requirements: mole to be detected, clear and in focus. The progress bar for mole detection is shown as a blue circle in the center of app screen and take mole event is happening automatically when detection is done. After the confirmation of mole image (Step 4) the app asks for prescription details (Step 5) and then AI analysis begin (Step 5.1). Final stage (Step 6) is the neural network recommendation with risk assessment. It is important to mention that the results of neural network are monitored by real doctors and SkinVision app can send an additional follow-up with updated recommendation later on. 2.4. Study Every dataset image was processed by each app: ProRodinki (versions for iOS: 5.2.1, 5.2.3), Skinive for home use (version for Android 1.0.13 and versions for iOS: 1.0.6, 1.0.7) and SkinVision (version for iOS: 6.20.1). Total number of test cases is 1 548. To run the test, the upload image from smart phone gallery capability was used for ProRodinki and Skinive apps. Meanwhile, the SkinVision app was used to take a photo of the neoplasm image showed on the display with 13.3In size, 2560x1600 resolution and 4 096 000 pixels. Due to the fact that Interpretation of results with risk assessment is vary for different apps, the results were aligned to the common binary format - test cases classified by Skinive as low and medium risk levels are considered to be equal to low risk assessment made by ProRodinki and SkinVision. High risk results didn’t require any additional manipulations for alignment. Each app has its own algorithms of image quality validation and detection. According to current study, not all dataset images were processed successfully and there is a number of unprocessed test cases listed in Table 2. The rest of images were processed successfully. The experiment results with calculated sensitivity and specificity metrics are provided in Table 3. Listed results reflect the instant AI-based smartphone app recommendations only and do not consider the follow-ups received later after a professional doctor check. Table 2 Failed screenings split by apps UNRECOGNIZED ProRodinki Skinive SkinVision Malignant 8 23 18 Benign 5 19 18 Total 13 42 36 Table 3 Study results TP 161 99 161 FP 19 22 170 TN 295 287 139 FN 28 66 9 Sensitivity 85.19% 60.00% 94.71% Specificity 93.95% 92.88% 44.98% 3. Discussion The highest sensitivity was demonstrated by SkinVision application - almost 95%. Meanwhile, the specificity metric has the lowest result - lower than 45 %. The assumption can be made that this configuration of SkinVision AI is made in the way to minimize the risk of missing malignant neoplasm and minimize the potential legal risks related to the wrong recommendation. It is important to emphasize that the study was made by taking the photos from the display, which might bring the certain distortion to the results. Low rate in specificity metric is mitigated well via follow-up by professional doctors. Skinive app provides the most informative results and recommendations - according to the information published on their website, there are 51 skin pathologies that can be recognized. It is definitely a challenge to train the neural network for such a significant skin disease nomenclature and presumably this can be a reason for high number of False Negatives equal to 66 and the sensitivity metric on the level of 60%. At the same time, the specificity rate is almost 93%. Skinive develops the app for doctors and such an informative way of results interpretation does seem to be very promising in combination with competence and experience doctor. ProRodinki AI-based smartphone application has demonstrated the best balance of sensitivity and specificity rates - 85 % and almost 94 % respectively. In common with SkinVision, there is a tight connection between AI technology and team of professional doctors who supervise and monitor the recommendations made by app. Conclusions The research around AI-based smartphone applications available for skin cancer risk assessment has been conducted and 3 applications were chosen for comparison analysis: ProRodinki, Skinive and SkinVision. Applications’ ways of work and user experience flows for self-service screening scenario were described and the practical experiment measuring the sensitivity and specificity using dataset validated by histological researches was made. Every application is unique and differs from each other by its principles of work, algorithms, user experience and design, and of course AI model and the set of input data that is analyzed by neural networks. Current research and practical experiment Вестник РУДН. Серия: Инженерные исследования. 2023. Т. 24. № 3. С. 262-270 were made with focus on images processing and the app risk assessment for each of the image, other details and mole prescriptions information were set neutral. This leads to a conclusion that there is a lack of methodology for testing and analysis of different AI-based applications and services. Having such methodology, the comparison analyses results can be more objective and transparent. It is important to compare and test different AIbased apps specialized on skin cancer risk assessment to measure their quality and efficiency, to monitor and manage the progress of AI technologies and, as a result, to use it as help for doctors and health organizations.

About the authors

Stepan S. Korotkiy

RUDN University

Author for correspondence.
Email: skorotkiy@gmail.com
ORCID iD: 0009-0004-4613-970X

Graduate student, Department of Mechanics and Control Processes, Academy of Engineering

Moscow, Russian Federation

Olga A. Saltykova

RUDN University

Email: saltykova-oa@rudn.ru
ORCID iD: 0000-0002-3880-6662

Ph.D. of Physico-Mathematical Sciences, Associate Professor of the Department of Mechanics and Control Processes, Academy of Engineering

Moscow, Russian Federation

Andrey O. Ukharov

Moscow State Technical University n.a. Bauman

Email: oukharov@gmail.com
ORCID iD: 0000-0003-3490-3657

Ph.D. of Technical Sciences, Researcher

Moscow, Russian Federation