2.1.

Overview

The research flowchart detailed in this paper is shown in Figure 1. The main tasks included nail detection, nail lesion detection, quadrant division and scoring. We use the constructed data set to train models of nail detection, lesion detection and quadrant classification. We use the nail detection model to locate the nail and cut the nail image and use the lesion detection model to obtain the location and type of the lesion. NAPSI and Target NAPSI scores are obtained by combining the results of quadrant division and scoring rules.

Figure 1.

A flowchart of the nail psoriasis smart system.

2.2.

Nail detection

The severity of nail psoriasis is mainly evaluated by the patients’ nail disease. Therefore, the model focuses on the nail area. The actual shooting photos take only a small part of the images of the hands and feet. The nails are so small, and the nail lesions are smaller. If the original image is directly trained by the neural network model. After multiple down sampling, lesion information may be lost, and it is difficult for the model to learn features. Therefore, we designed a nail detection and localization model.

The flowchart of nail detection is shown in Figure 2. We use Faster-RCNN¹⁹ as the nail detection model. In this model, the pre-trained ResNet-50 network is used to extract features and generate feature maps. The feature map is input into Region Proposal Network (RPN) to predict the regional proposals. The proposals and feature maps are input into the ROI-Pooling to obtain the proposal feature map. Finally, the proposal feature map is used to calculate the proposal category, and the final exact position of the detection box is obtained through rebound box regression.

Figure 2.

The flowchart of nail detection.

2.3.

Nail lesion detection

After the nails are detected, the nail images are cut according to bounding box. And then use the nail images to detect lesion. For lesion detection, we trained two models and compared Faster-RCNN and YOLO v5. YOLO v5 has the advantages of good performance of small target detection and fast inference. The YOLO v5 model is composed consist of Head, Backbone, Neck and Prediction using the mosaic mode in data augmentation. The four images were flipped, zoomed and color gamut changed, and spliced together after random clipping²⁰. It enhanced the robustness of the model and improved the efficiency of model training.

2.4.

Quarters division and scoring algorithm

In NAPSI and target NAPSI, each nail is divided into four quadrants. Therefore, after the nails and lesions are detected, the nails need to be divided into four quarters. Due to the different shooting postures, the nails will have various angles. We divide one quarter of the nail into 8 categories. According to nail angle, the nails are divided into 8 categories. Each category corresponds to a vertical line at a specified angle, from 0 to 180 degrees, and one category is 22.5°. The cropped nail images are divided into 8 classes, and the ratio of the training set to the rest set is 7: 3. This dataset is used to train the classification network. ResNet-50²¹ is used as the classification model.

According to the previous neural network model, we got nail lesion and the quadrant division. NAPSI and target NAPSI can be calculated using geometric methods according to the corresponding rules. The algorithm is as follows:

The main differences between NAPSI and Target NAPSI are the types and number of lesions. There are 2 types of NAPSI lesions, and 8 types of Target NAPSI. The potential total score of NAPSI is 160, while the Target NAPSl’s is 640.

3.1.

Datasets

The nail psoriasis dataset is made up of real patient images. These images are collected from the Dermatology Clinic, West China Hospital of Sichuan University. All images are taken in a specific way under the specific background. All photos were taken on smartphones or tablet with a high-resolution camera. The image resolution is not less than 2K * 3K. The nail shooting method is shown in Figure 3. Bend your fingers toward your palm, so that most of the nails face the camera and are focused. The background is blue to reduce the impact of the background. All collected images are annotated by professional dermatologists. Nail annotations and lesion annotations are annotated with bounding box on the collected images. The eight types of lesions are annotated, including onycholysis, splinter hemorrhages, hyperkeratosis, oil-drop dyschromia, pitting, leukonychia, crumbling, and red spots in the lunula. The labelling of lesion will be marked, reviewed and modified by different doctors to ensure the accuracy of labelling. Finally, we obtained and labelled 705 images.

Figure 3.

Shooting method of hands and feet.

We collected photos of hands and feet from normal people and annotated nails. Images of normal subjects and psoriasis patients constituted the nail localization dataset. The collected 875 images including 170 from normal people and 705 from psoriasis patients. As we all know, deep learning networks require large amounts of data for training. Our original images are not enough for model training. Therefore, through cutting, translating, rotating, adding noise, mirroring and cropping for enhancement, we finally got 7000 images. We cut out part of nail images from psoriasis patients’ images to form a lesion detection dataset. The nail images without lesions were deleted from the data set, and 3061 images were obtained. We divide the data set into training set, validation set and test set according to 6:2:2 for model training and testing.

3.2.

Metrics

Use two detection models and one classification model, and use the corresponding metrics to evaluate their performance. The accuracy (AC) is used to evaluate classification models, while the mean Average Precision (mAP) is used to analyze detection models. The metrics are calculated as follows:

In equations (1)-(4), TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives. The larger metrics will indicate the better performance.

3.3.

Experiments and results

Train and test the nail detection model, lesion detection model and quadrant classification model on the China Mobile Jiutian artificial intelligence platform. It provides Intel(R) Xeon(R) Gold 5118 2.30GHz CPU, TeslaV100 GPU and 500GB SDD. The primary software packages were Python 3.7, CUDA 10.1, pytorch 1.3.1, torchvision 0.4.2.

The input size of the training models for nail detection and quarters classifications are 800*600 and 224*224. For nail lesion detection, 512*512 is used.

The model results on the test set are shown in Table 1. Among them, the mAP of nails detection reached 0.909, the quadrant classification accuracy is 0.765. For nail lesion detection, YOLO v5 mAP is 0.24, while the Faster-RCNN is 0.206.

Table 1.

The results of the models on test sets.

3.4.

Intelligent assessment application use process

Based on the above algorithm, we have developed an intelligent assessment system for nail psoriasis. The intelligent assessment system is an application of H5. As shown in Figure 4a, the user should first enter name and medical card number. Then take photos of the left hand, right hand, left foot and right foot according to the instructions, shown in Figure 4b. Click to start evaluation, the system will upload images and call the nail detection model, the lesion detection model and the nail evaluation algorithm. Finally, return and display the results. The displayed information includes a scoring interface and a visual interface of detection results. The interface between NAPSI and Target NAPSI is shown in Figure 4c. Detection results visualization are shown in Figure 4d.

Figure 4.

The user interaction with the intelligent assessment application of nail psoriasis (in Chinese).