Comparison of semi and fully automated artificial intelligence driven softwares and manual system for cephalometric analysis

Abstract

Background

Cephalometric analysis has been used as one of the main tools for orthodontic diagnosis and treatment planning. The analysis can be performed manually on acetate tracing sheets, digitally by manual selection of landmarks or by recently introduced Artificial Intelligence (AI)-driven tools or softwares that automatically detect landmarks and analyze them. The use of AI-driven tools is expected to avoid errors and make it less time consuming with effective evaluation and high reproducibility.

Objective

To conduct intra- and inter-group comparisons of the accuracy and reliability of cephalometric tracing and evaluation done manually and with AI-driven tools that is WebCeph and CephX softwares.

Methods

Digital and manual tracing of lateral cephalometric radiographs of 54 patients was done. 18 cephalometric parameters were assessed on each radiograph by 3 methods, manual method and by using semi (WebCeph) and fully automatic softwares (Ceph X). Each parameter was assessed by two investigators using these three methods. SPSS was then used to assess the differences in values of cephalometric variables between investigators, between softwares, between human investigator means and software means. ICC and paired T test were used for intra-group comparisons while ANOVA and post-hoc were used for inter-group comparisons.

Results

Twelve out of eighteen variables had high intra-group correlation and significant ICC p-values, 5 variables had relatively lower values and only one variable (SNO) had significantly low ICC value. Fifteen out of eighteen variables had minimal detection error using fully-automatic method of cephalometric analysis. Only three variables had lowest detection error using semi-automatic method of cephalometric analysis. Inter-group comparison revealed significant difference between three methods for eight variables; Witts, NLA, SNGoGn, Y-Axis, Jaraback, SNO, MMA and McNamara to Point A.

Conclusion

There is a lack of significant difference among manual, semiautomatic and fully automatic methods of cephalometric tracing and analysis in terms of the variables measured by these methods. The mean detection errors were the highest for manual analysis and lowest for fully automatic method. Hence the fully automatic AI software has the most reproducible and accurate results.

Introduction

The use of lateral cephalogram and cephalometric analysis has been the gold standard for orthodontic diagnosis and treatment planning for decades. It was introduced by Broadbent in 1931 to study skeletal and dental malocclusions and dentofacial discrepancies [1, 2].

The cephalometric analysis can be conducted manually or by computerized methods. The manual method has been used traditionally and is taught and practiced mostly at undergraduate level in dental schools. It had been the method of choice for cephalometric analysis for almost two decades. Whereas, the digital/computerized methods are the recent innovation in dentistry that utilize AI (Artificial Intelligence) to record craniofacial landmarks and calculate desired measurements for various analyses. Due to this rapid advancement in technology and use of AI in digital radiography and other fields of dentistry, the traditional method of cephalometric analysis is being replaced by these latest computerized methods. The AI-based methods of cephalometric analysis can be semi-automatic or fully automatic. The fully-automatic method uses AI to trace, identify landmarks, and calculate the cephalometric measurements. Whereas, the semi-automatic method involves a combination of both methods i.e., manual selection of landmarks followed by automated calculation of values. Nowadays, smartphone applications are also used to facilitate cephalometric analysis [3, 4].

The traditional manual method of cephalometric analysis is prone to systematic and random errors and is also time consuming [5]. It has often been seen that there’s lack of intra-rater and inter-rater consistency and reproducibility. However, the computerized method is easier, quicker, more precise, allows data sharing and is capable of standardized evaluation with higher reproducibility [6, 7]. It is capable of completely eliminating the mechanical errors in drawing lines between landmarks and by-hand measurements with a protractor [8]. Moreover, it avoids the hassle of preparing and storing hardcopies of radiographs and radiographic analysis and provides a digital copy to the dental practice and the patients. However, there may be some disadvantages of these computerized softwares which include errors due to factors such as the quality of the radiograph, magnification errors, and due to adjustment of density, contrast and image quality. Moreover, most of these softwares are relatively expensive [9, 10].

These softwares are becoming increasingly popular due to so many merits which outweigh the minor demerits. Based on literature review, the evidence regarding the accuracy and reliability of one method of cephalometric analysis over others is inconclusive. A number of studies have shown that there’s no significant difference in accuracy between manual method and AI-based softwares for cephalometric analysis [2, 6, 11, 12]. Al-Nasseri reported angular measurements in cephalometrics to be comparable for two methods but of low clinical importance [11]. Farooq et al. assessed 30 variables from five analyses (Steiners, Witts, Tweeds, McNamara, Rakosi Jarabacks) in their study on 50 patients. They did not find any statistically significant difference but reported at least five parameters to have varying results between two methods [2].

However, some studies [2, 3, 10, 13] found statistically significant differences between manual and AI-based methods. Hwang et al., (2020) reported AI to be more accurate than manual method for 14 out of 46 landmarks measured in their study, while another 14 variables were found to be more accurately measured by manual method as compared to AI-based method. The results for these 28 (14 + 14) variables were statistically significant while remaining 18 out of 46 did not show statistically significant differences between AI and manual methods [13]. Another recent study used a fully automated algorithm and concluded 23 cephalometric landmarks to be identified quickly with high accuracy [14].

Considering these varied findings, the primary aim of this study was to compare the accuracy of cephalometric analysis conducted using AI-driven tools and conventional method of manual tracing. The secondary aim was to compare the reproducibility and reliability of the three methods i.e. the manual and the AI-driven softwares (WebCeph [15] and CephX [16]) used for cephalometric analysis.

Methods

This was a cross-sectional, comparative, and quantitative study approved by the Ethics Research Committee of Armed Forces Institute of Dentistry. The study started with the recruitment of participants in December 2021 and continued until mid-May, 2022. The study involved 54 participants presenting to the Orthodontics Department of AFID. The details about the study were explained to the patients and informed consent was taken regarding use of their lateral cephalometric radiographs for the study with anonymity ensured.

The sample included both male and female patients seeking orthodontic treatment. The inclusion criteria included patients reporting to the department for treatment of malocclusion with fixed braces (as lateral ceph is advised to all patients undergoing orthodontic treatment for initial diagnostic work up), having no history of cleft lip or palate or any other deformities. The criteria for recruitment excluded those who had any history of craniofacial deformity and/or transverse discrepancy or signs and symptoms of temporomandibular joint (TMJ) disorders.

The lateral cephalograms of 54 participants were selected for the study. All the radiographs were taken by the same machine (Sirona Dental Systems GmbH^®). The standard protocol for recording lateral cephalometric radiographs was followed whereby the patients were positioned in the cephalostat with the sagittal plane perpendicular to the path of the X-rays, the Frankfort plane parallel to the floor, natural head position with unstrained lips and teeth in centric occlusion i.e. with teeth in maximum intercuspation by asking the patients to bite on their back teeth. After that, each radiograph was traced and analyzed for various cephalometric parameters by both manual and automatic methods. The operational definitions for these methods are as follows:

• Manual method: It is the traditional method of direct measurement of linear and angular cephalometric parameters using a ruler and protractor. The procedure involves using an overlay of acetate tracing paper secured over the radiograph followed by tracing and identification of the landmarks. Finally, the lines are drawn with pencil and ruler and measurements are recorded with the ruler and a protractor.

• Automatic Method: It is the most recent technique where AI-driven softwares are used for cephalometric analysis. After importing the digital radiograph(s) into the computer or smartphone application, the AI-driven softwares trace the radiograph, identify and mark the craniofacial landmarks and complete the analysis by linear and angular measurements of cephalometric parameters. Examples of such softwares are QuickCeph and CephX.

• Semi-Automatic Method: This is a combination of manual and automatic methods. In this procedure, identification of landmarks and magnification correction can be done manually by the operator after importing the radiograph in the software. Whereas, the tracing and calculation of linear and angular measurements of cephalometric parameters is done by the software/AI. For example, Dolphin imaging, WebCeph, OneCeph, Ceph Ninja application. In current study, we used WebCeph and only performed magnification correction manually while tracing, landmark identification and cephalometric measurements were all calculated by the software. The magnification correction was done using 10-point rule i.e. based on the length of the cephalostat rod, that was 10 mm, hanging from the cepalostat machine and recorded on the cephalogram.

Each lateral cephalometric radiograph was traced and analyzed manually by two investigators. For the AI based analysis the cephalograms were imported to the softwares (WebCeph and CephX) for analysis by two investigators. Both the investigators in each group did the analysis within a period of two weeks. Eighteen commonly used cephalometric parameters which are used in making a diagnosis of skeletal, dental and soft tissue relationships or proportions were used (Figs. 1 and 2).

Fig. 1

Landmarks and points

1. SNA 2. SNB 3. Wits (Distance between A perpendicular and B perpendicular on occlusal plane) 4. MMA 5. ISN 6. IMPA 7. II 8. E line upper lip 9. E line lower lip 10. NLA 11. IPP

Fig. 2

Landmarks and points

12. SN-Mand 13. Y-axis 14. SNP 15. SNO 16. McNamara to point A 17. McNamara to Pogonion

18. PFH (Posterior facial height) 19. AFH (Anterior facial height) [Jaraback ratio is PFH/AFH]

Statistical analysis

The sample size calculation was done using the PS software version 3.1.6 that assumed that the difference in the response of matched pairs is normally distributed with mean difference and standard deviation of 2 each. The sample size was calculated to be minimum 30 for paired analysis and 54 for independent analysis. The mean values of the 18 variables calculated by the three methods of cephalometric analysis were compared. The data was found to be normal using Shapiro-Wilk tests hence parametric tests were applied. One way ANOVA (Analysis of variance) and post-hoc was used to compare the groups. The degree of agreement between examiners for each method of cephalometric analysis or intra-group comparisons was conducted using intraclass correlation (ICC; two-way mixed effect model, absolute agreement at 95% confidence interval) and paired-sample t-test. An ICC of 0.7 is generally acceptable and above 0.9 is considered excellent. Both tests (ICC and Paired T-Test) were conducted to compare two investigators with each other for all three methods of cephalometric analyses i.e., manual, as well as two different AI-driven softwares (WebCeph and CephX). The type I error of 5% was allowed and the statistical analyses were performed by the computer program SPSS version 26.0 [17] and Microsoft Excel.

Results

The mean age of the sample was 16.7 ± 1.6 years and the median age was 15.5 years. The sample comprised of 39% males(n = 21) and 61% females(n = 33). The mean and standard deviation of each variable has been presented in Table 1 by each method of analysis.

Table 1 Descriptives (mean, standard deviations and standard error of mean)

The intra-examiner reliability of each ‘method of analysis’ was assessed using intra-class correlation and paired sample T-test by comparing the two examiners (1 and 2) performing these analyses (Table 2 ). The coefficients for ICC showed high correlation between them for all three (manual, semi-automatic, and fully-automatic) cephalometric analysis for 12 out of 18 variables as indicated in Table 2. These variables included SNA (Sella-Nasion plane to point A angle), SNB (Sella-Nasion plane to point B angle), IMPA(Lower incisor to mandibular plane angle), Inter-Incisal angle(II), E-line to upper and lower lip, SN-Go-Gn (Sella Nasion-Gonion-Gnathion angle), Y-Axis, Jaraback, SNP (Sella-Nasion-Palatal plane angle), MMA (Maxillo-Mandibular plane angle), and IPP (Upper Incisor to Palatal plane angle) However, the ICC coefficient for five variables had relatively lower values, including manual analysis of Witts (0.597), NLA (Nasolabial angle)(0.635) and McNamara to Point A (0.578), semi-automatic analysis of Upper Incisal Inclination (0.629), and both manual and semi-automatic analyses of McNamara to Pogonion (0.503 and 0.621, respectively). The ICC coefficient values for other methods of analyses for these 5 variables were high. The lowest ICC correlation was recorded for semi-automatic analysis of SNO (0.293). Similarly, the p-values for the ICC analyses were highly statistically significant (p ≤ 0.001) for all the variables except semi-automatic cephalometric analysis of SNO (p = 0.108). All these aforementioned results depict high correlation of the two examiners performing all the analyses.

Table 2 Intra-group comparison using ICC and paired T-test

The paired sample t-test compared the difference in mean values (of the 18 variables) to test the variability between the two examiners performing all analyses (Table 2). An insignificant p-value (p > 0.05) in this case supports our results of high correlation and minimal variability between the two examiners. The results revealed statistically insignificant p-value for all three analyses of ten variables including SNB, Witts, IMPA, IPP, II angle, SNGoGn, Y-Axis, Jaraback, SNP and MMA. However, a significant difference in mean values of following eight analyses were recorded; manual analyses of E-line to upper lip distance, NLA, SNO, IPP, McNamara to Point A, and McNamara to Pogonion and semi-automatic analysis of SNA. The remaining analyses for these 8 variables also showed statistically insignificant results. These results also support our following results of highest accuracy of fully-automatic without a single discrepancy recorded in ICC and paired-sample T-test.

After the intra-group comparisons, the three methods of analyses were compared for inter-group differences for each variable. Table 3 shows the comparison of mean detection errors in each group and the one with lowest error was considered most reliable, consistent, and accurate method for analyses. More than 80% variables i.e., 15 out of 18 variables had minimal detection error using fully-automatic method of cephalometric analysis. Only three variables had lowest detection error using semi-automatic method of cephalometric analysis, i.e., Y-Axis, SNP, and MMA. The p-values for Y-Axis and MMA were statistically significant. (Table 3)

Table 3 Inter-group comparison of accuracy of manual vs. semi-automatic vs. fully-automatic using ANOVA

The difference in mean values of variables assessed manually, by semi-automatic and fully-automatic methods were analyzed using ANOVA and tested against p < 0.05. The results for ANOVA showed significant difference between three methods for eight variables; Witts, NLA, SNGoGn, Y-Axis, Jaraback, SNO, MMA and McNamara to Point A (Table 3). Therefore, post-hoc analyses were performed to study these differences. The results for post-hoc analyses revealed following:

• Mean value of Witts analysis to be significantly higher by Semi-automatic method of analysis than by Manual method.

• Mean values of NLA, SNGoGn, and MMA to be significantly greater by Fully-automatic method than both Manual and Semi-automatic methods.

• Mean values of Y-Axis and Jaraback to be significantly higher using Manual and Semi-automatic methods of analyses as compared to their values by Fully-automatic method.

• Mean value of SNO to be significantly greater by Manual method than by using Semi- and Fully-automatic methods.

• Mean value of McNamara to Point A to be significantly greater by Semi- and Fully-automatic methods than by manual method.

Discussion

The study provided a detailed comparative analysis of eighteen cephalometric parameters obtained by manual and AI based methods. Each method was performed by two investigators for the assessment of intra-group reliability i.e., author 1’s manual analysis vs. author 2’s manual analysis, author 1’s semi-automatic analysis vs. author 2’s semi-automatic analysis and author 1’s fully-automatic analysis vs. author 2’s fully-automatic analysis.

The intra-group comparisons were conducted using ICC and paired T-test and the goal was to obtain high ICC values with significant p-values and insignificant p-values for paired T-test to reconfirm lack of significant difference between examiners. Both the desired results were proof of high intra-group reliability. Our results (Table 2) showed that both examiners had high correlation for majority of variables, irrespective of method of analysis. 12 out of 18 variables had high intra-group correlation and significant ICC p-values, 5 variables had relatively lower ICC values and only one variable (SNO) had significantly low ICC value. The ICC revealed that the inter-examiner reliability was excellent for fully-automatic method of analysis i.e., 0.919 to 1.000. For semi-automatic method, it ranged from 0.621 to 0.985 with majority values above 0.8. Only one ICC value for semi-automatic method was very low (i.e., 0.293) for SNO, which could be due to an error. Finally, for manual method, the values ranged from 0.503 to 0.953. Therefore, we failed to reject the null hypothesis since there was no significant difference in the cephalometric analysis performed by 2 examiners using manual, semi-automatic WebCeph and fully-automatic CephX softwares. A similar study compared WebCeph and AutoCAD at different time intervals by a single examiner and reported both programs to have adequate reliability [4].

After the intra-group comparisons to assess the reliability of each method, the next aim of the study was to find the most accurate/valid method with minimal detection error by comparing the three methods of analyses. The results showed that 15 out of 18 variables had minimal detection error using fully-automatic method of cephalometric analysis. Remaining three variables had lowest detection error using semi-automatic method of cephalometric analysis. Hence, fully-automatic AI software CephX had the most reliable and accurate results, followed by semi-automatic WebCeph software.

Yassir et al [4] compared WebCeph (both semi and fully automated options) and AutoCAD (semi-automated) and reported adequate reproducibility with both. Their results were in agreement with our study as they found no significant difference between two programs using Paired T-Test or ICC. Another study [18] comparing the reliability of the android smartphone-based app OneCeph (semi-automated) with computer cephalometric tracing program Dolphin Imaging software (semi-automated) reported only four out of 15 parameters showing significant differences. One more study [19] comparing diagnostic accuracy of two smartphone applications (CephNinja and OneCeph, both semi automated) with Viewbox (semi automated) software for cephalometric analysis reported OneCeph to have high validity as compared Viewbox while the reliability of CephNinja was comparable to Viewbox. Hence, they also concluded the performance of these digital softwares to be satisfactory with high potential to replace manual methods in future.

While comparing our results of manual versus computerized, our study found significant differences for several variables (Witts, NLA, SNGoGn, Y-Axis, Jaraback, SNO, MMA and McNamara to Point A). The study of Farooq et al., [2] reported consistency between the two methods for majority of the variables except 1-NA, Y-axis and interincisal angle. Hence, the two studies had agreement regarding difference in manual vs. computerized for one variable i.e., Y-Axis only. Both studies had higher mean value of Y-Axis recorded by manual method as compared to computerized methods. Nasseri [11] found no significant difference in any measurements while comparing manual with digital, however he concluded angular measurements to be more comparable between manual and computerized methods of analyses. Albarakati et al. [3] reported that both conventional and digital cephalometric analysis, although showed some statistically significant differences, most differences were not clinically significant and therefore both are highly reliable. Erkan et al., however, reported no statistically significant difference was found between manual and digital programs [6].

Finally, one study similar to ours by Sommer et al. [20] compared the three methods as well but their findings were different from ours. The fully-automatic method for cephalometric analysis was found not to be reliable. In summary, reviewing results of majority of the studies, we conclude that even when the differences were statistically significant, they were clinically acceptable. Thus, this study indicates that all three methods provide quality results and hence the computerized (AI-based) methods are quite reliable, validating most studies that have compared different cephalometric tracing methods.

Our study also compared the inter-examiner reliability of these methods which has not been discussed in the previous studies. This shows that since the AI based methods of cephalometric analysis are quite accurate therefore they are helpful in saving the clinicians time and effort in analyzing the patients lateral cephalograms and making a quick diagnosis, even by the chairside. In modern world, time is a significant factor and tracing the radiographs, marking landmarks and doing all the calculations by hand could be very tiring and time consuming. Moreover the extra materials used for tracing like acetate tracing sheets and for their storage are also not required. These softwares are now becoming indispensable in the digital orthodontic workflow in orthodontic clinic setups.

The study strengths included adequate sample size, multiple appropriate statistical tests and examiners/operators with experience in orthodontics. Moreover, unlike most studies which compared only two i.e., either manual vs. digital or two digital applications or softwares, our study provided comparison of all three methods of analyses. In addition, the study had two examiners for each method of analysis with high ICC. Even though majority of similar studies had one examiner to avoid bias, in our study we included two examiners to assess difference in fully-automatic and semi-automatic methods when performed by two different examiners. The study wanted to assess the likelihood of differences in values when two different examiners conducted them.

Subdivision of the sample according to vertical and sagittal skeletal patterns could have further verified accuracy detection under different skeletal patterns as the morphology of the condyle also varies in varying skeletal patterns as described by Antonino Lo Giudice et al. [21]. This may be done in future studies. Cephalometric accuracy of 3D structures using AI segmentation tools may also be performed in the future as shown in recent studies on fully automatic segmentation of the mandible and upper airway using convoluted neural networks [22, 23].

Conclusion

There is a lack of significant difference in the majority of variables among the manual, semi automatic and fully automatic methods of cephalometric tracing and analysis. The mean detection errors were the highest for manual analysis and lowest for fully automatic method. Hence the fully automatic AI software has the most reproducible and accurate results. Moreover, these computerized cephalometric softwares are freely available, easy to use, efficient, and can help reduce human error.