brain tumor research paper pdf

• Search Menu
• Advance Articles
• Special Issues
• Author Guidelines
• Submission Site
• Open Access
• Reviewer Guidelines
• Review and Appeals Process
• About The Computer Journal
• About the BCS, The Chartered Institute for IT
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

• 1. INTRODUCTION
• 2. BRAIN TUMOUR
• 3. MAGNETIC RESONANCE IMAGING (MRI)
• 5. BRAIN TUMOUR DETECTION MECHANISM
• 6. DISCUSSION AND ANALYSIS
• 7. CONCLUSION
• 8. FUTURE SCOPE
• ETHICS APPROVAL
• DATA AVAILABILITY
• < Previous

A Comprehensive Review of Brain Tumour Detection Mechanisms

• Article contents
• Figures & tables
• Supplementary Data

Praveen Kumar Ramtekkar, Anjana Pandey, Mahesh Kumar Pawar, A Comprehensive Review of Brain Tumour Detection Mechanisms, The Computer Journal , Volume 67, Issue 3, March 2024, Pages 1126–1152, https://doi.org/10.1093/comjnl/bxad047

• Permissions Icon Permissions

The brain is regarded as the central part of the human body and has a very complicated structure. The abnormal growth of tissue inside the brain is called a brain tumour. Tumour detection at an early stage is the most difficult task in the discipline of health. In this review article, the authors have deeply analysed and reviewed the brain tumour detection mechanisms which include manual, semi- and fully automated techniques. Today, fully automated mechanisms apply deep learning (DL) methods for tumour detection in brain magnetic resonance images (MRIs). This paper deals with previously published research articles relevant to various brain tumour detection techniques. Review of various types of tumours, MRI modalities, datasets, filters, segmentation methods and DL techniques like long short-term memory, gated recurrent unit network, convolution neural network, auto encoder, deep belief network, recurrent neural network, generative adverse network and deep stacking networks have been included in this paper. It has been observed from the analysis that the use of DL techniques in the detection of brain tumours improves accuracy. Finally, this paper reveals research gaps, limitations of existing methods, challenges in tumour detection and contributions of the proposed article.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1460-2067
• Print ISSN 0010-4620
• Copyright © 2024 British Computer Society
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Brain tumor detection and classification using machine learning: a comprehensive survey

• Original Article
• Open access
• Published: 08 November 2021
• Volume 8 , pages 3161–3183, ( 2022 )

Cite this article

You have full access to this open access article

• Javaria Amin   ORCID: orcid.org/0000-0003-1080-5446 1 , 2 ,
• Muhammad Sharif 2 ,
• Anandakumar Haldorai 3 ,
• Mussarat Yasmin 2 &
• Ramesh Sundar Nayak 4  

48k Accesses

116 Citations

Explore all metrics

Brain tumor occurs owing to uncontrolled and rapid growth of cells. If not treated at an initial phase, it may lead to death. Despite many significant efforts and promising outcomes in this domain, accurate segmentation and classification remain a challenging task. A major challenge for brain tumor detection arises from the variations in tumor location, shape, and size. The objective of this survey is to deliver a comprehensive literature on brain tumor detection through magnetic resonance imaging to help the researchers. This survey covered the anatomy of brain tumors, publicly available datasets, enhancement techniques, segmentation, feature extraction, classification, and deep learning, transfer learning and quantum machine learning for brain tumors analysis. Finally, this survey provides all important literature for the detection of brain tumors with their advantages, limitations, developments, and future trends.

Similar content being viewed by others

A Comprehensive Survey of Machine Learning Techniques for Brain Tumor Detection

A Comprehensive Review of Deep Learning Techniques for Brain Tumor Prediction

A systematic analysis of magnetic resonance images and deep learning methods used for diagnosis of brain tumor

Shubhangi Solanki, Uday Pratap Singh, … Sanjeev Jain

Avoid common mistakes on your manuscript.

Introduction

The central nervous system disseminates sensory information and its corresponding actions throughout the body [ 1 , 2 , 3 ]. The brain, along with the spinal cord, assists in this dissemination. The brain’s anatomy [ 4 ] contains three main parts; brain stem, cerebrum, and cerebellum. The weight of a normal human brain is approximately 1.2–1.4 K, with a volume of 1260 cm 3 (male brain) and 1130 cm 3 (female brain) [ 5 ]. The frontal lobe of brain assists in problem-solving, motor control, and judgments. The parietal lobe manages body position. The temporal lobe controls memory and hearing functions, and occipital lobe supervises the brain’s visual processing activities. The outer part of cerebrum is known as cerebral cortex, and is a greyish material; it is composed of cortical neurons [ 6 ]. The cerebellum is relatively smaller than the cerebrum. It is responsible for motor control, i.e., systematic regulation of voluntary movements in living organisms with a nervous system. Due to variable size and stroke territory, ALI, lesionGnb, and LINDA methods fail to detect the small lesion region. Cerebellum is well-structured and well-developed in human beings as compared to other species [ 7 ]. The cerebellum has three lobes; an anterior, a posterior, and a flocculonodular. A round-shaped structure named vermis connects the anterior and posterior lobes. The cerebellum consists of an inner area of white matter (WM) and an outer greyish cortex, which is a bit thinner than that of the cerebrum. The anterior and posterior lobes assist in the coordination of complex motor movements. The flocculonodular lobe maintains the body’s balance [ 4 , 8 ]. The brain stem, as the name states, is a 7–10 cm-long stem-like structure. It contains cranial and peripheral nerve bundles and assists in eye movements and regulations, balance and maintenance, and some essential activities such as breathing. The nerve tracks originating from the cerebrum’s thalamus pass through the brain stem to reach the spinal cord. From there, they spread throughout the body. The main parts of the brain stem are midbrain, pons, and medulla. The midbrain assists in functions such as motor, auditory, and visual processing, as well as eye movements. The pons assists in breathing, intra-brain communication, and sensations, and medulla oblongata helps in blood regulation, swallowing, sneezing, etc. [ 9 ].

Brain tumor and stroke lesions

Brain tumors are graded as slow-growing or aggressive [ 2 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ]. A benign (slow-growing) tumor does not invade the neighboring tissues; in contrast, a malignant (aggressive) tumor propagates itself from an initial site to a secondary site [ 16 , 17 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ]. According to WHO, a brain tumor is categorized into grades I–IV. Grades I and II tumors are considered as slow-growing, whereas grades III and IV tumors are more aggressive, and have a poorer prognosis [ 28 ]. In this regard, the detail of brain tumor grades is as follows.

Grade I : These tumors grow slowly and do not spread rapidly. These are associated with better odds for long-term survival and can be removed almost completely by surgery. An example of such a tumor is grade 1 pilocyticastrocytoma.

Grade II : These tumors also grow slowly but can spread to neighboring tissues and become higher grade tumors. These tumors can even come back after surgery. Oligodendroglioma is a case of such a tumor.

Grade III : These tumors develop at a faster rate than grade II, and can invade the neighboring tissues. Surgery alone is insufficient for such tumors, and post-surgical radiotherapy or chemotherapy is recommended. An example of such a tumor is anaplastic astrocytoma.

Grade IV : These tumors are the most aggressive and are highly spreadable. They may even use blood vessels for rapid growth. Glioblastoma multiforme is such a type of tumor [ 29 ].

Ischemic stroke : Ischemic stroke is an aggressive disease of brain and it is major cause of disability and death around the globe [ 30 ]. An ischemic stroke occurs when the blood supply to the brain is cut off, resulting underperfusion (in tissue hypoxia) and dead the advanced tissues in hours [ 31 ]. Based on the severity, stroke lesions are categories into different stages such as acute (0–24 h), sub-acute (24 h–2 weeks) and chronic (> 2 weeks) [ 32 ].

• Brain imaging modalities

Three major methods (PET, CT, DWI and MRI) for brain tumors are widely used to analyze the brain structure.

Positron emission tomography

Positron emission tomography (PET) uses a special type of radioactive tracers. Metabolic brain tumor features such as blood flow, glucose metabolism, lipid synthesis, oxygen consumption, and amino acid metabolism are analyzed through PET. It is still considered as one of the most powerful metabolic techniques and utilizes the best nuclear medicine named as fluorodeoxyglucose (FDG) [ 33 ]. FDG is a widely used PET tracer in brain images. Nevertheless, FDG-PET images have limitations, e.g., an inability to differentiate between necrosis radiation and a recurrent high-grade (HG) tumor [ 34 ]. Moreover, during a PET scan, radioactive tracers can cause harmful effects to the human body, causing a post-scan allergic reaction. Some patients are allergic to aspartame and iodine. In addition, PET tracers do not provide accurate localization of anatomical structure, because they have a relatively poor spatial resolution as compared to an MRI scan [ 35 ].

Computed tomography

Computed tomography (CT) images provide more in-depth information than images obtained from normal X-rays. The CT scan has received widespread recommendation and adoption since its inception. A study [ 36 ] determined that in the USA alone, the annual CT scan rate is 62 million, with 4 million for children. CT scans show soft tissues, blood vessels, and bones of different human body parts. It uses more radiation than normal X-rays. This radiation may increase the risk of cancers when multiple CT scans are performed. The associated risks of cancers have been quantified according to CT radiation doses [ 37 , 38 ]. MRI can even help in evaluating structures obscured in a CT scan, and provides high contrast among the soft tissues, providing a clearer anatomical structure [ 39 ].

Magnetic resonance imaging

An MRI scan is used to completely analyze different bodyparts, and it also helps to detect abnormalities in the brain at earlier stages than other imaging modalities [ 40 ]. Hence, complex brain structures make tumor segmentation a challenging task [ 41 , 42 , 43 , 44 , 45 , 46 , 47 ]. This review discusses preprocessing approaches, segmentation techniques [ 48 , 49 ], feature extraction and reduction methods, classification methods, and deep learning approaches. Finally, benchmark datasets and performance measures are presented.

Diffusion weighting imaging

MRI sequences are utilized to analyze the stroke lesions based on the several parameters such as age, location and extent regions [ 50 ]. In the context of treatment, a computerized method might be utilized for accurate diagnosis of the disease progression rate [ 51 ]. The neuroscientists of cognitive, who frequently conduct research in which cerebral impairments are linked to cognitive function They observed that segmentation of the stroke lesions is a vital task to analyze the total infected region of brain that provide help in the treatment process [ 52 ]. However, segmentation of the stroke lesions is a difficult task, because stroke appearance is change as the passage of time. The MRI sequence such as diffusion weighted imaging (DWI) and FLAIR are utilized for stroke lesions detection. In acute stoke stage DWI sequence highlight the infection part as a hyperintensity. The underperfusion region represents the mapping magnitude of the perfusion [ 53 ]. The dis-similarity among two regions might be considered as penumbra tissue. Stroke lesions appear in distinct locations and shapes. Different types of lesions are appeared in a variable size and shape and these lesions are not aligned with vascular patterns and more than one lesions might appeared on similar time. The size of the stroke lesions is in radii of the few millimeters and appears in a full hemisphere. The structure of the hemisphere is dissimilar, and its intensity might significantly vary within the infected region. Furthermore, automated stroke segmentation is difficult due to the similar appearance of the pathology such as white matter hyperintensities and chronic stroke lesions [ 54 ].

Evaluation and validation

In the existing literature, experimental results are evaluated on publicly available datasets to verify the robustness of algorithms.

Publicly available datasets

Several datasets are publicly available that are used by the researchers to evaluate the proposed methods. Some important and challenging datasets are discussed in this section. BRATS are the most challenging MRI datasets [ 55 , 56 , 57 ]. BRATS Challenge is published in different years with more challenges having 1 mm 3 voxels resolution. The detail of datasets is given in Fig.  1 as well as in Table 1 .

Datasets for brain tumor detection

Performance metrics

The performance measures play a significant role to compute the method’s effectiveness. A list of performance metrics is provided in Fig.  2 .

List of performance measures for evaluation of brain tumor

Preprocessing

Preprocessing is a critical task [ 61 ] to extract the requisite region. 2D brain extraction algorithm (BEA) [ 62 ], FMRIB software library [ 63 ], and BSE [ 64 ] are used for non-brain tissue removal as shown in Fig.  3 . The bias field is a key problem that arises in MRI due to imperfections of radio frequency coil called intensity inhomogeneity [ 65 , 66 ]. It is corrected as shown in Fig.  4 [ 67 ]. The preprocessing methods like linear, nonlinear [ 68 ], fixed, multi-scale, and pixel-based are used in distinct circumstances [ 69 , 70 , 71 , 72 ]. The small variations among normal and abnormal tissues due to noise [ 68 ] and artifacts often provide difficulty in direct image analysis [ 73 , 74 ]. AFINITI is used for brain tumor segmentation [ 63 ]. Consequently, automated techniques are adopted in which computer software performs segmentation and eliminates the need for manual human interaction [ 75 , 76 ]. Fully and semi-automated techniques are used widely [ 77 , 78 ]. The results of brain tumor segmentation are mentioned in Table 2 . The segmentation methods are divided into the following categories.

Conventional methods.

Machine learning methods.

Different inhomogeneities related to MRI noise have shading artifacts and partial volume effects.

Skull removal a input, b skull removed [ 1 ]

Bias field correction a input, b estimated, c corrected [ 67 ]

When different types of tissues [ 61 ] take the same pixel, then it is called partial volume effect [ 92 ]. The random noise related to MRI [ 19 , 93 , 94 ] has Rician distribution [ 95 ]. In the literature, different filters such as wavelet, anisotropic diffusion, and adaptive are presented to enhance edges [ 96 ]. An anisotropic diffusion filter is more suitable in practical applications due to low computational speed [ 97 , 98 ]. When the noise level is high in the image, it is difficult to recover the edges [ 99 ]. Normalizing the image intensity is another part of the preprocessing phase [ 2 , 100 , 101 ] and modified curvature diffusion equation (MCDE) [ 102 ] are applied for intensity normalization. Wiener filter is used to enhance the local and spatial information in medical imaging [ 103 ]. The widely utilized preprocessing methods are N4ITK [ 104 ] for the correction of bias field, median filter [ 104 ] for image smoothing, anisotropic diffusion filter [ 105 ], image registration [ 106 ], sharpening [ 107 ], and skull stripping through brain extraction tool (BET) [ 108 ].

Conventional methods

The conventional methods [ 46 ] are further categorized into the following:

Thresholding methods.

Region growing methods.

Watershed methods.

• Segmentation

Segmentation extracts the required region from input images. Thus, segmenting accurate lesion regions is a more crucial task [ 109 ]. As manual segmentation process is erroneous [ 110 ]; therefore, semi- and fully automated methods are utilized [ 46 ]. Segmentation of tumor region using semi-automated methods achieves acceptable outcomes over manual segmentation [ 111 , 112 ]. Semi-automated methods are further divided into three forms: initialization, evaluation, and feedback response [ 113 , 114 ].

Thresholding methods

The thresholding method is a basic and powerful method to segment the required objects [ 18 ] and the selection of an optimized threshold is a difficult task in low-contrast images. Histogram analysis is used to select threshold values based on image intensity [ 115 ]. Thresholding methods are classified into local and global. If high homogeneous contrast or intensity exists among the objects and background, then the global thresholding method is the best option for segmentation. The optimal threshold value can be determined by Gaussian distribution method [ 116 ]. These methods are utilized when the threshold value cannot be measured from the whole image histogram or single value of the threshold does not provide good results of segmentation [ 117 ]. In most cases, the thresholding method is applied at the first stage for segmentation and many distinct regions are segmented within the gray-level images as shown in Fig.  5 .

Segmentation using Otsu thresholding a original images, b Otsu thresholding [ 82 ]

Region growing (RG) methods

In RG approaches, image pixels form disjoint areas are analyzed through neighboring pixels, which are merged with homogeneousness characteristics based on pre-defined similitude criteria. The region growing might fail to provide better accuracy due to the partial volume effect [ 118 , 119 ]. To overcome this effect, MRGM is preferred [ 86 , 120 ]. The region growing with BA methods is also introduced [ 87 ].

Watershed methods

As MR images have more proteinaceous fluid intensity, therefore, watershed methods are utilized to analyze the intensity of the image [ 114 , 121 , 122 ]. Due to noise [ 123 ], watershed method leads to over-segmentation [ 124 ]. The accurate segmentation [ 125 ] results can be obtained by the combination of watershed transform with the merging of statistical methods [ 126 , 127 ]. Some watershed algorithms are topological watershed [ 128 ], image foresting transform (IFT) watershed [ 129 ], and marker-based watershed [ 130 ].

The comprehensive literature review [ 131 ] on brain tumor detection shows that there is room for improvement [ 72 ]. As a brain tumor appears in variable sizes and shapes, existing segmentation approaches require additional improvements for tumor segmentation. In overcoming the limitations of existing methods, enhancement [ 132 , 133 , 134 ] and segmentation [ 135 , 136 , 137 ] have significance in tumor detection.

Feature extraction methods

The feature extraction approaches [ 12 , 138 , 139 , 140 ] including GLCM [ 15 , 141 , 142 ], geometrical features (area, perimeter, and circularity) [ 15 ], first-order statistical (FOS), GWT [ 143 , 144 ], Hu moment invariants (HMI) [ 145 ], multifractal features [ 146 ], 3D Haralick features [ 147 ], LBP [ 148 ], GWT [ 11 ], HOG [ 14 , 137 ], texture and shape [ 82 , 143 , 149 , 150 ], co-occurrence matrix, gradient, run-length matrix [ 151 ], SFTA, curvature features [ 152 , 153 ], Gabor like multi-scale texton features [ 154 ], Gabor wavelet and statistical features [ 142 , 143 ] are utilized for classification. Table 3 lists the summary of feature extraction methods.

Feature selection methods or feature selection/reduction methods

In machine learning and computer vision applications, high-dimensional features maximize the system execution time and memory requirement for processing. Therefore, to distinguish between relevant and non-relevant features, several feature selection methods are required to minimize redundant information [ 168 ]. The optimal feature extraction is still a challenging task [ 47 ]. The single-point heuristic search method, ILS, genetic algorithm (GA) [ 169 ], GA+ fuzzy rough set [ 170 ], hybrid wrapper-filter [ 171 ], TRSFFQR, tolerance rough set (TRS), firefly algorithm (FA) [ 172 ], minimum redundancy maximum relevance (mRMR) [ 152 ], Kullback–Leibler divergence measure [ 173 ], iterative sparse representation [ 174 ], recursive feature elimination (RFE) [ 175 ], CSO-SIFT [ 176 ], entropy [ 11 , 177 , 178 ], PCA [ 179 ], and LDA [ 180 ] are utilized to remove redundant features. A summary of classification methods as shown in Table 4 .

Classification methods

The classification approaches are used to categorize input data into different classes in which training and testing are performed on known and unknown samples [ 16 , 24 , 25 , 181 , 182 , 183 , 184 , 185 , 186 , 187 , 188 , 189 , 190 , 191 , 192 ]. Machine learning is widely used for tumor classification into appropriate classes, e.g., tumor substructure (complete/non-enhanced/enhanced) [ 193 ], tumor and non-tumor [ 26 ], and benign and malignant tumor [ 15 , 47 , 163 , 194 , 195 ]. KNN [ 196 ], SVM, nearest subspace classifier, and representation classifier [ 143 ] are supervised, whereas FCM [ 197 , 198 ], hidden Markova random field [ 199 ] self-organization map [ 101 ], and SSAE [ 200 ] are unsupervised methods.

Recent trends in medical imaging to detect malignancy

Deep learning and quantum machine learning methodologies are widely utilized for tumor localization and classification [ 201 ]. In these techniques, automatic feature learning helps to discriminate complicated patterns [ 186 , 202 , 203 , 204 , 205 , 206 , 207 , 208 , 209 , 210 , 211 , 212 , 213 ].

Deep learning methods

The variety of state of the art deep learning methodologies are used to learn the data in the medical domain [ 214 ] including CNN [ 215 , 216 ], Deep CNN, cascaded CNN [ 217 ], 3D-CNN [ 218 ], convolutional encoder network, LSTM, CRF [ 218 ], U-Net CNN [ 219 ], dual-force CNN [ 220 ] and WRN-PPNet [ 221 ].

The brain tumor classification problem has been solved by employing a LSTM model. In this method, input MRI images smooth using N4ITK and 5 × 5 Gaussian filter and passed as input to the four LSTM model. The LSTM model is constructed on the four hidden Units such as 200, 225, 200, 225, respectively. The performance of this model has been tested on BRATS (2012–2015 and 2018) series and SISS-2015 benchmark datasets [ 222 ]. In this work, a new framework is presented based on the fusion of different kinds of MRI sequences. The fused sequence provides more information as compared to single sequence. Later, fused sequence has been supplied to the 23 CNN model. The suggested model is trained on brat’s series for the detection of glioma [ 16 ]. The 14 layers CNN model has been trained from the scratch on six Brats series datasets for detection of glioma and stroke lesions [ 25 ]. The classification is performed using ELM and RELM classifiers. This method has been tested on BRATS series such as 2012 to 2015 [ 189 ]. The 09-layer CNN model is trained from the scratch for classification of different types of tumors such as pituitary, glioma and meningioma. The method achieved an accuracy of the classification is 98.71% [ 223 ]. This model is trained from the scratch on publicly 696 weighted-T1 sequences. The model provides an accuracy of greater than 99% for tumor classification [ 224 ]. The existing methods are summarized in Table 5 .

Although much work is done on deep learning methods, still there exist many challenges. The present methods do not achieve maximum results in the sub-structure of the tumor region. For example, if the accuracy of the complete tumor is increased, then the accuracy of the core and the enhanced tumor is decreased (as shown in Table 5 ).

Brain tumor detection using transfer learning

The manual detection of brain tumors is difficult due to asymmetrical lesions shape, location flexibility, and unclear boundaries. Therefore, a transfer-learning model has been suggested based on the super-pixel. The VGG-19 is a pre-trained model that has been utilized for the classification of the different grades of the glioma such as high/low glioma. The method achieved 0.99 AUC on the brats 2019 series[ 232 ]. The three different types of pre-trained models i.e., VGG network, Google network and Alex network are employed on the brain datasets for the classification of glioma, pituitary and meningioma. In this method, augmentation methods are also employed on MRI slices to generalize the outcomes and reduced the overfitting problem by increasing the quantity of the input data. After the experimental analysis using different pre-trained models, we conclude that VGG-16 provides greater than 98% classification accuracy [ 233 ]. The classification of brain tumors has been done using two different types of networks, i.e., visual attention network and CNN are utilized for classification of different types of brain tumor i.e., glioma, pituitary I and meningioma [ 234 ]. A pre-trained model i.e., VGG-16, Alex and Google net are investigated for the analysis of brain tumors. The frequency domain techniques have been applied on input slices to improve the image contrast. The contrast improved images are passed in the next phase. Where pre-trained VGG-16 provides maximum classification outcomes [ 235 ]. The Laplacian filter with a multi-layered dictionary model is utilized for the recognition of brain tumors. The model performed better as compared to existing works [ 236 ]. The method consists of the three major steps such as pre-processing, augmentation of data, and segmentation and classification using transfer learning models. In which ResNet-50, DenseNet-201, MobileNet-v2 and Inceptionv3 are utilized to classify the brain lesions with 0.95 IoU [ 237 ]. The deep features are extracted from the transfer learning AlexNet model. The model has eight layers, five of which are convolutional and three of which are fully linked. The SoftMax layer has been employed for classification between the different types of brain lesions [ 238 ]. The transfer learning ResNet-50 model with average global pooling is utilized to reduce the gradient vanishing and overfitting issues. The performance of this model has been evaluated on three distinct types of brain imaging benchmark samples that contain 3064 input images. The method achieved an accuracy of the 97.08% that is maximum as compared to latest existing works [ 239 ]. A deep CNN was used in this study that based on transfer learning such as ResNet, Xception and Mobilenetv2 are utilized for the extraction of deep features has been for tumors classification using MRI images. This method achieved an accuracy of up to 98% [ 240 ]. In this method, Grab Cut method has been employed for segmentation of the brain lesions. Later hand-crafted such as LBP features dimension of 1 × 20 and HOG features dimension of 1 × 100 are extracted and serially fused to the deep features dimension of 1 × 1000 that are extracted from the pre-trained VGG-19 model and final fused features vector length of 1 × that is supplied to the different kind of classifiers. The experimental analysis proves that fused features vector provide good results as compared to existing work in this domain [ 16 , 187 ]. The global thresholding method is applied to segment the actual lesion region. After segmentation, texture features such as LBP and GWF are extracted from the segmented images. After that, the retrieved features are fused to form a single fused feature vector, which is then provided to the classifiers for differentiation between healthy and unhealthy images [ 26 ]. There are two key stages to the procedure. The brain lesions are enhanced and segmented using spatial domain approaches in the first stage, then deep information’s are extracted using pre-trained models, i.e., Alex and Google-network and score vector is achieved from softmax layer that is supplied to the classifiers such as for discrimination between the glioma/non-glioma images of brain. The Brats series dataset was used to test this technique’s efficiency [ 241 ]. For brain tumor segmentation, the superpixel approach has been suggested. From the segmented images, Gabor wavelet information are retrieved and given to SVM and CRF for discrimination between the healthy/un-healthy MRI images [ 242 ].The transfer learning models such as inceptionv3, densenet-201, and to form a single vector, extracted features are merged serially and passed to softmax for tumor classification. Furthermore, different dense blocks of the densenet201 are extracted and classify the brain tumor using softmax. The approach had a 99% accuracy rate. The evaluation outcomes clearly state that the fused vector outperformed as compared to the single vector [ 243 ]. A novel U-net model with the RESnet model has been trained on the input MRI images. The classifiers are fed the salient features derived from its pictures. This method has been tested on BRATS 2017, 2018 and 2019 datasets [ 244 ]. The tumor region is localized on Flair sequences of brats 2012 series. The skull is removed from of the input pictures, and a noise-reduction filter is applied bilaterally. During the segmentation, texton features are recovered from the input images using the superpixel approach. For brain tumor classification, the leave out validation technique is used. This strategy yielded an 88 percent dice score [ 245 ]. The deep segmentation has been designed that contains two major parts such as encoder and decoder. The spatial information is extracted using a CNN in the encoder section. For determining the whole probability map resolution, the semantic mappings information is entered into the decoder component. On the basis of U-network distinct CNN networks such as ResNetwork, dense network and Nas-network are utilized for features extraction. This model has been tested successfully on Brats-2019 series. The method achieved dice scores of 0.84 [ 246 ]. The wavelet homomorphic filter has been employed for noise removal. The tumor infected region has been localized using improved YOLOv2 model [ 230 ]. The summary of the transfer learning methods is mentioned in Table 6 .

Brain tumor detection using quantum machine learning

Superposition of quantum states/parallelism/entanglement can all be used to establish quantum computer supremacy [ 258 ]. However, exploring entanglement of quantum features for efficient computation is a difficult undertaking due to a shortage of computational resources for execution of quantum algorithms. With the progress of quantum techniques, classical computers based on quantum theory and influenced through qubits are no longer able to fully exploit the benefits of quantum state and entanglement. QANN has been found to be effective in a variety of computer tasks, including classification and pattern recognition due to the intrinsic properties supplied by quantum physics [ 259 ]. On the other hand, quantum models based on genuine quantum computers use big bits of the quantum/qubits as a simple representation of matrix and the linear functions. However, the computational complexity of the quantum-inspired neural network (QINN) designs increases several fold due to complicated and time-consuming back-propagation quantum model [ 260 ]. The automatic segmentation of brain lesions from I (MRI), which removes the onerous manual work of human specialists or radiologists, greatly aids brain tumor detection. Manually, brain tumor diagnosis, on the other hand, suffers from large variances in size, shape, orientation, illumination variations, greyish overlaying, and cross-heterogeneity. Scientists in the computer vision field have paid a lot of emphasis in recent years to building robust and efficient automated segmentation approaches. The current research focuses on a unique quantum fully supervised learning process which is defined by qutrits for timely and effective lesions segmentation. The proposed work’s main goal is to speed up the QFS-convergence Net’s and make it appropriate for computerized segmentation of the brain lesions without the need for any learning/supervision. To leverage the properties of quantum correlation, suggested a quantum fully self-supervised neural network (QFS-Net) model uses qutrits/three states of quantum for segmentation of the brain lesions [ 261 ]. The QFS-Net uses a revolutionary fully supervised qutrit-based counter propagation method to replace the sophisticated quantum back-propagation method that utilized in supervised QINN networks. This approach allows for iterative quantum state that propagates among the layers of network.

Limitations of existing’s machine/deep learning methods

In this survey, recent literature regarding the detection of brain tumors is reviewed, and it is indicated that there is still room for improvement. During image acquisition, noise is included in MRI, and noise removal is an intricate task [ 2 , 262 , 263 , 264 ]. Accurate segmentation is a difficult task [ 265 ], as brain tumors have tentacles and diffused structures [ 43 , 193 , 220 , 266 ]. Selecting and extracting optimal features and appropriate number of training/testing samples for better classification is also an important task [ 191 , 192 ]. Deep learning models are gaining attention as the learning of features is accomplished automatically; however, they require high computing power and large memory. Therefore, still there is a need to design a lightweight model that provides high ACC in less computational time. Some existing machine learning methods with their limitations are mentioned in Table 7 .

The following are the main challenges of brain tumor detection.

The glioma and stroke tumors are not well contrasted. It consists of tentacle and diffused structures that make segmentation and classification processes more challenging [ 270 ].

A small volume of tumor detection is still a challenge as it can be detected as a normal region [ 269 , 273 ].

Some of the existing methods work well for only a complete tumor region and do not provide good results for other regions (enhanced, non-enhanced) and vice versa [ 267 , 271 , 274 ].

Research findings and discussion

After a comprehensive review of the state-of-the-art exiting methods, the following challenges are found:

The size of a brain tumor grows rapidly. Therefore, tumor diagnosis at an initial stage is an exigent task.

Brain tumor segmentation is difficult owing to the following factors.

MRI image owing to magnetic field fluctuations in the coil.

Gliomas are infiltrative, owing to fuzzy borders. Thus, they become more difficult to segment [ 43 ].

Stroke lesion segmentation is a very intricate task, as stroke lesions appear in complex shapes and with ambiguous boundaries and intensity variations.

The optimized and best feature extraction and selection is another difficult process inaccurate classification of brain tumors.

The accurate brain tumor detection is still very demanding because of tumor appearance, variable size, shape, and structure. Although tumor segmentation methods have shown high potential in analyzing and detecting the tumor in MR images, still many improvements are required to accurately segment and classify the tumor region. Existing work has limitations and challenges for identifying substructures of tumor region and classification of healthy and unhealthy images.

In short, this survey covers all important aspects and latest work done so far with their limitations and challenges. It will be helpful for the researchers to develop an understanding of doing new research in a short time and correct direction.

The deep learning methods have contributed significantly but still require a generic technique. These methods provided better results when training and testing are performed on similar acquisition characteristics (intensity range and resolution); however, a slight variation in the training and testing images directly affects the robustness of the methods. In future work, research can be conducted to detect brain tumors more accurately, using real patient data from any medium (different image acquisition (scanners). Handcrafted and deep features can be fused to improve the classification results. Similarly, lightweight methods such as quantum machine learning play significant role to improve the accuracy and efficacy that save the time of radiologists and increase the survival rate of patients.

Park JG, Lee C (2009) Skull stripping based on region growing for magnetic resonance brain images. Neuroimage 47:1394–1407

Article   Google Scholar  

Khan MA, Lali IU, Rehman A, Ishaq M, Sharif M, Saba T et al (2019) Brain tumor detection and classification: A framework of marker-based watershed algorithm and multilevel priority features selection. Microsc Res Tech 82:909–922

Raza M, Sharif M, Yasmin M, Masood S, Mohsin S (2012) Brain image representation and rendering: a survey. Res J Appl Sci Eng Technol 4:3274–3282

Google Scholar  

Watson C, Kirkcaldie M, Paxinos G (2010) The brain: an introduction to functional neuroanatomy. Academic Press, New York

(2015). https://en.wikipedia.org/wiki/Brain_size . Accessed 19 Oct 2019

Dubin MW (2013) How the brain works. Wiley, New York

Koziol LF, Budding DE, Chidekel D (2012) From movement to thought: executive function, embodied cognition, and the cerebellum. Cerebellum 11:505–525

Knierim J (1997) Neuroscience Online Chapter 5: Cerebellum. The University of Texas Health Science Center, Houston

Nuñez MA, Miranda JCF, de Oliveira E, Rubino PA, Voscoboinik S, Recalde R et al (2019) Brain stem anatomy and surgical approaches. Comprehensive overview of modern surgical approaches to intrinsic brain tumors. Elsevier, Amsterdam, pp 53–105

Chapter   Google Scholar  

DeAngelis LM (2001) Brain tumors. N Engl J Med 344:114–123

Amin J, Sharif M, Raza M, Saba T, Sial R, Shad SA (2020) Brain tumor detection: a long short-term memory (LSTM)-based learning model. Neural Comput Appl 32:15965–15973

Sajjad S, Hanan Abdullah A, Sharif M, Mohsin S (2014) Psychotherapy through video game to target illness related problematic behaviors of children with brain tumor. Curr Med Imaging 10:62–72

Yasmin M, Sharif M, Masood S, Raza M, Mohsin S (2012) Brain image reconstruction: a short survey. World Appl Sci J 19:52–62

Amin J, Sharif M, Raza M, Yasmin M (2018) Detection of brain tumor based on features fusion and machine learning. J Ambient Intell Human Comput:1–17

Amin J, Sharif M, Yasmin M, Fernandes SL (2020) A distinctive approach in brain tumor detection and classification using MRI. Pattern Recogn Lett 139:118–127

Saba T, Mohamed AS, El-Affendi M, Amin J, Sharif M (2020) Brain tumor detection using fusion of hand crafted and deep learning features. Cogn Syst Res 59:221–230

Sharif M, Amin J, Nisar MW, Anjum MA, Muhammad N, Shad SA (2020) A unified patch based method for brain tumor detection using features fusion. Cogn Syst Res 59:273–286

Sharif M, Tanvir U, Munir EU, Khan MA, Yasmin M (2018) Brain tumor segmentation and classification by improved binomial thresholding and multi-features selection. J Ambient Intell Human Comput:1–20

Sharif MI, Li JP, Khan MA, Saleem MA (2020) Active deep neural network features selection for segmentation and recognition of brain tumors using MRI images. Pattern Recogn Lett 129:181–189

Sharif MI, Li JP, Naz J, Rashid I (2020) A comprehensive review on multi-organs tumor detection based on machine learning. Pattern Recogn Lett 131:30–37

Ohgaki H, Kleihues P (2013) The definition of primary and secondary glioblastoma. Clin Cancer Res 19:764–772

Cachia D, Kamiya-Matsuoka C, Mandel JJ, Olar A, Cykowski MD, Armstrong TS et al (2015) Primary and secondary gliosarcomas: clinical, molecular and survival characteristics. J Neurooncol 125:401–410

Amin J, Sharif M, Gul N, Yasmin M, Shad SA (2020) Brain tumor classification based on DWT fusion of MRI sequences using convolutional neural network. Pattern Recogn Lett 129:115–122

Sharif M, Amin J, Raza M, Yasmin M, Satapathy SC (2020) An integrated design of particle swarm optimization (PSO) with fusion of features for detection of brain tumor. Pattern Recogn Lett 129:150–157

Amin J, Sharif M, Anjum MA, Raza M, Bukhari SAC (2020) Convolutional neural network with batch normalization for glioma and stroke lesion detection using MRI. Cogn Syst Res 59:304–311

Amin J, Sharif M, Raza M, Saba T, Anjum MA (2019) Brain tumor detection using statistical and machine learning method. Comput Methods Progr Biomed 177:69–79

Amin J, Sharif M, Gul N, Raza M, Anjum MA, Nisar MW et al (2020) Brain tumor detection by using stacked autoencoders in deep learning. J Med Syst 44:32

Johnson DR, Guerin JB, Giannini C, Morris JM, Eckel LJ, Kaufmann TJ (2017) 2016 updates to the WHO brain tumor classification system: what the radiologist needs to know. Radiographics 37:2164–2180

Wright E, Amankwah EK, Winesett SP, Tuite GF, Jallo G, Carey C et al (2019) Incidentally found brain tumors in the pediatric population: a case series and proposed treatment algorithm. J Neurooncol 141:355–361

Pellegrino MP, Moreira F, Conforto AB (2021) Ischemic stroke. Neurocritical care for neurosurgeons. Springer, New York, pp 517–534

Garrick R, Rotundo E, Chugh SS, Brevik TA (2021) Acute kidney injury in the elderly surgical patient. Emergency general surgery in geriatrics. Springer, New York, pp 205–227

Lehmann ALCF, Alfieri DF, de Araújo MCM, Trevisani ER, Nagao MR, Pesente FS, Gelinski JR, de Freitas LB, Flauzino T, Lehmann MF, Lozovoy MAB (2021) Carotid intima media thickness measurements coupled with stroke severity strongly predict short-term outcome in patients with acute ischemic stroke: a machine learning study. Metab Brain Dis 36:1747–1761

Scott AM (2005) PET imaging in oncology. In: Bailey DL, Townsend DW, Valk PE, Maisey MN (eds) Positron emission tomography. Springer, London, pp 311–325

Wong TZ, van der Westhuizen GJ, Coleman RE (2002) Positron emission tomography imaging of brain tumors. Neuroimaging Clin 12:615–626

Wong KP, Feng D, Meikle SR, Fulham MJ (2002) Segmentation of dynamic PET images using cluster analysis. IEEE Trans Nuclear Sci 49:200–207

Brenner DJ, Hall EJ (2007) Computed tomography—an increasing source of radiation exposure. N Engl J Med 357:2277–2284

Smith-Bindman R, Lipson J, Marcus R, Kim K-P, Mahesh M, Gould R et al (2009) Radiation dose associated with common computed tomography examinations and the associated lifetime attributable risk of cancer. Arch Intern Med 169:2078–2086

Fink JR, Muzi M, Peck M, Krohn KA (2015) Multimodality brain tumor imaging: MR imaging, PET, and PET/MR imaging. J Nucl Med 56:1554–1561

Hess CP, Purcell D (2012) Exploring the brain: Is CT or MRI better for brain imaging. UCSF Dep Radiol Biomed Imaging 11:1–11

Saad NM, Bakar SARSA, Muda AS, Mokji MM (2015) Review of brain lesion detection and classification using neuroimaging analysis techniques. J Teknol 74:1–13

Huang M, Yang W, Wu Y, Jiang J, Chen W, Feng Q (2014) Brain tumor segmentation based on local independent projection-based classification. IEEE Trans Biomed Eng 61:2633–2645

Khan MA, Arshad H, Nisar W, Javed MY, Sharif M (2021) An integrated design of Fuzzy C-means and NCA-based multi-properties feature reduction for brain tumor recognition. Signal and image processing techniques for the development of intelligent healthcare systems. Springer, New York, pp 1–28

Rewari R (2021) Automatic tumor segmentation from MRI scans. Stanford University, Stanford

Tandel GS, Biswas M, Kakde OG, Tiwari A, Suri HS, Turk M et al (2019) a review on a deep learning perspective in brain cancer classification. Cancers 11:1–32

El-Dahshan E-SA, Mohsen HM, Revett K, Salem A-BM (2014) Computer-aided diagnosis of human brain tumor through MRI: A survey and a new algorithm. Expert Syst Appl 41:5526–5545

Gordillo N, Montseny E, Sobrevilla P (2013) State of the art survey on MRI brain tumor segmentation. Magn Reson Imaging 31:1426–1438

Mohan G, Subashini MM (2018) MRI based medical image analysis: Survey on brain tumor grade classification. Biomed Signal Process Control 39:139–161

Amin J, Sharif M, Yasmin M (2016) Segmentation and classification of lung cancer: a review. Immunol Endocr Metab Agents Med Chem 16:82–99

Shahzad A, Sharif M, Raza M, Hussain K (2008) Enhanced watershed image processing segmentation. J Inf Commun Technol 2:9

Joo L, Jung SC, Lee H, Park SY, Kim M, Park JE et al (2021) Stability of MRI radiomic features according to various imaging parameters in fast scanned T2-FLAIR for acute ischemic stroke patients. Sci Rep 11:1–11

Chen H, Zou Q, Wang Q (2021) Clinical manifestations of ultrasonic virtual reality in the diagnosis and treatment of cardiovascular diseases. J Healthc Eng 2021:1–12

Henneghan AM, Van Dyk K, Kaufmann T, Harrison R, Gibbons C, Heijnen C, Kesler SR (2021) Measuring self-reported cancer-related cognitive impairment: recommendations from the Cancer Neuroscience Initiative Working Group. JNCI:1–9

Drake-Pérez M, Boto J, Fitsiori A, Lovblad K, Vargas MI (2018) Clinical applications of diffusion weighted imaging in neuroradiology. Insights Imaging 9:535–547

Okorie CK, Ogbole GI, Owolabi MO, Ogun O, Adeyinka A, Ogunniyi A (2015) Role of diffusion-weighted imaging in acute stroke management using low-field magnetic resonance imaging in resource-limited settings. West Afr J Radiol 22:61

Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J et al (2014) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34:1993–2024

Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS et al (2017) Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data 4:170117

Kistler M, Bonaretti S, Pfahrer M, Niklaus R, Büchler P (2013) The virtual skeleton database: an open access repository for biomedical research and collaboration. J Med Internet Res 15:e245

Summers D (2003) Harvard whole brain Atlas: www. med. harvard. edu/AANLIB/home. html. J Neurol Neurosurg Psychiatry 74:288–288

Armato S, Beichel R, Bidaut L, Clarke L, Croft B, Fenimore C, Gavrielides M et al (2008) RIDER (Reference Database to Evaluate Response) Committee Combined Report, 9/25/2008 Sponsored by NIH, NCI, CIP, ITDB Causes of and Methods for Estimating/Ameliorating variance in the evaluation of tumor change in response-to therapy. https://wiki.cancerimagingarchive.net/display/Public/Collections

Kistler M, Bonaretti S, Pfahrer M, Niklaus R, Büchler P (2013) The virtual skeleton database: an open access repository for biomedical research and collaboration. J Med Internet Res 15:1–14

Yasmin M, Mohsin S, Sharif M, Raza M, Masood S (2012) Brain image analysis: a survey. World Appl Sci J 19:1484–1494

Somasundaram K, Kalaiselvi T (2010) Fully automatic brain extraction algorithm for axial T2-weighted magnetic resonance images. Comput Biol Med 40:811–822

Zhu Y, Young GS, Xue Z, Huang RY, You H, Setayesh K et al (2012) Semi-automatic segmentation software for quantitative clinical brain glioblastoma evaluation. Acad Radiol 19:977–985

Prabhu LAJ, Jayachandran A (2018) Mixture model segmentation system for parasagittal meningioma brain tumor classification based on hybrid feature vector. J Med Syst 42:1–6

Park CR, Kim K, Lee Y (2019) Development of a bias field-based uniformity correction in magnetic resonance imaging with various standard pulse sequences. Optik 178:161–166

Patel P, Bhandari A (2019) A review on image contrast enhancement techniques. Int J Online Sci 5:14–18

Zhang Z, Song J (2019) A robust brain MRI segmentation and bias field correction method integrating local contextual information into a clustering model. Appl Sci 9:1332

Irum I, Sharif M, Yasmin M, Raza M, Azam F (2014) A noise adaptive approach to impulse noise detection and reduction. Nepal J Sci Technol 15:67–76

Robb RA (2000) 3-dimensional visualization in medicine and biology. Handb Med Imaging Process Anal:685–712

Mehmood I, Ejaz N, Sajjad M, Baik SW (2013) Prioritization of brain MRI volumes using medical image perception model and tumor region segmentation. Comput Biol Med 43:1471–1483

Lu X, Huang Z, Yuan Y (2015) MR image super-resolution via manifold regularized sparse learning. Neurocomputing 162:96–104

Irum I, Sharif M, Raza M, Mohsin S (2015) A nonlinear hybrid filter for salt & pepper noise removal from color images. J Appl Res Technol 13:79–85

Stadler A, Schima W, Ba-Ssalamah A, Kettenbach J, Eisenhuber E (2007) Artifacts in body MR imaging: their appearance and how to eliminate them. Eur Radiol 17:1242–1255

Masood S, Sharif M, Masood A, Yasmin M, Raza M (2015) A survey on medical image segmentation. Curr Med Imaging 11:3–14

Irum I, Sharif M, Raza M, Yasmin M (2014) Salt and pepper noise removal filter for 8-bit images based on local and global occurrences of grey levels as selection indicator. Nepal J Sci Technol 15:123–132

Sharif M, Irum I, Yasmin M, Raza M (2017) Salt & pepper noise removal from digital color images based on mathematical morphology and fuzzy decision. Nepal J Sci Technol 18:1–7

Prastawa M, Bullitt E, Moon N, Van Leemput K, Gerig G (2003) Automatic brain tumor segmentation by subject specific modification of atlas priors1. Acad Radiol 10:1341–1348

Wu Y, Yang W, Jiang J, Li S, Feng Q, Chen W (2013) Semi-automatic segmentation of brain tumors using population and individual information. J Digit Imaging 26:786–796

Xie K, Yang J, Zhang Z, Zhu Y (2005) Semi-automated brain tumor and edema segmentation using MRI. Eur J Radiol 56:12–19

Agn M, Puonti O, af Rosenschöld PM, Law I, Van Leemput K (2015) Brain tumor segmentation using a generative model with an RBM prior on tumor shape. In: BrainLes vol 2015, pp 168–180

Haeck T, Maes F, Suetens P (2015) ISLES challenge 2015: Automated model-based segmentation of ischemic stroke in MR images. BrainLes 2015:246–253

Abbasi S, Tajeripour F (2017) Detection of brain tumor in 3D MRI images using local binary patterns and histogram orientation gradient. Neurocomputing 219:526–535

Sauwen N, Acou M, Sima DM, Veraart J, Maes F, Himmelreich U et al (2017) Semi-automated brain tumor segmentation on multi-parametric MRI using regularized non-negative matrix factorization. BMC Med Imaging 17:29

Ilunga-Mbuyamba E, Avina–Cervantes JG, Garcia-Perez A, de Jesus Romero–Troncoso R, Aguirre–Ramos H, Cruz–Aceves I et al (2017) Localized active contour model with background intensity compensation applied on automatic MR brain tumor segmentation. Neurocomputing 220:84–97

Akbar S, Akram MU, Sharif M, Tariq A, Khan SA (2018) Decision support system for detection of hypertensive retinopathy using arteriovenous ratio. Artif Intell Med 90:15–24

Banerjee S, Mitra S, Shankar BU (2018) Automated 3D segmentation of brain tumor using visual saliency. Inf Sci 424:337–353

Article   MathSciNet   Google Scholar  

Raja NSM, Fernandes SL, Dey N, Satapathy SC, Rajinikanth V (2018) Contrast enhanced medical MRI evaluation using Tsallis entropy and region growing segmentation. J Ambient Intell Human Comput:1–12

Subudhi A, Dash M, Sabut S (2020) Automated segmentation and classification of brain stroke using expectation-maximization and random forest classifier. Biocybern Biomed Eng 40:277–289

Gupta N, Bhatele P, Khanna P (2019) Glioma detection on brain MRIs using texture and morphological features with ensemble learning. Biomed Signal Process Control 47:115–125

Myronenko A, Hatamizadeh A (2020) Robust semantic segmentation of brain tumor regions from 3D MRIs. arXiv:2001.02040

Karayegen G, Aksahin MF (2021) Brain tumor prediction on MR images with semantic segmentation by using deep learning network and 3D imaging of tumor region. Biomed Signal Process Control 66:102458

Prima S, Ayache N, Barrick T, Roberts N (2001) Maximum likelihood estimation of the bias field in MR brain images: Investigating different modelings of the imaging process. In: International conference on medical image computing and computer-assisted intervention, pp 811–819

Haider W, Sharif M, Raza M (2011) Achieving accuracy in early stage tumor identification systems based on image segmentation and 3D structure analysis. Comput Eng Intell Syst 2:96–102

Irum I, Shahid MA, Sharif M, Raza M (2015) A review of image denoising methods. J Eng Sci Technol Rev 8:1–11

Kumar SS, Dharun VS (2016) A study of MRI segmentation methods in automatic brain tumor detection. Int J Eng Technol 8:609–614

Dhas A, Madheswaran M (2018) An improved classification system for brain tumours using wavelet transform and neural network. West Indian Med J 67:243–247

Krissian K, Aja-Fernández S (2009) Noise-driven anisotropic diffusion filtering of MRI. IEEE Trans Image Process 18:2265–2274

Article   MathSciNet   MATH   Google Scholar  

Tahir B, Iqbal S, Usman Ghani Khan M, Saba T, Mehmood Z, Anjum A et al (2019) Feature enhancement framework for brain tumor segmentation and classification. Microsc Res Tech 82:803–811

Said AB, Hadjidj R, Foufou S (2019) Total variation for image denoising based on a novel smart edge detector: an application to medical images. J Math Imaging Vision 61:106–121

Bojorquez JAZ, Jodoin P-M, Bricq S, Walker PM, Brunotte F, Lalande A (2019) Automatic classification of tissues on pelvic MRI based on relaxation times and support vector machine. PLoS ONE 14:1–17

Sandhya G, Kande GB, Satya ST (2019) An efficient MRI brain tumor segmentation by the fusion of active contour model and self-organizing-map. J Biomim Biomater Biomed Eng 40:79–91

Yang Y, Huang S (2006) Novel statistical approach for segmentation of brain magnetic resonance imaging using an improved expectation maximization algorithm. Opt Appl 36:125–136

Mittal M, Goyal LM, Kaur S, Kaur I, Verma A, Hemanth DJ (2019) Deep learning based enhanced tumor segmentation approach for MR brain images. Appl Soft Comput 78:346–354

Roy S, Bandyopadhyay SK (2012) Detection and quantification of brain tumor from MRI of brain and it’s symmetric analysis. Int J Inf Commun Technol Res 2:1–7

Gao J, Xie M (2009) Skull-stripping MR brain images using anisotropic diffusion filtering and morphological processing. In: 2009 IEEE international symposium on computer network and multimedia technology, pp 1–4

Maintz JA, Viergever MA (1998) A survey of medical image registration. Med Image Anal 2:1–36

Sharma P, Diwakar M, Choudhary S (2012) Application of edge detection for brain tumor detection. Int J Comput Appl 58:1–6

Popescu V, Battaglini M, Hoogstrate W, Verfaillie SC, Sluimer I, van Schijndel RA et al (2012) Optimizing parameter choice for FSL-brain extraction tool (BET) on 3D T1 images in multiple sclerosis. Neuroimage 61:1484–1494

Despotović I, Goossens B, Philips W (2015) MRI segmentation of the human brain: challenges, methods, and applications. Comput Math Methods Med 2015:1–23

Wong KP (2005) Medical image segmentation: methods and applications in functional imaging. In: Handbook of biomedical image analysis, pp 111–182

Chae SY, Suh S, Ryoo I, Park A, Noh KJ, Shim H et al (2017) A semi-automated volumetric software for segmentation and perfusion parameter quantification of brain tumors using 320-row multidetector computed tomography: a validation study. Neuroradiology 59:461–469

Sauwen N, Acou M, Sima DM, Veraart J, Maes F, Himmelreich U et al (2017) Semi-automated brain tumor segmentation on multi-parametric MRI using regularized non-negative matrix factorization. BMC Med Imaging 17:1–14

Foo JL (2006) A survey of user interaction and automation in medical image segmentation methods. In: Tech rep ISUHCI20062, Human Computer Interaction Department, Iowa State Univ, pp 1–11

Işın A, Direkoğlu C, Şah M (2016) Review of MRI-based brain tumor image segmentation using deep learning methods. Procedia Comput Sci 102:317–324

Shanthi KJ, Kumar MS (2007) Skull stripping and automatic segmentation of brain MRI using seed growth and threshold techniques. In: 2007 International conference on intelligent and advanced systems, pp 422–426

Yao J (2006) Image processing in tumor imaging, new techniques in oncologic imaging. Zhang, F., & Hancock, ER Zhang. New Riemannian techniques for directional and tensorial image data. Pattern Recogn 43:1590–1606

Stadlbauer A, Moser E, Gruber S, Buslei R, Nimsky C, Fahlbusch R et al (2004) Improved delineation of brain tumors: an automated method for segmentation based on pathologic changes of 1H-MRSI metabolites in gliomas. Neuroimage 23:454–461

Lakare S, Kaufman A (2000) 3D segmentation techniques for medical volumes, Center for Visual Computing. Department of Computer Science, State University of New York, New York, pp 59–68

Sato M, Lakare S, Wan M, Kaufman A, Nakajima M (2000) A gradient magnitude based region growing algorithm for accurate segmentation. In: Proceedings 2000 International Conference on Image Processing, vol 3, pp 448–451

Salman YM (2009) Modified technique for volumetric brain tumor measurements. J Biomed Sci Eng 2:16

Singh NP, Dixit S, Akshaya AS, Khodanpur BI (2017) Gradient magnitude based watershed segmentation for brain tumor segmentation and classification. In: Proceedings of the 5th international conference on frontiers in intelligent computing: theory and applications, pp 611–619

Husain RA, Zayed AS, Ahmed WM, Elhaji HS (2015) Image segmentation with improved watershed algorithm using radial bases function neural networks. In: 2015 16th International conference on sciences and techniques of automatic control and computer engineering (STA), pp 121–126

Masood S, Sharif M, Yasmin M, Raza M, Mohsin S (2013) Brain image compression: a brief survey. Res J Appl Sci Eng Technol 5:49–59

Mughal B, Muhammad N, Sharif M (2018) Deviation analysis for texture segmentation of breast lesions in mammographic images. Eur Phys J Plus 133:1–15

Anjum MA, Amin J, Sharif M, Khan HU, Malik MSA, Kadry S (2020) Deep semantic segmentation and multi-class skin lesion classification based on convolutional neural network. IEEE Access 8:129668–129678

Gies V, Bernard TM (2004) Statistical solution to watershed over-segmentation. In: 2004 International Conference on Image Processing. ICIP'04, vol 3, pp 1863–1866

Kong J, Wang J, Lu Y, Zhang J, Li Y, Zhang B (2006) A novel approach for segmentation of MRI brain images. In: MELECON 2006-2006 IEEE Mediterranean Electrotechnical Conference, pp 525–528

Couprie M, Bertrand G (1997) Topological gray-scale watershed transformation. In: Vision Geometry VI International Society for Optics and Photonics, vol 3168, pp 136–146

Lotufo RA, Falcão AX, Zampirolli FA (2002) IFT-watershed from gray-scale marker. In: Proceedings. XV Brazilian Symposium on Computer Graphics and Image Processing, pp 146–152

Benson CC, Lajish VL, Rajamani K (2015) Brain tumor extraction from MRI brain images using marker based watershed algorithm. In: 2015 International Conference on advances in computing, communications and informatics (ICACCI), pp 318–323

Nasir M, Attique Khan M, Sharif M, Lali IU, Saba T, Iqbal T (2018) An improved strategy for skin lesion detection and classification using uniform segmentation and feature selection based approach. Microsc Res Tech 81:528–543

Yasmin M, Sharif M, Masood S, Raza M, Mohsin S (2012) Brain image enhancement—a survey. World Appl Sci J 17:1192–1204

Shah GA, Khan A, Shah AA, Raza M, Sharif M (2015) A review on image contrast enhancement techniques using histogram equalization. Sci Int 27:1-10

Khan MA, Akram T, Sharif M, Shahzad A, Aurangzeb K, Alhussein M et al (2018) An implementation of normal distribution based segmentation and entropy controlled features selection for skin lesion detection and classification. BMC Cancer 18:1–20

Khan MA, Akram T, Sharif M, Saba T, Javed K, Lali IU et al (2019) Construction of saliency map and hybrid set of features for efficient segmentation and classification of skin lesion. Microsc Res Tech 82:741–763

Akram T, Khan MA, Sharif M, Yasmin M (2018) Skin lesion segmentation and recognition using multichannel saliency estimation and M-SVM on selected serially fused features. J Ambient Intell Human Comput:1–20

Yasmin M, Sharif M, Mohsin S, Azam F (2014) Pathological brain image segmentation and classification: a survey. Curr Med Imaging 10:163–177

Mughal B, Muhammad N, Sharif M (2018) Deviation analysis for texture segmentation of breast lesions in mammographic images. Eur Phys J Plus 133:455

Hameed M, Sharif M, Raza M, Haider SW, Iqbal M (2012) Framework for the comparison of classifiers for medical image segmentation with transform and moment based features. Res J Recent Sci 2277:2502

Irum I, Raza M, Sharif M (2012) Morphological techniques for medical images: a review. Res J Appl Sci Eng Technol 4:2948–2962

Jafarpour S, Sedghi Z, Amirani MC (2012) A robust brain MRI classification with GLCM features. Int J Comput Appl 37:1–5

Mughal B, Muhammad N, Sharif M (2019) Adaptive hysteresis thresholding segmentation technique for localizing the breast masses in the curve stitching domain. Int J Med Inf 126:26–34

Nabizadeh N, Kubat M (2015) Brain tumors detection and segmentation in MR images: Gabor wavelet vs. statistical features. Comput Electr Eng 45:286–301

Tiwari P, Sachdeva J, Ahuja CK, Khandelwal N (2017) Computer aided diagnosis system—a decision support system for clinical diagnosis of brain tumours. Int J Comput Intell Syst 10:104–119

Zhang Y, Yang J, Wang S, Dong Z, Phillips P (2017) Pathological brain detection in MRI scanning via Hu moment invariants and machine learning. J Exp Theor Artif Intell 29:299–312

Lahmiri S (2017) Glioma detection based on multi-fractal features of segmented brain MRI by particle swarm optimization techniques. Biomed Signal Process Control 31:148–155

Xu X, Zhang X, Tian Q, Zhang G, Liu Y, Cui G et al (2017) Three-dimensional texture features from intensity and high-order derivative maps for the discrimination between bladder tumors and wall tissues via MRI. Int J Comput Assist Radiol Surg 12:645–656

Shanthakumar P, Ganeshkumar P (2015) Performance analysis of classifier for brain tumor detection and diagnosis. Comput Electr Eng 45:302–311

Zhang B, Chang K, Ramkissoon S, Tanguturi S, Bi WL, Reardon DA et al (2017) Multimodal MRI features predict isocitrate dehydrogenase genotype in high-grade gliomas. Neuro Oncol 19:109–117

Srinivas B, Rao GS (2019) Performance evaluation of fuzzy C means segmentation and support vector machine classification for MRI brain tumor. In: Soft computing for problem solving. Springer, New York, pp 355–367

Herlidou-Meme S, Constans J, Carsin B, Olivie D, Eliat P, Nadal-Desbarats L et al (2003) MRI texture analysis on texture test objects, normal brain and intracranial tumors. Magn Reson Imaging 21:989–993

Soltaninejad M, Yang G, Lambrou T, Allinson N, Jones TL, Barrick TR et al (2017) Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI. Int J Comput Assist Radiol Surg 12:183–203

Yang D, Rao G, Martinez J, Veeraraghavan A, Rao A (2015) Evaluation of tumor-derived MRI-texture features for discrimination of molecular subtypes and prediction of 12-month survival status in glioblastoma. Med Phys 42:6725–6735

Islam A, Reza SM, Iftekharuddin KM (2013) Multifractal texture estimation for detection and segmentation of brain tumors. IEEE Trans Biomed Eng 60:3204–3215

Pei L, Bakas S, Vossough A, Reza SM, Davatzikos C, Iftekharuddin KM (2020) Longitudinal brain tumor segmentation prediction in MRI using feature and label fusion. Biomed Signal Process Control 55:101648

Khan H, Shah PM, Shah MA, ul Islam S, Rodrigues JJ (2020) Cascading handcrafted features and convolutional neural network for IoT-enabled brain tumor segmentation. Comput Commun 153:196–207

Dixit A, Nanda A (2021) An improved whale optimization algorithm-based radial neural network for multi-grade brain tumor classification. Visual Comput:1–16

Zhang Y, Dong Z, Wu L, Wang S (2011) A hybrid method for MRI brain image classification. Expert Syst Appl 38:10049–10053

Zöllner FG, Emblem KE, Schad LR (2012) SVM-based glioma grading: optimization by feature reduction analysis. Z Med Phys 22:205–214

Arakeri MP, Reddy GRM (2015) Computer-aided diagnosis system for tissue characterization of brain tumor on magnetic resonance images. SIViP 9:409–425

Nachimuthu DS, Baladhandapani A (2014) Multidimensional texture characterization: on analysis for brain tumor tissues using MRS and MRI. J Digit Imaging 27:496–506

Pinto A, Pereira S, Dinis H, Silva CA, Rasteiro DM (2015) Random decision forests for automatic brain tumor segmentation on multi-modal MRI images. In: 2015 IEEE 4th Portuguese meeting on bioengineering (ENBENG), pp 1–5

Tustison NJ, Shrinidhi K, Wintermark M, Durst CR, Kandel BM, Gee JC et al (2015) Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with ANTsR. Neuroinformatics 13:209–225

Yang G, Zhang Y, Yang J, Ji G, Dong Z, Wang S et al (2016) Automated classification of brain images using wavelet-energy and biogeography-based optimization. Multimedia Tools Appl 75:15601–15617

Wang S, Du S, Atangana A, Liu A, Lu Z (2018) Application of stationary wavelet entropy in pathological brain detection. Multimedia Tools Appl 77:3701–3714

Padlia M, Sharma J (2019) Fractional Sobel filter based brain tumor detection and segmentation using statistical features and SVM. In: Nanoelectronics, circuits and communication systems, pp 161–175

Ayadi W, Elhamzi W, Charfi I, Atri M (2021) Deep CNN for brain tumor classification. Neural Process Lett 53:671–700

Afza F, Khan MA, Sharif M, Rehman A (2019) Microscopic skin laceration segmentation and classification: a framework of statistical normal distribution and optimal feature selection. Microsc Res Tech 82:1471–1488

Adair J, Brownlee A, Ochoa G (2017) Evolutionary algorithms with linkage information for feature selection in brain computer interfaces. In: Advances in computational intelligence systems. Springer, New York, pp 287–307

Sharma M, Mukharjee S (2012) Brain tumor segmentation using hybrid genetic algorithm and artificial neural network fuzzy inference system (anfis). Int J Fuzzy Logic Syst 2:31–42

Huda S, Yearwood J, Jelinek HF, Hassan MM, Fortino G, Buckland M (2016) A hybrid feature selection with ensemble classification for imbalanced healthcare data: a case study for brain tumor diagnosis. IEEE Access 4:9145–9154

Jothi G (2016) Hybrid tolerance rough set-firefly based supervised feature selection for MRI brain tumor image classification. Appl Soft Comput 46:639–651

Ahmed S, Iftekharuddin KM, Vossough A (2011) Efficacy of texture, shape, and intensity feature fusion for posterior-fossa tumor segmentation in MRI. IEEE Trans Inf Technol Biomed 15:206–213

Wu G, Chen Y, Wang Y, Yu J, Lv X, Ju X et al (2018) Sparse representation-based Radiomics for the diagnosis of brain tumors. IEEE Trans Med Imaging 37:893–905

Fernandez-Lozano C, Seoane JA, Gestal M, Gaunt TR, Dorado J, Campbell C (2015) Texture classification using feature selection and kernel-based techniques. Soft Comput 19:2469–2480

Dandu JR, Thiyagarajan AP, Murugan PR, Govindaraj V (2019) Brain and pancreatic tumor segmentation using SRM and BPNN classification. Health Technol:1–9

Saritha M, Joseph KP, Mathew AT (2013) Classification of MRI brain images using combined wavelet entropy based spider web plots and probabilistic neural network. Pattern Recogn Lett 34:2151–2156

Sharif M, Khan MA, Akram T, Javed MY, Saba T, Rehman A (2017) A framework of human detection and action recognition based on uniform segmentation and combination of Euclidean distance and joint entropy-based features selection. EURASIP J Image Video Process 2017:1–18

Lakshmi A, Arivoli T, Rajasekaran MP (2018) A Novel M-ACA-based tumor segmentation and DAPP feature extraction with PPCSO-PKC-based MRI classification. Arab J Sci Eng 43:7095–7111

Rathi V, Palani S (2012) Brain tumor MRI image classification with feature selection and extraction using linear discriminant analysis. arXiv:1208.2128

Naqi SM, Sharif M, Yasmin M (2018) Multistage segmentation model and SVM-ensemble for precise lung nodule detection. Int J Comput Assist Radiol Surg 13:1083–1095

Amin J, Anjum MA, Sharif M, Kadry S, Nam Y, Wang S (2021) Convolutional bi-LSTM based human gait recognition using video sequences. CMC Comput Mater Contin 68:2693–2709

Amin J, Sharif M, Anjum MA, Nam Y, Kadry S, Taniar D (2021) Diagnosis of COVID-19 infection using three-dimensional semantic segmentation and classification of computed tomography images. Comput Mater Contin:2451–2467

Amin J, Sharif M, Raza M, Saba T, Rehman A (2019) Brain tumor classification: feature fusion. In: 2019 international conference on computer and information sciences (ICCIS), pp 1–6

Amin J, Sharif M, Yasmin M, Ali H, Fernandes SL (2017) A method for the detection and classification of diabetic retinopathy using structural predictors of bright lesions. J Comput Sci 19:153–164

Amin J, Sharif M, Yasmin M, Fernandes SL (2018) Big data analysis for brain tumor detection: deep convolutional neural networks. Futur Gener Comput Syst 87:290–297

Muhammad N, Sharif M, Amin J, Mehboob R, Gilani SA, Bibi N et al (2018) Neurochemical alterations in sudden unexplained perinatal deaths—a review. Front Pediatr 6:6

Sharif M, Amin J, Raza M, Anjum MA, Afzal H, Shad SA (2020) Brain tumor detection based on extreme learning. Neural Comput Appl:1–13

Umer MJ, Amin J, Sharif M, Anjum MA, Azam F, Shah JH (2021) An integrated framework for COVID-19 classification based on classical and quantum transfer learning from a chest radiograph. Concurr Comput Pract Exp 19:153–164

Jiang J, Wu Y, Huang M, Yang W, Chen W, Feng Q (2013) Brain tumor segmentation in multimodal MR images based on learning population-and patient-specific feature sets. Comput Med Imaging Graph 37:512–521

Ortiz A, Gorriz JM, Ramírez J, Salas-Gonzalez D, Alzheimer’s Disease Neuroimaging Initiative (2013) Improving MRI segmentation with probabilistic GHSOM and multiobjective optimization 114:118–131

Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31

Zhang N, Ruan S, Lebonvallet S, Liao Q, Zhu Y (2011) Kernel feature selection to fuse multi-spectral MRI images for brain tumor segmentation. Comput Vis Image Underst 115:256–269

Ortega-Martorell S, Lisboa PJ, Vellido A, Simoes RV, Pumarola M, Julià-Sapé M et al (2012) Convex non-negative matrix factorization for brain tumor delimitation from MRSI data. PLoS ONE 7:e47824

Ali AH, Al-hadi SA, Naeemah MR, Mazher AN (2018) Classification of brain lesion using K-nearest neighbor technique and texture analysis. J Phys Conf Ser:012018

Supot S, Thanapong C, Chuchart P, Manas S (2007) Segmentation of magnetic resonance images using discrete curve evolution and fuzzy clustering. In: 2007 IEEE International Conference on Integration Technology, pp 697–700

Fletcher-Heath LM, Hall LO, Goldgof DB, Murtagh FR (2001) Automatic segmentation of non-enhancing brain tumors in magnetic resonance images. Artif Intell Med 21:43–63

Abdulbaqi HS, Mat MZ, Omar AF, Mustafa ISB, Abood LK (2014) Detecting brain tumor in magnetic resonance images using hidden Markov random fields and threshold techniques. In: 2014 IEEE student conference on research and development, pp 1–5

Parekh VS, Laterra J, Bettegowda C, Bocchieri AE, Pillai JJ, Jacobs MA (2019) Multiparametric deep learning and radiomics for tumor grading and treatment response assessment of brain cancer: preliminary results, pp 1–6. arXiv:1906.04049

Zadeh Shirazi A, Fornaciari E, McDonnell MD, Yaghoobi M, Cevallos Y, Tello-Oquendo L et al (2020) The Application of Deep Convolutional Neural Networks to Brain Cancer Images: A Survey. J Pers Med 10:224

Guan B, Yao J, Zhang G, Wang XJPRL (2019) Thigh fracture detection using deep learning method based on new dilated convolutional feature pyramid network 125:521–526

Raza M, Sharif M, Yasmin M, Khan MA, Saba T, Fernandes SL (2018) Appearance based pedestrians’ gender recognition by employing stacked auto encoders in deep learning. Future Gener Comput Syst 88:28–39

Liaqat A, Khan MA, Shah JH, Sharif M, Yasmin M, Fernandes SL (2018) Automated ulcer and bleeding classification from WCE images using multiple features fusion and selection. J Mech Med Biol 18:1850038

Naqi S, Sharif M, Yasmin M, Fernandes SL (2018) Lung nodule detection using polygon approximation and hybrid features from CT images. Curr Med Imaging 14:108–117

Ansari GJ, Shah JH, Yasmin M, Sharif M, Fernandes SL (2018) A novel machine learning approach for scene text extraction. Futur Gener Comput Syst 87:328–340

Fatima Bokhari ST, Sharif M, Yasmin M, Fernandes SL (2018) Fundus image segmentation and feature extraction for the detection of glaucoma: a new approach. Curr Med Imaging 14:77–87

Jain VK, Kumar S, Fernandes SL (2017) Extraction of emotions from multilingual text using intelligent text processing and computational linguistics. J Comput Sci 21:316–326

Fernandes SL, Gurupur VP, Lin H, Martis RJ (2017) A novel fusion approach for early lung cancer detection using computer aided diagnosis techniques. J Med Imaging Health Inf 7:1841–1850

Raja N, Rajinikanth V, Fernandes SL, Satapathy SC (2017) Segmentation of breast thermal images using Kapur’s entropy and hidden Markov random field. J Med Imaging Health Inf 7:1825–1829

Rajinikanth V, Madhavaraja N, Satapathy SC, Fernandes SL (2017) Otsu’s multi-thresholding and active contour snake model to segment dermoscopy images. J Med Imaging Health Inf 7:1837–1840

Shah JH, Chen Z, Sharif M, Yasmin M, Fernandes SL (2017) A novel biomechanics-based approach for person re-identification by generating dense color sift salience features. J Mech Med Biol 17:1740011

Fernandes SL, Bala GJ (2017) A comparative study on various state of the art face recognition techniques under varying facial expressions. Int Arab J Inf Technol 14:254–259

Yasmin M, Sharif M, Mohsin S (2013) Neural networks in medical imaging applications: a survey. World Appl Sci J 22:85–96

Chen L, Bentley P, Rueckert D (2017) Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks. NeuroImage Clin 15:633–643

Abd-Ellah MK, Awad AI, Khalaf AA, Hamed HF (2018) Two-phase multi-model automatic brain tumour diagnosis system from magnetic resonance images using convolutional neural networks. EURASIP J Image Video Process 2018:97

Larochelle H, Jodoin P-M (2016) A convolutional neural network approach to brain tumor segmentation. In: Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries: first international workshop, brainles 2015, Held in Conjunction with MICCAI 2015, Munich, Germany, 5 Oct 5, revised selected papers

Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78

Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. In: Annual conference on medical image understanding and analysis, pp 506–517

Chen S, Ding C, Liu M (2019) Dual-force convolutional neural networks for accurate brain tumor segmentation. Pattern Recogn 88:90–100

Wang Y, Li C, Zhu T, Zhang J (2019) Multimodal brain tumor image segmentation using WRN-PPNet. Comput Med Imaging Graph 75:56–65

Chelghoum R, Ikhlef A, Hameurlaine A, Jacquir S (2020) Transfer learning using convolutional neural network architectures for brain tumor classification from MRI images. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp 189–200

Mehrotra R, Ansari MA, Agrawal R, Anand RS (2020) A transfer learning approach for AI-based classification of brain tumors. Mach Learn Appl 2:1–12

Zikic D, Ioannou Y, Brown M, Criminisi A (2014) Segmentation of brain tumor tissues with convolutional neural networks. In: Proceedings MICCAI-BRATS, pp 36–39

Dvořák P, Menze B (2015) Local structure prediction with convolutional neural networks for multimodal brain tumor segmentation. In: International MICCAI workshop on medical computer vision, pp 59–71

Pereira S, Pinto A, Alves V, Silva CA (2016) Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging 35:1240–1251

Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y (2018) A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Med Image Anal 43:98–111

Hussain S, Anwar SM, Majid M (2018) Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing 282:248–261

Sharif MI, Li JP, Amin J, Sharif A (2021) An improved framework for brain tumor analysis using MRI based on YOLOv2 and convolutional neural network. Complex Intell Syst:1–14

Kao PY, Shailja S, Jiang J, Zhang A, Khan A, Chen JW, Manjunath BS (2020) Improving patch-based convolutional neural networks for MRI brain tumor segmentation by leveraging location information. Front Neurosci 13:1–12

Ahuja S, Panigrahi B, Gandhi T (2020) Transfer learning based brain tumor detection and segmentation using superpixel technique. In: 2020 International Conference on Contemporary Computing and Applications (IC3A), pp 244–249

Rehman A, Naz S, Razzak MI, Akram F, Imran M (2020) A deep learning-based framework for automatic brain tumors classification using transfer learning. Circuits Syst Signal Process 39:757–775

Guy-Fernand KN, Zhao J, Sabuni FM, Wang J (2020) Classification of brain tumor leveraging goal-driven visual attention with the support of transfer learning. In: 2020 Information Communication Technologies Conference (ICTC), pp 328–332

Kaur T, Gandhi TK (2020) Deep convolutional neural networks with transfer learning for automated brain image classification. Mach Vis Appl 31:1–16

Gu Y, Li K (2021) A Transfer Model Based on Supervised Multi-Layer Dictionary Learning for Brain Tumor MRI Image Recognition. Front Neurosci 15:550

Sadad T, Rehman A, Munir A, Saba T, Tariq U, Ayesha N et al (2021) Brain tumor detection and multi-classification using advanced deep learning techniques. Microsc Res Tech 84:1296–1308

Panwar SA, Munot MV, Gawande S, Deshpande PS (2021) A reliable and an efficient approach for diagnosis of brain tumor using transfer learning. Biomed Pharmacol J 14:283–294

Kumar RL, Kakarla J, Isunuri BV, Singh M (2021) Multi-class brain tumor classification using residual network and global average pooling. Multimedia Tools Appl 80:13429–13438

Arbane M, Benlamri R, Brik Y, Djerioui M (2021) Transfer learning for automatic brain tumor classification using MRI images. In: 2020 2nd International Workshop on Human-Centric Smart Environments for Health and Well-being (IHSH), pp 210–214

Amin J, Sharif M, Yasmin M, Saba T, Anjum MA, Fernandes SL (2019) A new approach for brain tumor segmentation and classification based on score level fusion using transfer learning. J Med Syst 43:1–16

Wu W, Chen AY, Zhao L, Corso JJ (2014) Brain tumor detection and segmentation in a CRF (conditional random fields) framework with pixel-pairwise affinity and superpixel-level features. Int J Comput Assist Radiol Surg 9:241–253

Noreen N, Palaniappan S, Qayyum A, Ahmad I, Imran M, Shoaib M (2020) A deep learning model based on concatenation approach for the diagnosis of brain tumor. IEEE Access 8:55135–55144

Zhang J, Jiang Z, Dong J, Hou Y, Liu B (2020) Attention gate resU-Net for automatic MRI brain tumor segmentation. IEEE Access 8:58533–58545

Rehman ZU, Zia MS, Bojja GR, Yaqub M, Jinchao F, Arshid K (2020) Texture based localization of a brain tumor from MR-images by using a machine learning approach. Med Hypotheses 141:109705

Zeineldin RA, Karar ME, Coburger J, Wirtz CR, Burgert O (2020) DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images. Int J Comput Assist Radiol Surg 15:909–920

Swati ZNK, Zhao Q, Kabir M, Ali F, Ali Z, Ahmed S et al (2019) Content-based brain tumor retrieval for MR images using transfer learning. IEEE Access 7:17809–17822

Pravitasari AA, Iriawan N, Almuhayar M, Azmi T, Fithriasari K, Purnami SW et al (2020) UNet-VGG16 with transfer learning for MRI-based brain tumor segmentation. Telkomnika 18:1310–1318

Deepak S, Ameer P (2019) Brain tumor classification using deep CNN features via transfer learning. Comput Biol Med 111:3345

Lu S, Lu Z, Zhang Y-D (2019) Pathological brain detection based on AlexNet and transfer learning. J Comput Sci 30:41–47

Wacker J, Ladeira M, Nascimento JEV (2019) Transfer learning for brain tumor segmentation. arXiv:1912.12452

Soumik MFI, Hossain MA (2020) Brain tumor classification with inception network based deep learning model using transfer learning. In: 2020 IEEE Region 10 Symposium (TENSYMP), pp 1018–1021

Khamparia A, Gupta D, de Albuquerque VHC, Sangaiah AK, Jhaveri RH (2020) Internet of health things-driven deep learning system for detection and classification of cervical cells using transfer learning. J Supercomput 76:1–19

Yang Y, Yan LF, Zhang X, HanY, Nan HY, Hu YC, Hu B, Yan SL, Zhang J, Cheng DL, Ge XW (2018) Glioma grading on conventional MR images: a deep learning study with transfer learning. Front Neurosci 12:1–10

Banerjee S, Mitra S, Masulli F, Rovetta S (2019) Deep radiomics for brain tumor detection and classification from multi-sequence MRI. arXiv:1903.09240

Hao R, Namdar K, Liu L, Khalvati F (2021) A transfer learning–based active learning framework for brain tumor classification. Front Artif Intell 4

Saxena P, Maheshwari A, Maheshwari S (2021) Predictive modeling of brain tumor: a deep learning approach. In: Innovations in computational intelligence and computer vision, pp 275–285

Bergholm V, Izaac J, Schuld M, Gogolin C, Alam MS, Ahmed S et al (2018) Pennylane: automatic differentiation of hybrid quantum-classical computations. arXiv:1811.04968

Zhou R, Zhou L, Jiang N, Ding Q (2006) Dynamic analysis and application of QANN. In: First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS’06), pp 347–351

Li P, Xiao H (2014) Sequence input-based quantum-inspired neural networks with applications. Neural Process Lett 40:143–168

Konar D, Bhattacharyya S, Panigrahi BK, Behrman EC (2021) Qutrit-inspired fully self-supervised shallow quantum learning network for brain tumor segmentation. IEEE Trans Neural Netw Learn Syst

Agrawal U, Brown EN, Lewis LD (2020) Model-based physiological noise removal in fast fMRI. Neuroimage 205:1–18

Dubey YK, Mushrif MM (2016) FCM clustering algorithms for segmentation of brain MR images. Adv Fuzzy Syst 2016:8

MathSciNet   Google Scholar  

Sharif MI, Khan MA, Alhussein M, Aurangzeb K, Raza M (2021) A decision support system for multimodal brain tumor classification using deep learning. Complex Intell Syst:1–14

Irshad M, Muhammad N, Sharif M, Yasmeen M (2018) Automatic segmentation of the left ventricle in a cardiac MR short axis image using blind morphological operation. Eur Phys J Plus 133:1–14

Han C, Rundo L, Araki R, Furukawa Y, Mauri G, Nakayama H et al (2020) Infinite brain MR images: PGGAN-based data augmentation for tumor detection. Neural approaches to dynamics of signal exchanges. Springer, New York, pp 291–303

Juan-Albarracín J, Fuster-Garcia E, Manjon JV, Robles M, Aparici F, Martí-Bonmatí L et al (2015) "Automated glioblastoma segmentation based on a multiparametric structured unsupervised classification. PLoS ONE 10:e0125143

Soltaninejad M, Yang G, Lambrou T, Allinson N, Jones TL, Barrick TR et al (2018) "Supervised learning based multimodal MRI brain tumour segmentation using texture features from supervoxels. Comput Methods Progr Biomed 157:69–84

Ito KL, Kim H, Liew SL (2019) A comparison of automated lesion segmentation approaches for chronic stroke T1-weighted MRI data. Hum Brain Mapp 40:4669–4685

Chen X, You S, Tezcan KC, Konukoglu EJMIA (2020) Unsupervised lesion detection via image restoration with a normative prior

Liu P, Dou Q, Wang Q, Heng PA (2020) An encoder-decoder neural network with 3D squeeze-and-excitation and deep supervision for brain tumor segmentation. IEEE Access 8:34029–34037

Deb D, Roy S (2021) Brain tumor detection based on hybrid deep neural network in MRI by adaptive squirrel search optimization. Multimedia Tools Appl 80:2621–2645

Feng X, Tustison NJ, Patel SH, Meyer CH (2020) "Brain tumor segmentation using an ensemble of 3d u-nets and overall survival prediction using radiomic features. Front Comput Neurosci 14:25

Zhou C, Ding C, Wang X, Lu Z, Tao D (2020) "One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Trans Image Process 29:4516–4529

Download references

Author information

Authors and affiliations.

Department of Computer Science, University of Wah, Wah Cantt, Pakistan

Javaria Amin

Department of Computer Science, COMSATS University Islamabad, Wah Campus, Islamabad, Pakistan

Javaria Amin, Muhammad Sharif & Mussarat Yasmin

Department of Computer Science and Engineering, Sri Eshwar College of Engineering, Coimbatore, Tamil Nadu, India

Anandakumar Haldorai

Department of IS&E, Canara Engineering College, Mangaluru, Karnataka, India

Ramesh Sundar Nayak

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Javaria Amin .

Ethics declarations

Conflict of interest.

There is no grant received from any resources. All authors declare that they have no conflict of interest.

Research involving human participants and/or animals

It is declared that research has not involved any human participants and animals.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Amin, J., Sharif, M., Haldorai, A. et al. Brain tumor detection and classification using machine learning: a comprehensive survey. Complex Intell. Syst. 8 , 3161–3183 (2022). https://doi.org/10.1007/s40747-021-00563-y

Download citation

Received : 28 July 2021

Accepted : 12 October 2021

Published : 08 November 2021

Issue Date : August 2022

DOI : https://doi.org/10.1007/s40747-021-00563-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Feature extraction
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 27 January 2022

Classification of brain tumours in MR images using deep spatiospatial models

• Soumick Chatterjee 1 , 2 , 3   na1 ,
• Faraz Ahmed Nizamani 4   na1 ,
• Andreas Nürnberger 2 , 3 , 5 &
• Oliver Speck 1 , 5 , 6 , 7  

Scientific Reports volume  12 , Article number:  1505 ( 2022 ) Cite this article

20k Accesses

44 Citations

16 Altmetric

Metrics details

• Cancer imaging
• Cancer screening
• Computer science

A brain tumour is a mass or cluster of abnormal cells in the brain, which has the possibility of becoming life-threatening because of its ability to invade neighbouring tissues and also form metastases. An accurate diagnosis is essential for successful treatment planning, and magnetic resonance imaging is the principal imaging modality for diagnosing brain tumours and their extent. Deep Learning methods in computer vision applications have shown significant improvement in recent years, most of which can be credited to the fact that a sizeable amount of data is available to train models, and the improvements in the model architectures yield better approximations in a supervised setting. Classifying tumours using such deep learning methods has made significant progress with the availability of open datasets with reliable annotations. Typically those methods are either 3D models, which use 3D volumetric MRIs or even 2D models considering each slice separately. However, by treating one spatial dimension separately or by considering the slices as a sequence of images over time, spatiotemporal models can be employed as “spatiospatial” models for this task. These models have the capabilities of learning specific spatial and temporal relationships while reducing computational costs. This paper uses two spatiotemporal models, ResNet (2+1)D and ResNet Mixed Convolution, to classify different types of brain tumours. It was observed that both these models performed superior to the pure 3D convolutional model, ResNet18. Furthermore, it was also observed that pre-training the models on a different, even unrelated dataset before training them for the task of tumour classification improves the performance. Finally, Pre-trained ResNet Mixed Convolution was observed to be the best model in these experiments, achieving a macro F1-score of 0.9345 and a test accuracy of 96.98%, while at the same time being the model with the least computational cost.

Similar content being viewed by others

Detection and classification of brain tumor using hybrid deep learning models

Baiju Babu Vimala, Saravanan Srinivasan, … Gemmachis Teshite Dalu

Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images

Linmin Pei, Lasitha Vidyaratne, … Khan M. Iftekharuddin

A brain MRI dataset and baseline evaluations for tumor recurrence prediction after Gamma Knife radiotherapy

Yibin Wang, William Neil Duggar, … Haifeng Wang

Introduction

A brain tumour is the growth of abnormal cells in the brain. Brain tumours are classified based on their speed of growth and the likeness of them growing back after treatment. They are mainly divided into two overall categories: malignant and benign. Benign tumours are not cancerous, they grow slowly and are less likely to return after treatment. Malignant tumours, on the other hand, are essentially made up of cancer cells, they have the ability to invade the tissues locally, or they can spread to different parts of the body, a process called metastasise 1 . Glioma tumours are the result of glial cell mutations resulting in malignancy of normal cells. They are the most common types of Astrocytomas (tumour of the brain or spinal cord), account for 30% of all brain and central nervous system tumours, and 80% of all malignant tumours 2 . The phenotypical makeup of glioma tumours can consist of Astrocytomas, Oligodendrogliomas, or Ependymomas. Each of these tumours behaves differently, and World Health Organisation (WHO) uses the following grading-based method to categorise each tumour based upon its aggressiveness:

Grade I tumours are generally benign tumours, which means they are mostly curable, and they are commonly found in children.

Grade II includes three types of tumours: Astrocytomas, Oligodendrogliomas, and Oligoastrocytoma—which is a mix of both 3 . They are common in adults. Eventually, all low-grade gliomas can progress to high-grade tumours 3 .

Grade III tumour can include Anaplastic Astrocytomas, Anaplastic Oligodendrogliomas or Anaplastic Oligoastrocytoma. They are more aggressive and infiltrating than grade II.

Grade IV glioma, also called Glioblastoma Multiforme (GBM), is the most aggressive tumour in the WHO category.

In general, grades I and II gliomas are considered low-grade gliomas (LGG), while grades III and IV are known as high-grade glioma (HGG). The LGG are benign tumours, and they can be excised using surgical resection. In contrast, HGGs are malignant tumours that are hard to excise by surgical methods because of their extent of nearby tissue invasion. Figure  1 shows an example MRI of LGG and HGG.

An example MRI of Low-grade glioma (LGG, on the left) and High-grade glioma (HGG, on the right). Source: BraTS 2019.

A Glioblastoma Multiforme (GBM) typically has the following types of tissues (shown in Fig.  2 ):

The Tumour Core : This is the region of the tumour that has the malignant cells that are actively proliferating.

Necrosis : The necrotic region is the important distinguishing factor between low-grade gliomas and GBM 4 . This is the region where the cells/tissue are dying, or they are dead.

Perifocal oedema : The swelling of the brain is caused by fluid build-up around the tumour core, which increases the intracranial pressure; perifocal oedema is caused by the changes in glial cell distribution 5 .

High-grade glioma structure on T1ce, T2 and FLAIR contrast images (from left to right), (red circle) Necrotic core, (blue circle) Perifocal oedema. Source: BraTS 2019.

The prognosis of a brain tumour depends on many factors, such as the tumour’s location, the histological subtype of the tumour, and the tumour margins. In many cases, the tumour reoccurs and progresses to grade IV even after treatment 3 . Modern imaging methods such as MRI can be used for multiple diagnostic purposes; they can be used to identify the tumour location—which is used for investigating tumour progression and surgical pre-planning. MR imaging is also used to study the anatomy of the lesion, physiology, and metabolic activity along with its haemodynamics. Therefore MR imaging remains the primary diagnostic modality for brain tumours.

Detection of cancer, specifically an earlier detection, holds the potential to make a difference in treatment. Earlier detection is vital because lesions in earlier stages are more likely curable; therefore, if intervened early on, this can make the difference between life and death. Deep learning methods can help automate the process of detecting and classifying brain lesions—they can also reduce the radiologists’ burden of reading many images by prioritising only malignant lesions. This will eventually improve the overall efficiency, and it can reduce diagnostic errors 6 . Recent studies have shown that deep learning methods in the field of radiology have already achieved comparable and super-human performance for some pathologies 7 .

Related work

Various deep learning based methods have been proposed in recent times to classify brain tumours. Mzoughi et al. 8 proposed an approach using volumetric CNNs to classify high-grade glioma and low-grade glioma using T1 contrast-enhanced images. Another similar work on glioma classification based on grading was done by Pei et al. 9 , where they first segmented the tumour and then classified the tumour between HGG and LGG. Most of the literature on glioma tumour classification and grading used one single MR contrast image at a time, but Ge et al. 10 used a fusion framework that uses T1 contrast-enhanced, T2, and FLAIR images simultaneously for classifying the tumour. Ouerghi et al. 11 used a novel fusion method for the inclusion of multiple MRI contrasts, first, the T1 images are transformed by non-subsampled shearlet transform (NSST) into low frequency (LF) and high frequency (HF) subimages, essentially separating principle information in the source image from edge information, then the images are fused by predefined rules to include the coefficients, resulting in fusion of T1 and T2 or FLAIR images. Most of the literature only classifies between the different grades of tumour and does not consider healthy brains as an additional class.

Technical background

ResNet or residual network, proposed by He et al. 12 , has shown to be one of the most efficient network architectures for image recognition tasks, dealing with problems of deep networks, e.g. vanishing gradients. This paper introduced residual-link, the identity mappings, which are “skipped connections”, whose outputs are added to the outputs of the rest of the stacked layers. These identity connections do not add any complexity to the network while improving the training process. The spatiotemporal models introduced by Tran et al. 13 for action recognition are fundamentally 3D Convolutional Neural Networks based on ResNet. There are two spatial dimensions and one temporal dimension in video data, making the data three dimensional. For handling such data (e.g. action recognition task), using a network with 3D convolution layers is an obvious choice. Tran et al. 13 introduced two variants of spatiotemporal models: ResNet (2+1)D and ResNet Mixed Convolution. The ResNet(2+1)D model consists of 2D and 1D convolutions, where the 2D convolutions are used spatially while the 1D convolutions are reserved for the temporal element. This gives an advantage of increased non-linearity by using non-linear rectification, which allows this kind of mixed model to be more “learnable” than conventional full 3D models. On the other hand, the ResNet Mixed Convolution model is constructed as a mixture of 2D and 3D Convolution operations. The initial layers of the model are made of 3D convolution operations, while the later layers consist of 2D convolutions. The rationale behind using this type of configuration is that the motion-modelling occurs mostly at the initial layers, and applying 3D convolution there encapsulates action better.

Apart from trying to improve the network architecture, one frequently used technique to improve the performance of the same architecture is transfer learning 14 . This is a technique for re-purposing a model for another task that is different from the task the model was originally trained for performing. Typically, the model parameters are initialised randomly before starting the training. However, in the case of transfer learning, model parameters learned from task one are used as the starting point (called pre-training), instead of random values, for training the model for task two. Pre-training has shown to be an effective method to improve the initial training process, eventually achieving better accuracy 15 , 16 .

Contribution

Spatiotemporal models are typically used for video classification tasks, which are three dimensional in nature. Their potential in classifying 3D volumetric images like MRI, considering them as “spatiospatial” models, has not been explored yet. This explores the possibility of applying spatiotemporal models (ResNet(2+1)D and ResNet Mixed Convolution) as “spatiospatial” models by treating one dimension (slice dimension) differently than the other two spatial dimensions of the 3D volumetric images. “Spatiospatial” were employed to classify brain tumours of the different types of gliomas based on their grading as well as healthy brains from 3D volumetric MR Images using a single MR contrast, and compare their performances against a pure 3D convolutional model (ResNet3D). Furthermore, the models are to be compared with and without pre-training—to judge the usability of transfer learning for this task.

Methodology

This section explains the network models used in this research, implementation details, pre-training and training methods, data augmentation techniques, dataset information, data pre-processing steps, and finally, the evaluation metrics.

Network models

Spatiotemporal models are mainly used for video-related tasks, where there are two spatial and one temporal dimension. These models deal with the spatial and temporal dimensions differently, unlike pure 3D convolution-based models. There is no temporal component in 3D volumetric image classification tasks; hence, using a 3D convolution-based model is a frequent choice. At times, they are divided into 2D slices, and 2D convolution-based models are applied to them. For the task of tumour classification, the rationale for using 3D filters is grounded in the morphological heterogeneity of gliomas 17 , it is to make the convolution kernels invariant to tissue discrimination in all dimensions, learning more complex features spanning voxels, while 2D convolution filters will capture the spatial representation within the slices. Spatiotemporal models combine two different types of convolution into one model while having the possibility of reducing the complexity of the model or of incorporating more non-linearity. These advantages might be possible to exploit while working with volumetric data by considering the spatiotemporal models as “spatiospatial” models—the motivation behind using such models for a tumour classification task. In this paper, the slice-dimension is treated as the pseudo-temporal dimension of spatiotemporal models, and in-plane dimensions are treated as the spatial dimensions. The spatiotemporal models used here as spatiospatial models are based on the work of Tran et al. 13 .

Two different spatiospatial models are explored here: ResNet (2+1)D and ResNet Mixed Convolution. Their performances are compared against ResNet3D, which is a pure 3D convolution-based model.

Schematic representations of the network architectures. ( a ) ResNet (2+1)D, ( b ) ResNet Mixed Convolution, and ( c ) ResNet 3D.

ResNet (2+1)D

ResNet (2+1)D uses a combination of 2D convolution followed by 1D convolution instead of a single 3D convolution. The benefit of using this configuration is that it allows an added non-linear activation unit between the two convolutions, as in comparison to using a single 3D Convolution 13 . This then results in an overall increase of ReLU units in the network, giving the model the ability to learn even more complex functions. The ResNet(2+1)D uses a stem that contains a 2D convolution with a kernel size of seven and a stride of two, accepting one channel as an input and providing 45 channels as output; followed by a 1D convolution with a kernel size of three and a stride of one, providing 64 channels as the final output. Next, there are four convolutional blocks; each of them contains two sets of basic residual blocks. Each residual block contains one 2D convolution with a kernel size of three and a stride of one, followed by a 1D convolution with a kernel size of three and a stride of one. Each convolutional layer in the model (both 2D and 1D) is followed by a 3D batch normalisation layer and a ReLU activation function. The residual blocks inside the convolutional blocks, except for the first convolutional block, are separated by a pair of 3D convolution layers with a kernel size of one and a stride of two—to downsample the input by half. The 2D convolutions are applied in-plane, and the 1D convolutions are applied on the slice dimension. After the final convolutional block, an adaptive average pooling layer has been added, with an output size of one for all three dimensions. After the pooling layer, a dropout layer followed by a fully connected layer with n output neurons for n classes were added to obtain the final output. Figure  3 (a) portrays the schematic diagram of the ResNet (2+1)D architecture.

ResNet mixed convolution

ResNet Mixed Convolution uses a combination of 2D and 3D Convolutions. The stem of this model contains a 3D convolution layer with a kernel size of (3,7,7), a stride of (1,2,2), and padding of (1,3,3)—where the first dimension is the slice dimension and the other two dimensions are the in-plane dimensions, and accepts a single channel as input while providing 64 channels as output. After the stem, there is one 3D convolution block, followed by three 2D convolution blocks. All the convolution layers (both 3D and 2D) have a kernel size of three and a stride of one, identical for all dimensions. Each of these convolution blocks contains a pair of residual blocks, each of which contains a pair of convolution layers. Similar to ResNet (2+1)D, the residual blocks inside the convolutional blocks, except for the first convolutional block, are separated by a pair of 3D convolution layers with a kernel size of one and a stride of two—to downsample the input by half. Each convolutional layer in the model (both 3D and 2D) is followed by a 3D batch normalisation layer and a ReLU activation function. The motivation behind using both modes of convolution in 2D and 3D is that the 3D filters can learn the spatial features of the tumour in 3D space while 2D can learn representation within each 2D slice. After the convolutional blocks, the final pooling, dropout, and fully connected layers are identical to the ResNet (2+1)D architecture. Figure  3 (b) shows the schematic representation of this model.

The performance of the spatiospatial models are compared against a pure 3D ResNet model, schematic diagram shown in Fig.  3 (c). The architecture of the ResNet3D model used here is almost identical to the architecture of ResNet Mixed Convolution (“ Network models ” section), except for the fact that this model uses only 3D convolutions. The stem of these models are identical, the only difference being that this model uses four 3D convolution blocks, unlike ResNet Mixed Convolution, which uses one 3D convolution block, followed by three 2D convolution blocks. This configuration of ResNet3D architecture results in a 3D ResNet18 model.

Summary and comparison

The general structure of the network models can be divided into the following: input goes to the stem, then there are four convolutional blocks, followed by the output block—which contains an adaptive pooling layer, followed by a dropout layer, and finally a fully connected layer. ResNet Mixed Convolution and ResNet 3D have the same stem, including a 3D convolutional layer with a kernel size of (3,7,7), followed by a batch normalisation layer and a ReLU. ResNet (2+1)D uses a different stem: a 2D convolution layer with a kernel size of seven, then a 1D convolution with a kernel size of three—splitting the 3D convolution (3,7,7) used by the other models into a pair of 2D and 1D convolution: (7,7) and (3). Both 2D and 1D convolution inside this stem is followed by a batch normalisation layer and ReLU pair. The convolutional blocks in the ResNet3D and ResNet Mixed Convolution architectures follow the same architecture: two residual blocks consisting of two sub-blocks consisting of a 3D convolution with a kernel size of three, followed by batch normalisation layer and a ReLU. On the other hand, the first convolutional block of the ResNet (2+1)D architecture uses a pair of 2D and 1D convolutions with the kernel size of three instead of the 3D convolutional layers used by the other models. The rest of the architecture is the same. It is noteworthy that this model has more non-linearity because the 3D convolutions are split into a pair of 2D and 1D convolutions; additional pair of batch normalisation and ReLU could have been used between the 2D 1D convolution. There is one difference between the first convolutional block and the other three blocks (applicable for all three models): the second, third and fourth convolutional blocks included a downsampling pair, which consisted of a 3D convolutional layer with a kennel size of one and a stride of two, followed by a batch normalisation layer. This was not present in the first convolutional block. The convolution blocks of each of all three models double the input features by two (number of input features to the first block: 64, number of output features of the fourth (and final) block: 512). All of these models end with an adaptive average pooling layer, which forces the output to have a shape of 1×1×1, with 512 different features. A dropout with a probability of 0.3 is then applied to introduce regularisation to prevent over-fitting before supplying them to a fully connected linear layer that generates n classes as output. The width and depth of these models are comparable, but they differ in terms of the number of trainable parameters depending upon the type of convolution used, as shown in Table  1 . It is noteworthy that the less the number of trainable parameters - the less the computational costs. A model with a lesser number of parameters would require lesser memory for computation (GPU and RAM), and also the complexity of the model is lesser—reducing the overall computational costs for both training and inference. Moreover, a lesser number of trainable parameters would also reduce the risk of overfitting.

Implementation and training

The models were implemented using PyTorch 18 , by modifying the Torchvision models 19 and were trained with a batch-size of 1 using an Nvidia RTX 4000 GPU, which has a memory of 8 GB. Models were compared with and without pre-training. Models with pre-training were pre-trained on Kinetics-400 20 , except for the stems and fully connected layers. Images from the Kinetics dataset contain three channels (RGB Images), whereas the 3D volumetric MRIs have only one channel. Therefore, the stem trained on the Kinetics dataset could not be used and was initialised randomly. Similarly, for the fully connected layer, Kinetics-400 has 400 output classes, whereas the task at hand has three classes (LGG, HGG and Healthy)—hence, this layer was also initialised with random weights.

Trainings were performed using mixed-precision 21 with the help of Nvidia’s Apex library 22 . The loss was calculated using the weighted cross-entropy loss function to minimise the under-representation of classes with fewer samples during training and was optimised using the Adam optimiser with a learning rate of 1e−5 and weight decay coefficient \(\lambda =1\) e−3. The code of this research is publicly available on GitHub: https://github.com/farazahmeds/Classification-of-brain-tumor-using-Spatiotemporal-models .

Weighted cross-entropy loss

The normalised weight value for each class ( \(W_c\) ) is calculated using:

where \(samples_c\) is the number of samples from class c and \(samples_t\) are the total number of samples from all classes. The normalised weight values from this equation is then used to scale cross-entropy loss of the respective class loss:

Where \(x_{c}\) is the true distribution and P(c) is the estimate distribution for class c. The total cross-entropy loss then is the sum of individual class losses.

Data augmentation

Different data augmentation techniques were applied to the dataset before training the models, and for that purpose, TorchIO 23 was used. Initial experiments were performed using different amounts of augmentation and can be categorised as light and heavy augmentation, where light augmentation included only random affine (scale 0.9-1.2, degrees 10) and random flip (L-R, probability 0.25); on the other hand, heavy augmentation included the ones from light augmentation together with elastic deformation and random k-space transformations (motion, spike, and ghosting). It was observed that the training of the network with heavily augmented data not only performed poorly in terms of final accuracy, but the loss took a much longer time to converge. Therefore, only light augmentation was used throughout this research.

Two different datasets were used in this work - the pathological brain images were obtained from the Brain Tumour Segmentation (BraTS) 2019 dataset, which includes images with four different MR contrasts (T1, T1 contrast-enhanced, T2 and FLAIR) 6 , 24 , 25 ; and non-pathological images were collected from the IXI Dataset 26 . Among the available four types of MRIs, T1 contrast-enhanced (T1ce) is the most commonly used contrast while performing single-contrast tumour classification 8 , 27 . Hence in this research, T1ce images of 332 subjects were used from the BRaTS dataset: 259 volumes of Glioblastoma Multiforme (high-grade glioma, HGG), and 73 volumes of low-grade glioma (LGG). 259 T1 weighted volumes were chosen randomly from the IXI dataset as healthy samples to have the same number of subjects as HGG. The final combined dataset was then randomly divided into 3-folds of training and testing split with a ratio of 7:3.

Data pre-processing

The IXI images were pre-processed first by using the brain extraction tool (BET2) of FSL 28 , 29 . This was done to keep the input data uniform throughout, as the BraTS images are already skull stripped. Moreover, the intensity values of all the volumes from the combined datasets were normalised by scaling intensities to [0.5,99.5] percentile, as used by Isensee et al. 30 . Finally, the volumes were re-sampled to the same voxel-resolution of 2mm isotropic.

Evaluation metrics

The performance of the models was compared using precision, recall, F1 score, specificity, and testing accuracy. Furthermore, a confusion matrix was used to show class-wise accuracy.

The performance of the models were compared with and without pre-training. Figures  4 , 5 , and 6 show the average accuracy over 3-fold cross validation using confusion metrics, for ResNet (2+1)D, ResNet Mixed Convolution, and ResNet 3D, respectively.

Confusion matrix for 3-fold cross-validation on pre-trained ResNet(2+1)D.

Confusion matrix for 3-fold cross-validation on ResNet mixed convolution.

Confusion matrix for 3-fold cross-validation on ResNet3D18.

Figure  7 shows the class-wise performance of the different models, both with and without pre-training, using precision, recall, specificity, and F1-score.

Heatmaps showing the class-wise performance of the classifiers, compared using precision, recall, specificity, and F1-score: ( a ) LGG, ( b ) HGG, and ( c ) healthy.

Comparison of the models

The mean F1-score over 3-fold cross-validation was used as the metric to compare the performance of the different models. Tables 2 , 3 and 4 show the results of the different models for the classes LGG, HGG, and Heathy, respectively; and finally Table  5 shows the consolidated scores.

For low-grade glioma (LGG), ResNet Mixed Convolution with pre-training achieved the highest F1 score of 0.8949 with a standard deviation of 0.033. The pre-trained ResNet(2+1)D is not far behind, with 0.8739 \({\pm }\) 0.033.

For the high-grade glioma (HGG) class, the highest F1 was achieved by the pre-trained ResNet Mixed Convolution model, with an F1 score of 0.9123 \({\pm }\) 0.029. This is higher than the best model’s F1 score for the class LGG. This can be expected because of the class imbalance between LGG and HGG. As with low-grade glioma, the second-best model for HGG is also the Pre-trained ResNet(2+1)D with the F1 score of 0.8979 \({\pm }\) 0.032.

The healthy brain class achieved the highest F1 score of 0.9998 \({\pm }\) 0.0002, with the pre-trained ResNet 3D model, which can be expected because of the complete absence of any lesion in the MR images making it far less challenging for the model to learn and distinguish it from the brain MRIs with pathology. Even though the pre-trained ResNet 3D model achieved the highest mean F1 score, all pre-trained models achieved similar F1 scores, i.e. all the mean scores are more than 0.9960—making it difficult to choose a clear winner.

ResNet Mixed Convolution with pre-training came up as the best model for both classes with pathology (LGG and HGG) and achieved a similar score as the other models while classifying healthy brain MRIs, as well as based on macro and weighted F1 scores - making this model as the clear overall winner. It can also be observed that the spatiospatial models performed better with pre-training, but ResNet 3D performed better without pre-training.

Comparison against literature

This sub-section compares the best model from the previous sub-section (i.e. ResNet Mixed Convolution with pre-training) against seven other research papers (in no specific order), where they classified LGG and HGG tumours. Mean test accuracy was used as the metric to compare the results as that was the common metric used in those papers.

Starting from Shahzadi et al. 31 , where they used LSTM-CNN to classify between HGG and LGG, using T2-FLAIR images from the BraTS 2015 dataset. Their work focuses on using a smaller sample size, and they were able to achieve 84.00% accuracy 31 . Pei et al. 9 achieved even less classification accuracy of 74.9% although they did use all of the available contrasts of the BraTS dataset, and their method performed segmentation using a U-Net like model before performing classification. Ge et al. 10 uses a novel method of fusing the contrasts into multiple streams to be trained simultaneously. Their model achieved an accuracy of 90.87% overall on all the contrasts, and they achieved 83.73% on T1ce. Mzoughi et al. 8 achieved 96.59% using deep convolutional neural networks on T1ce images. Their work does not present any other metric for their results, except for the overall accuracy of their model, which makes it difficult to compare against their results. Next, Yang et al. 27 did similar work; they used pre-trained GoogLeNet on 2D images, achieving an overall accuracy of 94.5%. They did not use the BraTS dataset, but the purpose of their work was similar - to classify glioma tumours based on LGG and HGG grading. Their dataset had fewer samples of LGG and HGG class in comparison to this research, with the former having 52 samples, and later 61 samples 27 . Ouerghi et al. 11 used different machine learning methods in their paper to train on the fusion images, one of which is the random forest, on which they achieved 96.5% for classification between High-Grade and Low-Grade Glioma. Finally, the Zhuge et al. 32 achieved an impressive 97.1% using Deep CNN for classification of glioma based on LGG and HGG grading, beating the proposed model by 0.12%. This difference can be explained by two factors, 1) their use of an additional dataset from The Cancer Imaging Archive (TCIA) in combination with BraTS 2018 2) and their use of four different contrasts - both these factors increase the size of the training set significantly. Furthermore, no cross-validation has been reported in their paper. Table  6 shows the complete comparative results.

The F1 scores of all the models in classifying healthy brains were very close to one, as segregating healthy brains from brains with pathology is comparatively a simpler task than classifying the grade of the tumour. Furthermore, using two different datasets for healthy and pathological brain, MRIs could have also introduced a dataset bias. In classifying the grade of the tumour, the pre-trained ResNet Mixed Convolution model performed best, while in classifying healthy brains, all the three pre-trained models performed similarly. For comparing the models based on consolidated scores, macro and weighted F1 scores were used. However, the macro F1 score is to be given more importance as the dataset was imbalanced. Both of the metrics declared the pre-trained ResNet Mixed Convolution as the clear winner.

One interesting observation that can be made from the confusion matrices is that the classification performance of the models for the LGG class has been lower than the other two classes. Even the best performing model managed to get an accuracy of 81% for LGG while achieving 96% for HGG and nearly perfect results for healthy. This might be attributed to the fact that the dataset was highly imbalanced (“ Dataset ” section), i.e. 259 volumes each for HGG and healthy, while having 73 volumes for LGG. Even though weighted cross-entropy loss (“ Weighted cross-entropy loss ” section) was used in this research to deal with the problem of class imbalance, increasing the number of LGG samples or employing further techniques to deal with this problem further and might improve the performance of the models for LGG 33 .

It is noteworthy that the pre-trained ResNet Mixed Convolution resulted in the best classification performance, even though it is the model with the least number of trainable parameters (see Table  1 ). Moreover, it is to be noted that both spatiospatial models performed better than the pure 3D ResNet18 model, even though they had a fewer number of trainable parameters than the 3D ResNet18. A fewer number of trainable parameters can reduce the computational costs, as well as the chance of overfitting. The authors hypothesise that the increased non-linearity due to the additional activation functions between the 2D and 1D convolutions in (2+1)D convolutional layers helped the ResNet (2+1)D model to achieve better results than ResNet3D, and the reduction of trainable parameters while having a similar number of layers, in turn preserving the level of non-linearity, contributed to the success of ResNet Mixed Convolution. Even though it has been seen that the spatiospatial models performed better, it is worthy of mention that the spatiospatial models do not adequately maintain the 3D nature of the data—the spatial relationship between the three dimensions is not preserved within the network like a fully 3D network as ResNet3D—which is a limitation of this architecture, which might have some unforeseen adverse effects. The authors hypothesised that this relationship was indirectly maintained through the channels of the network, and the network could learn the general representation to be able to classify appropriately. The experiments have also shown that the spatiospatial models are superior to a fully 3D model for the brain tumour classification problem shown here. Nevertheless, before creating a common consensus about this finding, these models should be further evaluated for other tasks.

In this research, the slice dimension in the axial orientation was considered as the “specially-treated” spatial dimension of the spatiospatial models, which can also be seen as the pseudo-temporal dimension of the spatiotemporal models. The authors hypothesise that using the data in sagittal or coronal orientation in a similar way might also be possible to exploit the advantages of such models, which it is yet to be tested.

It can also be observed that the pre-trained models were the winners for all three different classes. However, the effect of pre-training was not the same on all three models. For both the spatiospatial models, pre-training improved the model’s performance, but in different amounts: 2.24% improvement for ResNet (2+1)D and 8.57% for ResNet Mixed Convolution (based on macro F1 scores). However, pre-training had a negative impact on the 3D ResNet18 model (for two out of three classes), causing it to decrease the macro F1 score by 1.87%. Average macro F1 scores for all the models with and without pre-training (0.9169 with pre-training, 0.8912 without pre-training) show that the pre-training resulted in an overall improvement of 2.88% across models. It is noteworthy that the pre-trained networks were initially trained on RGB videos. Pre-training them on MRI volumes or MR videos (dynamic MRIs) might further improve the performance of the models.

Regarding the comparisons to other published works, an interesting point to note is that the previous papers only classified different grades of brain tumours (LGG and HGG), whereas this paper also classified healthy brains as an additional class. Thus, the results are not fully comparable as more classes increase the difficulty of the task. Even then, the results obtained by the winning model are better than all previously published methods, except for one, which reported comparable results to the ResNet Mixed Convolution (that paper reported 0.12% better accuracy, and 0.41% less specificity). However, this paper used four different contrasts and an additional dataset apart from BraTS, making them have a larger dataset for training.

This paper shows that the spatiotemporal models, ResNet(2+1)D and ResNet Mixed Convolution, working as spatiospatial models, could improve the classification of grades of brain tumours (i.e. low-grade and high-grade glioma), as well as classifying brain images with and without tumours, while reducing the computational costs. A 3D ResNet18 model was used to compare the performance of the spatiospatial models against a pure 3D convolution model. Each of the three models was trained from scratch and also trained using weights from pre-trained models that were trained on an action recognition dataset—to compare the effectiveness of pre-training in this setup. The final results were generated using cross-validation with three folds. It was observed that the spatiospatial models performed better than a pure 3D convolutional ResNet18 model, even though having fewer trainable parameters. It can be observed further that pre-training improved the performance of the models. Overall, the pre-trained ResNet Mixed Convolution model was observed to be the best model in terms of F1-score, obtaining a macro F1-score of 0.9345 and a mean test accuracy of 96.98%, while achieving 0.8949 and 0.9123 F1-scores for low-grade glioma and high-grade glioma, respectively. This study shows that the spatiospatial models have the potential to outperform a fully 3D convolutional model. However, this was only shown for a specific task here—brain tumour classification, using one dataset—BraTS. These models should be compared for other tasks in the future to build a common consensus regarding the spatiospatial models. One limitation of this study is that it only used T1 contrast-enhanced images for classifying the tumours, which already resulted in good accuracy. Incorporating all four available types of images (T1, T1ce, T2, T2-Flair) or any combination of them might improve the performance of the model even further.

Fritz, A. et al. International Classification of Diseases for Oncology Vol. 3 (World Health Organization, Geneva, 2001).

Google Scholar  

Goodenberger, M. L. et al. Genetics of adult glioma. Cancer Genet. 205 , 613–621 (2012).

Article   CAS   Google Scholar  

Claus, E. B. et al. Survival and low-grade glioma: The emergence of genetic information. Neurosurg. Focus 38 , E6 (2015).

Article   Google Scholar  

Raza, S. M. et al. Necrosis and glioblastoma: A friend or a foe? A review and a hypothesis. Neurosurgery 51 , 2–13 (2002).

Engelhorn, T. et al. Cellular characterization of the peritumoral edema zone in malignant brain tumors. Cancer Sci. 100 , 1856–1862 (2009).

Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. 34 , 1993–2024 (2014).

Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLoS Med. 15 , e1002686 (2018).

Mzoughi, H. et al. Deep multi-scale 3d convolutional neural network (cnn) for mri gliomas brain tumor classification. J. Digit. Imaging 33 , 903–915 (2020).

Pei, L. et al. Brain tumor classification using 3d convolutional neural network. In International MICCAI Brain lesion Workshop , 335–342 (2019).

Ge, C. et al. Deep learning and multi-sensor fusion for glioma classification using multistream 2d convolutional networks. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , 5894–5897 (2018).

Ouerghi, H. et al. Glioma classification via mr images radiomics analysis. Vis. Comput. 2021 , 1–15 (2021).

He, K. et al. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 770–778 (2016).

Tran, D. et al. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 6450–6459 (2018).

Torrey, L. et al. Transfer learning. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques , 242–264 (2010).

Zhuang, F. et al. A comprehensive survey on transfer learning. Proc. IEEE 109 , 43–76 (2020).

Sarasaen, C. et al. Fine-tuning deep learning model parameters for improved super-resolution of dynamic mri with prior-knowledge. Artif. Intell. Med. 121 , 102196 (2021).

Pallud, J. et al. Quantitative morphological magnetic resonance imaging follow-up of low-grade glioma: A plea for systematic measurement of growth rates. Neurosurgery 71 , 729–740 (2012).

Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32 , 8026–8037 (2019).

Torchvision models. https://pytorch.org/vision/stable/models.html#video-classification . Accessed on 15th December 2021.

Kinetics-400 dataset. https://deepmind.com/research/open-source/kinetics . Accessed on 15th December 2021.

Micikevicius, P. et al. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017).

Nvidia apex. https://github.com/NVIDIA/apex . Accessed on 15th December 2021.

Pérez-García, F. et al. Torchio: A python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. Comput. Methods Programs Biomed. 2021 , 106236 (2021).

Bakas, S. et al. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Sci. data 4 , 1–13 (2017).

Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

Ixi dataset. https://brain-development.org/ixi-dataset . Accessed on 15th December 2021.

Yang, Y. et al. Glioma grading on conventional mr images: A deep learning study with transfer learning. Front. Neurosci. 12 , 804 (2018).

Smith, S. M. et al. Advances in functional and structural mr image analysis and implementation as fsl. Neuroimage 23 , S208–S219 (2004).

Jenkinson, M. et al. Smith sm. FSL Neuroimage 62 , 782–90 (2012).

Isensee, F. et al. nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv:1809.10486 (2018).

Shahzadi, I. et al. Cnn-lstm: Cascaded framework for brain tumour classification. In 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES) , 633–637 (IEEE, 2018).

Zhuge, Y. et al. Automated glioma grading on conventional mri images using deep convolutional neural networks. Med. Phys. 47 , 3044–3053 (2020).

Johnson, J. M. et al. Survey on deep learning with class imbalance. J. Big Data 6 , 1–54 (2019).

Download references

Acknowledgements

This work was in part conducted within the context of the International Graduate School MEMoRIAL at Otto von Guericke University (OVGU) Magdeburg, Germany, kindly supported by the European Structural and Investment Funds (ESF) under the programme “Sachsen-Anhalt WISSENSCHAFT Internationalisierung” (Project No. ZS/2016/08/80646).

Open Access funding enabled and organized by Projekt DEAL.

Author information

These authors contributed equally: Soumick Chatterjee and Faraz Ahmed Nizamani.

Authors and Affiliations

Biomedical Magnetic Resonance, Otto von Guericke University Magdeburg, Magdeburg, Germany

Soumick Chatterjee & Oliver Speck

Data and Knowledge Engineering Group, Otto von Guericke University Magdeburg, Magdeburg, Germany

Soumick Chatterjee & Andreas Nürnberger

Faculty of Computer Science, Otto von Guericke University, Magdeburg, Germany

Institute for Medical Engineering, Otto von Guericke University Magdeburg, Magdeburg, Germany

Faraz Ahmed Nizamani

Center for Behavioral Brain Sciences, Magdeburg, Germany

Andreas Nürnberger & Oliver Speck

German Center for Neurodegenerative Disease, Magdeburg, Germany

Oliver Speck

Leibniz Institute for Neurobiology, Magdeburg, Germany

You can also search for this author in PubMed   Google Scholar

Contributions

S.C. developed the idea and wrote the manuscript. F.N. Implemented the idea and performed the experiments. A.N. and O.S. created the concept and design the work, and finally reviewed the manuscript.

Corresponding author

Correspondence to Soumick Chatterjee .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Chatterjee, S., Nizamani, F.A., Nürnberger, A. et al. Classification of brain tumours in MR images using deep spatiospatial models. Sci Rep 12 , 1505 (2022). https://doi.org/10.1038/s41598-022-05572-6

Download citation

Received : 28 May 2021

Accepted : 14 January 2022

Published : 27 January 2022

DOI : https://doi.org/10.1038/s41598-022-05572-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain mri.

• Ramy A. Zeineldin
• Mohamed E. Karar
• Franziska Mathis-Ullrich

Scientific Reports (2024)

BrainNet: a fusion assisted novel optimal framework of residual blocks and stacked autoencoders for multimodal brain tumor classification

• Muhammad Sami Ullah
• Muhammad Attique Khan
• Mohammad Shabaz

Deep learning for multi-grade brain tumor detection and classification: a prospective survey

• K. Bhagyalaxmi
• B. Dwarakanath
• P. Vijaya Pal Reddy

Multimedia Tools and Applications (2024)

The impact of image augmentation techniques of MRI patients in deep transfer learning networks for brain tumor detection

• Peshraw Ahmed Abdalla
• Bashdar Abdalrahman Mohammed
• Ari M. Saeed

Journal of Electrical Systems and Information Technology (2023)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Help | Advanced Search

Electrical Engineering and Systems Science > Image and Video Processing

Title: a novel framework for brain tumor detection based on convolutional variational generative models.

Abstract: Brain tumor detection can make the difference between life and death. Recently, deep learning-based brain tumor detection techniques have gained attention due to their higher performance. However, obtaining the expected performance of such deep learning-based systems requires large amounts of classified images to train the deep models. Obtaining such data is usually boring, time-consuming, and can easily be exposed to human mistakes which hinder the utilization of such deep learning approaches. This paper introduces a novel framework for brain tumor detection and classification. The basic idea is to generate a large synthetic MRI images dataset that reflects the typical pattern of the brain MRI images from a small class-unbalanced collected dataset. The resulted dataset is then used for training a deep model for detection and classification. Specifically, we employ two types of deep models. The first model is a generative model to capture the distribution of the important features in a set of small class-unbalanced brain MRI images. Then by using this distribution, the generative model can synthesize any number of brain MRI images for each class. Hence, the system can automatically convert a small unbalanced dataset to a larger balanced one. The second model is the classifier that is trained using the large balanced dataset to detect brain tumors in MRI images. The proposed framework acquires an overall detection accuracy of 96.88% which highlights the promise of the proposed framework as an accurate low-overhead brain tumor detection system.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Diagnostics (Basel)

Convolutional Neural Network Techniques for Brain Tumor Classification (from 2015 to 2022): Review, Challenges, and Future Perspectives

1 Department of Biomedical and Neuromotor Sciences, University of Bologna, 40126 Bologna, Italy; [email protected] (Y.X.); [email protected] (F.Z.); [email protected] (R.L.); [email protected] (C.T.)

Fulvio Zaccagna

2 Functional and Molecular Neuroimaging Unit, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy; [email protected]

Leonardo Rundo

3 Department of Information and Electrical Engineering and Applied Mathematics, University of Salerno, 84084 Fisciano, Italy; ti.asinu@odnurl

Claudia Testa

4 Department of Physics and Astronomy, University of Bologna, 40127 Bologna, Italy

Raffaele Agati

5 Programma Neuroradiologia con Tecniche ad elevata complessità, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy; [email protected]

Raffaele Lodi

6 IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy

David Neil Manners

Caterina tonon, associated data.

Not applicable.

Convolutional neural networks (CNNs) constitute a widely used deep learning approach that has frequently been applied to the problem of brain tumor diagnosis. Such techniques still face some critical challenges in moving towards clinic application. The main objective of this work is to present a comprehensive review of studies using CNN architectures to classify brain tumors using MR images with the aim of identifying useful strategies for and possible impediments in the development of this technology. Relevant articles were identified using a predefined, systematic procedure. For each article, data were extracted regarding training data, target problems, the network architecture, validation methods, and the reported quantitative performance criteria. The clinical relevance of the studies was then evaluated to identify limitations by considering the merits of convolutional neural networks and the remaining challenges that need to be solved to promote the clinical application and development of CNN algorithms. Finally, possible directions for future research are discussed for researchers in the biomedical and machine learning communities. A total of 83 studies were identified and reviewed. They differed in terms of the precise classification problem targeted and the strategies used to construct and train the chosen CNN. Consequently, the reported performance varied widely, with accuracies of 91.63–100% in differentiating meningiomas, gliomas, and pituitary tumors (26 articles) and of 60.0–99.46% in distinguishing low-grade from high-grade gliomas (13 articles). The review provides a survey of the state of the art in CNN-based deep learning methods for brain tumor classification. Many networks demonstrated good performance, and it is not evident that any specific methodological choice greatly outperforms the alternatives, especially given the inconsistencies in the reporting of validation methods, performance metrics, and training data encountered. Few studies have focused on clinical usability.

1. Introduction

Brain tumors are a heterogenous group of common intracranial tumors that cause significant mortality and morbidity [ 1 , 2 ]. Malignant brain tumors are among the most aggressive and deadly neoplasms in people of all ages, with mortality rates of 5.4/100,000 men and 3.6/100,000 women per year being reported between 2014 and 2018 [ 3 ]. According to the 2021 World Health Organization (WHO) Classification of Tumors of the Central Nervous System, brain tumors are classified into four grades (I to IV) of increasingly aggressive malignancy and worsening prognosis. Indeed, in clinical practice, tumor type and grade influence treatment choice. Within WHO Grade IV tumors, glioblastoma is the most aggressive primary brain tumor, with a median survival after diagnosis of just 12–15 months [ 4 ].

The pathological assessment of tissue samples is the reference standard for tumor diagnosis and grading. However, a non-invasive tool capable of accurately classifying tumor type and of inferring grade would be highly desirable [ 5 ]. Although there are several non-invasive imaging modalities that can visualize brain tumors, i.e., Computed Tomography (CT), Positron Emission Tomography (PET), and Magnetic Resonance Imaging (MRI), the last of these remains the standard of care in clinical practice [ 6 ]. MRI conveys information on the lesion location, size, extent, features, relationship with the surrounding structures, and associated mass effect [ 6 ]. Beyond structural information, MRI can also assess microstructural features such as lesion cellularity [ 7 ], microvascular architecture [ 8 ], and perfusion [ 9 ]. Advanced imaging techniques may demonstrate many aspects of tumor heterogeneity related to type, aggressiveness, and grade; however, they are limited in assessing the mesoscopic changes that predate macroscopic ones [ 10 ]. Many molecular imaging techniques have recently been developed to better reveal and quantify heterogeneity, permitting a more accurate characterization of brain tumors. However, in order to make use of this wealth of new information, more sophisticated and potentially partially automated tools for image analysis may be useful [ 10 ].

Computer-aided detection and diagnosis (CADe and CADx, respectively), which refer to software that combines artificial intelligence and computer vision to analyze radiological and pathology images, have been developed to help radiologists diagnose human disease in several body districts, including in applications for colorectal polyp detection and segmentation [ 11 , 12 ] and lung cancer classification [ 13 , 14 , 15 ].

Machine learning has vigorously accelerated the development of CAD systems [ 16 ]. One of the most recent applications of machine learning in CAD is classifying objects of interest, such as lesions, into specific classes based on input features [ 17 , 18 , 19 , 20 ]. In machine learning, various image analysis tasks can be performed by finding or learning informative features that successfully describe the regularities or patterns in data. However, conventionally, meaningful or task-relevant features are mainly designed by human experts based on their knowledge of the target domain, making it challenging for those without domain expertise to leverage machine learning techniques. Furthermore, traditional machine learning methods can only detect superficial linear relationships, while the biology underpinning living organisms is several orders of magnitude more complex [ 21 ].

Deep learning [ 22 ], which is inspired by an understanding of the neural networks within the human brain, has achieved unprecedented success in facing the challenges mentioned above by incorporating the feature extraction and selection steps into the training process [ 23 ]. Generically, deep learning models are represented by a series of layers, and each is formed by a weighted sum of elements in the previous layer. The first layer represents the data, and the last layer represents the output or solution. Multiple layers enable complicated mapping functions to be reproduced, allowing deep learning models to solve very challenging problems while typically needing less human intervention than traditional machine learning methods. Deep learning currently outperforms alternative machine learning approaches [ 24 ] and, for the past few years, has been widely used for a variety of tasks in medical image analysis [ 25 ].

A convolutional neural network (CNN) is a deep learning approach that has frequently been applied to medical imaging problems. It overcomes the limitations of previous deep learning approaches because its architecture allows it to automatically learn the features that are important for a problem using a training corpus of sufficient variety and quality [ 26 ]. Recently, CNNs have gained popularity for brain tumor classification due to their outstanding performance with very high accuracy in a research context [ 27 , 28 , 29 , 30 , 31 ].

Despite the growing interest in CNN-based CADx within the research community, translation into daily clinical practice has yet to be achieved due to obstacles such as the lack of an adequate amount of reliable data for training algorithms and imbalances within the datasets used for multi-class classification [ 32 , 33 ], among others. Several reviews [ 31 , 32 , 33 , 34 , 35 , 36 ] have been published in this regard, summarizing the classification methods and key achievements and pointing out some of the limitations in previous studies, but as of yet, none of them have focused on the deficiencies regarding clinical adoption or have attempted to determine the future research directions required to promote the application of deep learning models in clinical practice. For these reasons, the current review considers the key limitations and obstacles regarding the clinical applicability of studies in brain tumor classification using CNN algorithms and how to translate CNN-based CADx technology into better clinical decision making.

In this review, we explore the current studies on using CNN-based deep learning techniques for brain tumor classification published between 2015 and 2022. We decided to focus on CNN architectures, as alternative deep-learning techniques, such as Deep Belief Networks or Restricted Boltzmann Machines, are much less represented in the current literature.

The objectives of the review were three-fold: to (1) review and analyze article characteristics and the impact of CNN methods applied to MRI for glioma classification, (2) explore the limitations of current research and the gaps in bench-to-bedside translation, and (3) find directions for future research in this field. This review was designed to answer the following research questions: How has deep learning been applied to process MR images for glioma classification? What level of impact have papers in this field achieved? How can the translational gap be bridged to deploy deep learning algorithms in clinical practice?

The review is organized as follows: Section 2 introduces the methods used to search and select literature related to the focus of the review. Section 3 presents the general steps of CNN-based deep learning methods for brain tumor classification, and Section 4 introduces relevant primary studies, with an overview of their datasets, preprocessing techniques, and computational methods for brain tumor classification, and presents a quantitative analysis of the covered studies. Furthermore, we introduce the factors that may directly or indirectly degrade the performance and the clinical applicability of CNN-based CADx systems and provide an overview of the included studies with reference to the degrading factors. Section 5 presents a comparison between the selected studies and suggests directions for further improvements, and finally, Section 6 summarizes the work and findings of this study.

2. Materials and Methods

2.1. article identification.

In this review, we identified preliminary sources using two online databases, PubMed and Scopus. The search queries used to interrogate each database are described in Table 1 . The filter option for the publication year (2015–2022) was selected so that only papers in the chosen period were fed into the screening process ( Supplementary Materials ). Searches were conducted on 30 June 2022. PubMed generated 212 results, and Scopus yielded 328 results.

The search queries used to interrogate the PubMed and Scopus databases.

2.2. Article Selection

Articles were selected for final review using a three-stage screening process ( Supplementary Materials ) based on a series of inclusion and exclusion criteria. After removing duplicate records that were generated from using two databases, articles were first screened based on the title alone. The abstract was then assessed, and finally, the full articles were checked to confirm eligibility. The entire screening process ( Supplementary Materials ) was conducted by one author (Y.T.X). In cases of doubt, records were reviewed by other authors (D.N.M, C.T), and the decision regarding inclusion was arrived at by consensus.

The meet the inclusion criteria, articles had to:

• Be original research articles published in a peer-reviewed journal with full-text access offered by the University of Bologna;
• Involve the use of any kind of MR images;
• Be published in English;
• Be concerned with the application of CNN deep learning techniques for brain tumor classification.

Included articles were limited to those published from 2015 to 2022 to focus on deep learning methodologies. Here, a study was defined as work that employed a CNN-based deep learning algorithm to classify brain tumors and that involved the use of one or more of the following performance metrics: accuracy, the area under the receiver operating characteristics curve, sensitivity, specificity, or F 1 score.

Exclusion criteria were:

• Review articles;
• Book or book chapters;
• Conference papers or abstracts;
• Short communications or case reports;
• Unclear descriptions of data;
• No validation performed.

If a study involved the use of a CNN model for feature extraction but traditional machine learning techniques for the classification task, it was excluded. Studies that used other deep learning networks, for example, artificial neural networks (ANNs), generative adversarial networks (GANs), or autoencoders (AEs), instead of CNN models were excluded. Studies using multiple deep learning techniques as well as CNNs were included in this study, but only the performance of the CNNs will be reviewed.

Figure 1 reports the numbers of articles screened after exclusion at each stage as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 37 ]. A review of 83 selected papers is presented in this paper. All of the articles cover the classification of brain tumors using CNN-based deep learning techniques.

The PRISMA flowchart of this review. n : number of articles.

3. Literature Review

This section presents a detailed overview of the research papers dealing with brain tumor classification using CNN-based deep learning techniques published during the period from 2015 to 2022. This section is formulated as follows: Section 3.1 presents a brief overview of the general methodology adopted in the majority of the papers for the classification of brain MRI images using CNN algorithms. Section 3.2 presents a description of the popular publicly available datasets that have been used in the research papers reviewed in the form of a table. Section 3.3 introduces the commonly applied preprocessing methods used in the reviewed studies. Section 3.4 provides an introduction of widely used data augmentation methods. Finally, Section 3.5 provides a brief overview of the performance metrics that provide evidence about the credibility of a specific classification algorithm model.

3.1. Basic Architecture of CNN-Based Methods

Recently, deep learning has shown outstanding performance in medical image analysis, especially in brain tumor classification. Deep learning networks have achieved higher accuracy than classical machine learning approaches [ 24 ]. In deep learning, CNNs have achieved significant recognition for their capacity to automatically extract deep features by adapting to small changes in the images [ 26 ]. Deep features are those that are derived from other features that are relevant to the final model output.

The architecture of a typical deep CNN-based brain tumor classification frame is described in Figure 2 . To train a CNN-based deep learning model with tens of thousands of parameters, a general rule of thumb is to have at least about 10 times the number of samples as parameters in the network for the effective generalization of the problem [ 38 ]. Overfitting may occur during the training process if the training dataset is not sufficiently large [ 39 ]. Therefore, many studies [ 40 , 41 , 42 , 43 , 44 ] use 2D brain image slices extracted from 3D brain MRI volumes to solve this problem, which increases the number of examples within the initial dataset and mitigates the class imbalance problem. In addition, it has the advantage of reducing the input data dimension and reducing the computational burden of training the network.

The basic workflow of a typical CNN-based brain tumor classification study with four high-level steps: Step 1. Input Image: 2D or 3D Brain MR samples are fed into the classification model; Step 2. Preprocessing: several preprocessing techniques are used to remove the skull, normalize the images, resize the images, and augment the number of training examples; Step 3. CNN Classification: the preprocessed dataset is propagated into the CNN model and is involved in training, validation, and testing processes; Step 4. Performance Evaluation: evaluation of the classification performance of a CNN algorithm with accuracy, specificity, F 1 score, area under the curve, and sensitivity metrics.

Data augmentation is another effective technique for increasing both the amount and the diversity of the training data by adding modified copies of existing data with commonly used morphological techniques, such as rotation, reflection (also referred to as flipping or mirroring), scaling, translation, and cropping [ 44 , 45 ]. Such strategies are based on the assumption that the size and orientation of image patches do not yield robust features for tumor classification.

In deep learning, overfitting is also a common problem that occurs when the learning capacity is so large that the network will learn spurious features instead of meaningful patterns [ 39 ]. A validation set can be used in the training process to avoid overfitting and to obtain the stable performance of the brain tumor classification system on future unseen data in clinical practice. The validation set provides an unbiased evaluation of a classification model using multiple subsets of the training dataset while tuning the model’s hyperparameters during the training process [ 46 ]. In addition, validation datasets can be used for regularization by early stopping when the error on the validation dataset increases, which is a sign of overfitting to the training data [ 39 , 47 ]. Therefore, in the article selection process, we excluded the articles that omitted validation during the training process.

Evaluating the classification performance of a CNN algorithm is an essential part of a research study. The accuracy, specificity, F 1 score (also known as the Dice similarity coefficient) [ 48 ], the area under the curve, and sensitivity are important metrics to assess the classification model’s performance and to compare it to similar works in the field.

3.2. Datasets

A large training dataset is required to create an accurate and trustworthy deep learning-based classification system for brain tumor classification. In the current instance, this usually comprises a set of MR image volumes, and for each, a classification label is generated by a domain expert such as a neuroradiologist. In the reviewed literature, several datasets were used for brain tumor classification, targeting both binary tasks [ 27 , 40 , 41 , 45 ] and multiclass classification tasks [ 24 , 30 , 49 , 50 , 51 ]. Table 2 briefly lists some of the publicly accessible databases that have been used in the studies reviewed in this paper, including the MRI sequences as well as the size, classes, unbiased Gini Coefficient, and the web address of the online repository for the specific dataset.

An overview of publicly available datasets.

The Gini coefficient (G) [ 52 ] is a property of distribution that measures its difference using uniformity. It can be applied to categorical data in which classes are sorted by prevalence. Its minimum value is zero if all of the classes are equally represented, and its maximum values varies between 0.5 for a two-class distribution to an asymptote of 1 for many classes. The unbiased Gini coefficient divides G by the maximum value of the number of classes present and takes values in the range of 0–1. The maximum value for a distribution with n classes is (n − 1)/n. The values of the unbiased Gini coefficient were calculated using R package DescTools [ 52 ]. Table 2 shows the characteristics of public datasets in terms of balancing the samples of the available classes of tumors (unbiased Gini coefficient) while considering the total number of samples in the datasets (“Size” column).

Among the public datasets, the dataset from Figshare provided by Cheng [ 55 ] is the most popular dataset and has been widely used for brain tumor classification. BraTS, which refers to the Multimodal Brain Tumor Segmentation Challenge (a well-known challenge that has taken place every year since 2012), is another dataset that is often used for testing brain tumor classification methods. The provided data are pre-processed, co-registered to the same anatomical template, interpolated to the exact resolution (1 mm 3 ), and skull stripped [ 55 ].

Most MR techniques can generate high-resolution images, while different imaging techniques show distinct contrast, are sensitive to specific tissues or fluid regions, and highlight relevant metabolic or biophysical properties of brain tumors [ 64 ]. The datasets listed in Table 2 collect one or more MRI sequences, including T 1 -weighted (T 1 w), T 2 -weighted (T 2 w), contrast-enhanced T 1 -weighted (ceT 1 w), fluid-attenuated inversion recovery (FLAIR), diffusion-weighted imaging (DWI), and dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) sequences. Among these, the T 1 w, T 2 w, ceT 1 w, and FLAIR sequences are widely used for brain tumor classification in both research and in clinical practice. Each sequence is distinguished by a particular series of radiofrequency pulses and magnetic field gradients, resulting in images with a characteristic appearance [ 64 ]. Table 3 lists the imaging configurations and the main clinical distinctions of T 1 w, T 2 w, ceT 1 w, and FLAIR with information retrieved from [ 64 , 65 , 66 , 67 ].

The imaging configurations and main clinical distinctions of T 1 w, T 2 w, ceT 1 w, and FLAIR.

* Pictures from [ 68 ]. TR, repetition time. TE, echo time.

3.3. Preprocessing

Preprocessing is used mainly to remove extraneous variance from the input data and to simplify the model training task. Other steps, such as resizing, are needed to work around the limitations of neural network models.

3.3.1. Normalization

The dataset fed into CNN models may be collected with different clinical protocols and various scanners from multiple institutions. The dataset may consist of MR images with different intensities because the intensities of MR image are not consistent across different MR scanners [ 69 ]. In addition, the intensity values of MR images are sensitive to the acquisition condition [ 70 ]. Therefore, input data should be normalized to minimize the influence of differences between the scanners and scanning parameters. Otherwise, any CNN network that is created will be ill-conditioned.

There are many methods for data normalization, including min-max normalization, z-score normalization, and normalization by decimal scaling [ 71 ]. Min-max normalization is one of the most common ways to normalize MR images found in the included articles [ 27 , 36 , 40 ]. In that approach, the intensity values of the input MR images are rescaled into the range of (0, 1) or (−1, 1).

Z-score normalization refers to the process of normalizing every intensity value found in MR images such that the mean of all of the values is 0 and the standard deviation is 1 [ 71 ].

3.3.2. Skull Stripping

MRI images of the brain also normally contain non-brain regions such as the dura mater, skull, meninges, and scalp. Including these parts in the model typically deteriorates its performance during classification tasks. Therefore, in the studies on brain MRI datasets that retain regions of the skull and vertebral column, skull stripping is widely applied as a preprocessing step in brain tumor classification problems to improve performance [ 24 , 72 , 73 ].

3.3.3. Resizing

Since deep neural networks require inputs of a fixed size, all of the images need to be resized before being fed into CNN classification models [ 74 ]. Images larger than the required size can be downsized by either cropping the background pixels or by downscaling using interpolation [ 74 , 75 ].

3.3.4. Image Registration

Image registration is defined as a process that spatially transforms different images into one coordinate system. In brain tumor classification, it is often necessary to analyze multiple images of a patient to improve the treatment plan, but the images may be acquired from different scanners, at different times, and from different viewpoints [ 76 ]. Registration is necessary to be able to integrate the data obtained from these different measurements.

Rigid image registration is one of the most widely utilized registration methods in the reviewed studies [ 77 , 78 ]. Rigid registration means that the distance between any two points in an MR image remains unchanged before and after transformation. This approach only allows translation and rotation transformations.

3.3.5. Bias Field Correction

In medical images, the bias field is an undesirable artifact caused by factors such as the scan position and instrument used as well as by other unknown issues [ 79 ]. This artifact is characterized by differences in brightness across the image and can significantly degrade the performance of many medical image analysis techniques. Therefore, a preprocessing step is needed to correct the bias field signal before submitting corrupted MR images to a CNN classification model.

The N4 bias field correction algorithm and the Statistical Parametric Mapping (SPM) module are common approaches for correcting the inhomogeneity in the intensity of MR images. The N4 bias field correction algorithm is a popular method for correcting the low-frequency-intensity non-uniformity present in MR image data [ 80 ]. SPM contains several software packages that are used for brain segmentation. These packages usually contain a set for skull stripping, intensity non-uniformity (bias) correction, and segmentation routines [ 81 ].

3.4. Data Augmentation

CNN-based classification requires a large number of data. A general rule of thumb is to have at least about 10 times the number of samples set as parameters in the network for the effective generalization of the problem [ 38 ]. If the database is significantly smaller, overfitting might occur. Data augmentation is one of the foremost data techniques to subside imbalanced distribution and data scarcity problems. It has been used in many studies focusing brain tumor classification [ 24 , 45 , 49 , 50 ] and involves geometrical transformation operations such as rotation, reflection (also referred to as flipping or mirroring), scaling, translation, and cropping ( Figure 3 ).

Data augmentation: ( a ) original image; ( b ) 18° rotation. When rotating by an arbitrary number of degrees (non-modulo 90), rotation will result in the image being padded in each corner. Then, a crop is taken from the center of the newly rotated image to retain the largest crop possible while maintaining the image’s aspect ratio; ( c ) left–right flipping; ( d ) top–bottom flipping; ( e ) scaling by 1.5 times; ( f ) cropping by center cropping to the size 150 × 150; ( g ) random brightness enhancement; ( h ) random contrast enhancement.

Data augmentation techniques can be divided into two classes: position augmentation and color augmentation. Some of the most popular position augmentation methods include rotation, reflection (also referred to as flipping or mirroring), scaling, translation, and cropping, and they have been commonly used to enlarge MR datasets in studies focusing on brain tumor classification [ 45 , 51 , 72 , 77 ]. Color augmentation methods such as contrast enhancement and brightness enhancement have also been applied in the included studies [ 28 , 43 ].

Recently, well-established data augmentation techniques have begun to be supplemented by automatic methods that use deep learning approaches. For example, the authors in [ 44 ] proposed a progressively growing generative adversarial network (PGGAN) augmentation model to help overcome the shortage of images needed for CNN classification models. However, such methods are rare in the literature reviewed.

3.5. Performance Measures

Evaluating the classification performance of a CNN algorithm is an essential part of a research study. Here, we outline the evaluation metrics that are the most commonly encountered in the brain tumor classification literature, namely accuracy, precision, sensitivity, F1 score, and the area under the curve.

In classification tasks, true positive ( TP ) represents an image that is correctly classified into the positive class according to the ground truth. Similarly, true negative is an outcome in which the model correctly classifies an imagine into the negative class. On the other hand, false positive ( FP ) is an outcome in which the model incorrectly classifies an image into the positive class when the ground truth is negative. False negative ( FN ) is an outcome in which the model incorrectly classifies an image that should be placed in the positive class.

3.5.1. Accuracy

Accuracy ( ACC ) is a metric that measures the performance of a model in correctly classifying the classes in a given dataset and is given as the percentage of total correct classifications divided by the total number of images.

3.5.2. Specificity

Specificity ( SPE ) represents the proportion of correctly classified negative samples to all of the negative samples identified in the data.

3.5.3. Precision

Precision ( PRE ) represents the ratio of true positives to all of the identified positives.

3.5.4. Sensitivity

Sensitivity ( SEN ) measures the ability of a classification model to identify positive samples. It represents the ratio of true positives to the total number of (actual) positives in the data.

3.5.5. F 1 Score

The F 1 score [ 48 ] is one of the most popular metrics and considers both precision and recall. It can be used to assess the performance of classification models with class imbalance problems [ 82 ] and considers the number of prediction errors that a model makes and looks at the type of errors that are made. It is higher if there is a balance between PRE and SEN .

3.5.6. Area under the Curve

The area under the curve (AUC) measures the entire two-dimensional area underneath the ROC curve from (0, 0) to (1, 1). It measures the ability of a classifier to distinguish between classes.

Clinicians and software developers need to understand how performance metrics can measure the properties of CNN models for different medical problems. In research studies, several metrics are typically used to evaluate a model’s performance.

Accuracy is among the most commonly used metric to evaluate a classification model but is also known for being misleading in cases when the classes have different distributions in the data [ 83 , 84 ]. Precision is an important metric in cases when the occurrence of false positives is unacceptable/intolerable [ 84 ]. Specificity measures the ability of a model to correctly identify people without the disease in question. Sensitivity, also known as recall, is an important metric in cases where identifying the number of positives is crucial and when the occurrence of false negatives is unacceptable/intolerable [ 83 , 84 ]. It must be interpreted with care in cases with strongly imbalanced classes.

It is important to recognize that there is always a tradeoff between sensitivity and specificity. Balancing between two metrics has to be based on the medical use case and the associated requirements [ 83 ]. Precision and sensitivity are both proportional to TP but have an inverse relationship. Whether to maximize recall or precision depends on the application: Is it more important to only identify relevant instances, or to make sure that all relevant instances are identified? The balance between precision and sensitivity has to be considered in medical use cases in which some false positives are tolerable; for example, in cancer detection, it is crucial to identify all positive cases. On the other hand, for a less severe disease with high prevalence, it is important to achieve the highest possible precision [ 83 ].

This section provides an overview of the research papers focusing on brain tumor classification using CNN techniques. Section 4.1 presents a quantitative analysis of the number of articles published from 2015 to 2022 on deep learning and CNN in brain tumor classification and the usage of the different CNN algorithms applied in the studies covered. Then, Section 4.2 introduces the factors that may directly or indirectly degrade the performance and the clinical applicability of CNN-based CADx systems. Finally, in Section 4.3 , an overview of the included studies will be provided with reference to the degrading factors introduced in Section 4.2 .

4.1. Quantitative Analysis

As mentioned in the introduction, many CNN models have been used to classify the MR images of brain tumor patients. They overcome the limitations of earlier deep learning approaches and have gained popularity among researchers for brain tumor classification tasks. Figure 4 shows the number of research articles on brain tumor classification using deep learning methods and CNN-based deep learning techniques published on PubMed and Scopus in the years from 2015 to June 2022; the number of papers related to brain tumor classification using CNN techniques grows rapidly from 2019 onwards and accounts for the majority of the total number of studies published in 2020, 2021, and 2022. This is because of the high generalizability, stability, and accuracy rate of CNN algorithms.

Number of articles published from 2015 to 2022.

Figure 5 shows the usage of the most commonly used preprocessing techniques for addressing problems in brain tumor classification, including data augmentation, normalization, resizing, skull stripping, bias field correction, and registration. In this figure, only data from 2017 to 2022 are visualized, as no articles using the preprocessing methods mentioned were published in 2015 or 2016. Since 2020, data augmentation has been used in the majority of studies to ease data scarcity and overfitting problems. However, the bias field problem has yet to be taken seriously, and few studies have included bias field correction in the preprocessing process.

Usage of preprocessing techniques from 2017 to 2022.

Figure 6 breaks down the usage of the publicly available CNN architectures used in the articles included in this review, including custom CNN models, VGG, AlexNet, ResNet, GoogLeNet, DenseNet, and EfficientNet.

Usage of state-of-the-art CNN models from 2015 and 2022.

AlexNet [ 85 ] came out in 2012 and was a revolutionary advancement in deep learning; it improved traditional CNNs by introducing a composition of consecutively stacked convolutional layers and became one of the best models for image classification. VGG, which refers to the Visual Geometry Group, was a breakthrough in the world of convolutional neural networks after AlexNet. It is a type of deep CNN architecture with multiple layers that was originally proposed by K. Simonyan and A. Zisserman in [ 86 ] and was developed to improve model performance by increasing the depth of such CNNs.

GoogLeNet is a deep convolutional neural network with 22 layers based on the Inception architecture; it was developed by researchers at Google [ 87 ]. GoogLeNet addresses most of the problems that large networks face, such as computational expense and overfitting, by employing the Inception module. This module can use max pooling and three varied sizes of filters (1 × 1, 3 × 3, 5 × 5) for convolution in a single image block; such blocks are then concatenated and passed onto the next layer. An extra 1 × 1 convolution can be added to the neural network before the 3 × 3 and 5 × 5 layers to make the process even less computationally expensive [ 87 ]. ResNet stands for Deep Residual Network. It is an innovative convolutional neural network that was originally proposed in [ 88 ]. ResNet makes use of residual blocks to improve the accuracy of models. A residual block is a skip-connection block that typically has double- or triple-layer skips that contain nonlinearities (ReLU) and batch normalization in between; it can help to reduce the problem of vanishing gradients or can help to mitigate accuracy saturation problems [ 88 ]. DenseNet, which stands for Dense Convolutional Network, is a type of convolutional neural network that utilizes dense connections between layers. DenseNet was mainly developed to improve the decreased accuracy caused by the vanishing gradient in neural networks [ 89 ]. Additionally, those CNNs take in images with a pixel resolution of 224 × 224. Therefore, for brain tumor classification, the authors need to center crop a 224 × 224 patch in each image to keep the input image size consistent.

Convolutional neural networks are commonly built using a fixed resource budget. When more resources are available, the depth, width, and resolution of the model need to be scaled up for better accuracy and efficiency [ 90 ]. Unlike previous CNNs, EfficientNet is a novel baseline network that uses a different model-scaling technique based on a compound coefficient and neural architecture search methods that can carefully balance network depth, width, and resolution [ 90 ].

4.2. Clinical Applicability Degrading Factors

This section introduces the factors that hinder the adoption and development of CNN-based brain tumor classification CADx systems into clinic practice, including data quality, data scarcity, data mismatch, data imbalance, classification performance, research value towards clinic needs, and the Black-Box characteristics of CNN models.

4.2.1. Data Quality

During the MR image acquisition process, both the scanner and external sources may produce electrical noise in the receiver coil, generating image artifacts in the brain MR volumes [ 69 ]. In addition, the MR image reconstruction process is sensitive to acquisition conditions, and further artifacts are introduced if the subject under examination moves during the acquisition of a single image [ 69 ]. These errors are inevitable and reduce the quality of the MR images used to train networks. As a result, the quality of the training data degrades the sensitivity/specificity of CNN models, thus compromising their applicability in a clinic setting.

4.2.2. Data Scarcity

Big data is one of the biggest challenges that CNN-based CADx systems face today. A large number of high-quality annotated data is required to build high-performance CNN classification models, while it is a challenge to label a large number of medical images due to the complexity of medical data. When a CNN classification system does not have enough data, overfitting can occur—as classification is based on extraneous variance in the training set—affecting the capacity of the network to generalize new data [ 91 ].

4.2.3. Data Mismatch

Data mismatch refers to a situation in which a model that has been well-trained in a lab environment fails to generalize real-world clinical data. It might be caused by overfitting of the training set or due to mismatch between research images and clinic ones [ 82 ]. Studies are at high risk of generalization failure if they omit a validation step or if the test set does not reflect the characteristics of the clinical data.

4.2.4. Class Imbalance

In brain MRI datasets such as the BraTS 2019 dataset [ 92 ], which consists of 210 HGG and 75 LGG patients (unbiased Gini coefficient 0.546, as shown in Table 2 ), HGG is represented by a much higher percentage of samples than LGG, leading to so-called class imbalance problems, in which inputting all of the data into the CNN classifier to build up the learning model will usually lead to a learning bias to the majority class [ 93 ]. When an unbalanced training set is used, it is important to assess model performance using several performance measures ( Section 3.5 ).

4.2.5. Research Value towards Clinical Needs

Different brain tumor classification tasks were studied using CNN-based deep learning techniques during the period from 2015 to 2022, including clinically relevant two-class classification (normal vs. tumorous [ 29 , 41 , 94 , 95 ], HGG vs. LGG [ 27 , 40 , 45 , 73 ], LGG-II vs. LGG-III [ 96 ], etc.); three-class classification (normal vs. LGG vs. HGG [ 24 ], meningioma (MEN) vs. pituitary tumor (PT) vs. glioma [ 39 , 42 , 49 , 50 ], glioblastoma multiforme (GBM) vs. astrocytoma (AST) vs. oligodendroglioma (OLI) [ 30 ], etc.); four-class classification (LGG vs. OLI vs. anaplastic glioma (AG) vs. GBM [ 72 ], normal vs. AST-II vs. OLI-III vs. GBM-IV [ 24 ], normal vs. MEN vs. PT vs. glioma [ 97 ], etc.); five-class classification (AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM-IV [ 24 ]); and six-class classification (normal vs. AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM-IV [ 24 ]).

Not all classification tasks are equally difficult, and this is the case for the deep learning research community and clinical practice. The authors in [ 24 ] used AlexNet for multi-class classification tasks, including two-class classification: normal vs. tumor, three-class classification: normal vs. LGG vs. HGG; four-class classification: normal vs. AST vs. OLI vs. GBM; five-class classification: AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM-IV, and six-class classification: normal vs. AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM-IV. The results reported 100% accuracy for the normal vs. tumorous classification. The accuracy for the five-class classification (AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM-IV) was only 87.14%. Similarly, in a recent publication [ 98 ], the authors utilized the same CNN model for multi-class brain tumor classification. The overall accuracy obtained for normal vs. tumorous classification reached 100% compared to the lower accuracy of 90.35% obtained for the four-class classification task (Grade I vs. Grade II vs. Grade III vs. Grade IV) and 86.08% for the five-class classification of AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM.

The goal of research in the field of CADx is to help address existing unmet clinical needs and to provide assistance methods and tools for the difficult tasks that human professionals cannot easily handle in clinical practice. It is observed that CNN-based models have achieved quite high accuracies for normal/tumorous image classification, while more research is needed to improve the classification performance of more difficult tasks, especially in five-class classification (e.g., AST-II vs. AST-III vs. OLI-II vs. OLI-III vs. GBM) and four-class classification (e.g., Grade I vs. Grade II vs. Grade III vs. Grade IV) tasks. Therefore, studies that use normal vs. tumorous as their target problem have little clinical value.

4.2.6. Classification Performance

Classification performance, which indicates the reliability and trustworthiness of CADx systems, is one of the most important factors to be considered when translating research findings into clinical practice. It has been shown that CNN techniques perform well in most of brain tumor classification tasks, such as in two-class classification (normal and tumorous [ 94 , 95 ] and HGG and LGG [ 45 , 73 ]) and three-class classification (normal vs. LGG vs. HGG [ 24 ] and MEN vs. PT vs. glioma [ 49 , 50 ]) tasks. However, the classification performance obtained for more difficult classification tasks, such as a five-class classification between AST-II, AST-III, OLI-II, OLI-III, and GBM, remains poor [ 24 , 98 ] and justifies further research.

4.2.7. Black-Box Characteristics of CNN Models

The brain tumor classification performance of some of the CNN-based deep learning techniques reviewed here is remarkable. Still, their clinical application is also limited by another factor: the “Black-Box” problem. Even the designers of a CNN model cannot usually explain the internal workings of the model or why it arrived at a specific decision. The features used to decide the classification of any given image are not an output of the system. This lack of explainability reduces the confidence of clinicians in the results of the techniques and impedes the adoption and development of deep learning tools into clinical practice [ 99 ].

4.3. Overview of Included Studies

Many research papers have emerged following the wave of enthusiasm for CNN-based deep learning techniques from 2015 to present day. In this review, 83 research papers are assessed to summarize the effectiveness of CNN algorithms in brain tumor classification and to suggest directions for future research in this field.

Among the articles included, twenty-five use normal/tumorous as their classification target. However, as mentioned in Section 4.2.5 , the differentiation between normal and tumorous images is not a difficult task. It has been well-solved both in research and clinic practice and thus has little value for clinical application. Therefore, studies that use normal vs. tumorous as their target problem will not be reviewed further in the following assessment steps.

Table 4 a provides an overview of the included studies that focus on CNN-based deep learning methods for brain tumor classification but does not include studies working with a normal vs. tumorous classification. The datasets, MRI sequences, size of the datasets, and the preprocessing methods are summarized. Table 4 b summarizes the classification tasks, classification architecture, validation methods, and performance metrics of the reviewed articles.

(a) Overview of included studies that focus on CNN-based deep learning methods for brain tumor classification, with the exception of studies focusing on normal vs. tumorous classification. Datasets, MRI sequences, size of the datasets, and preprocessing methods are summarized. (b) Overview of included studies that focus on CNN-based deep learning methods for brain tumor classification, with the exception of study focusing on normal vs. tumorous classification. Classification tasks, classification architecture, validation methods, and performance metrics are summarized.

Notes: 1 Rigid registration unless otherwise notes; 2 translation also referred to as shifting; 3 scaling also referred to as zooming; 4 reflection also referred to as flipping or mirroring; ** The Cancer Imaging Archive, https://www.cancerimagingarchive.net/ (accessed on 27 July 2022). 5 Referring to overall accuracy, mean accuracy, or highest accuracy depending on the information provided by the paper or the highest accuracy when multiple models are used.

As introduced in Section 4.2 , the major challenge confronting brain tumor classification using CNN techniques in MR images lies in the training data, including the challenges caused by data quality, data scarcity, data mismatch, and data imbalance, which hinder the adoption and development of CNN-based brain tumor classification CADx systems into clinic practice. Here, we assess several recently published studies to provide a convenient collection of the state-of-the-art techniques that have been used to address these issues and the problems that have not been solved in those studies.

Currently, data augmentation is recognized as the best solution to the problem caused by data scarcity and has been widely utilized in brain tumor classification studies.

The authors in [ 100 ] used different data augmentation methods, including rotation, flipping, Gaussian blur, sharpening, edge detection, embossing, skewing, and shearing, to increase the size of the dataset. The proposed system aims to classify between Grade I, Grade II, Grade III, and Grade IV, and the original data consist of 121 images (36 Grade I images, 32 Grade II images, 25 Grade III images, and 28 Grade IV images), and by using data augmentation techniques, 30 new images are generated from each MR image. The proposed model is experimentally evaluated using both augmented and original data. The results show that the overall accuracy after data augmentation reaches 90.67%, which is greater than the accuracy of 87.38% obtained without augmentation.

While most data augmentation techniques aim to increase extraneous variance in the training set, deep learning can be used by itself, at least in theory, to increase meaningful variance. In a recent publication by Allah et al. [ 44 ], a novel data augmentation method called a progressive growing generative adversarial network (PGGAN) was proposed and combined with rotation and flipping methods. The method involves an incremental increase of the size of the model during the training to produce MR images of brain tumors and to help overcome the shortage of images for deep learning training. The brain tumor images were classified using a VGG19 feature extractor coupled with a CNN classifier. The accuracy of the combined VGG19 + CNN and PGGAN data augmentation framework achieved an accuracy of 98.54%.

Another approach that helps overcome the problem of data scarcity and that can also reduce computational costs and training time is transfer learning. Transfer learning is a hot research topic in machine learning; previously learned knowledge can be transferred for the performance of a new task by fine-tuning a previously generated model with a smaller dataset that is more specific to the aim of the study. Transfer learning is usually expressed using pre-trained models such as VGG, GoogLeNet, and AlexNet that have been trained on the large benchmark dataset ImageNet [ 101 ].

Many attempts have been made to investigate the value of transfer learning techniques for brain tumor classification [ 39 , 45 , 50 , 102 , 104 , 108 , 116 , 121 ]. Deepak and Ameer [ 39 ] used the GoogLeNet with the transfer learning technique to differentiate between glioma, MEN, and PT from the dataset provided by Cheng [ 55 ]. This proposed system achieved a mean classification accuracy of 98%.

In a study conducted by Yang et al. [ 45 ], AlexNet and GoogLeNet were both trained from scratch and fine-tuned from pre-trained models from the ImageNet database for HGG and LGG classification. The dataset used in this method consisted of ceT 1 w images from 113 patients (52 LGG, 61 HGG) with pathologically proven gliomas. The results show that GoogLeNet proved superior to AlexNet for the task. The performance measures, including validation accuracy, test accuracy, and test AUC of GoogLeNet trained from scratch, were 0.867, 0.909, and 0.939, respectively. With fine-tuning, the pre-trained GoogLeNet obtained performed better during glioma grading, with a validation accuracy of 0.867, a test accuracy of 0.945, and a test AUC 0.968.

The authors in [ 50 ] proposed a block-wise fine-tuning strategy using a pre-trained VGG19 for brain tumor classification. The dataset consisted of 3064 images (708 MEN, 1426 glioma, and 930 PT) from 233 patients (82 MEN, 89 glioma, and 62 PT). The authors achieved an overall accuracy of 94.82% under five-fold cross-validation. In another study by Bulla et al. [ 108 ], classification was performed in a pre-trained InceptionV3 CNN model using data from the same dataset. Several validation methods, including holdout validation, 10-fold cross-validation, stratified 10-fold cross-validation, and group 10-fold cross-validation, were used during the training process. The best classification accuracy of 99.82% for patient-level classification was obtained under group 10-fold cross-validation.

The authors in [ 104 ] used InceptionResNetV2, DenseNet121, MobileNet, InceptionV3, Xception, VGG16, and VGG19, which have already been pre-trained on the ImageNet dataset, to classify HGG and LGG brain images. The MR images used in this research were collected from the BraTS 2019 database, which contains 285 patients (210 HGG, 75 LGG). The 3D MRI volumes from the dataset were then converted into 2D slices, generating 26,532 LGG images and 94,284 HGG images. The authors selected 26,532 images from HGG to balance these two classes to reduce the impact on classification performance due to class imbalance. The average precision, f1-score, and sensitivity for the test dataset were 98.67%, 98.62%, and 98.33%, respectively.

Lo et al. [ 116 ] used transfer learning with fine-tuned AlexNet and data augmentation to classify Grade II, Grade III, and Grade IV brain tumor images from a small dataset comprising 130 patients (30 Grade II, 43 Grade III, 57 Grade IV). The results demonstrate much higher accuracy when using the pre-trained AlexNet. The proposed transferred DCNN CADx system achieved a mean accuracy of 97.9% and a mean AUC of 0.9991, while the DCNN without pre-trained features only achieved a mean accuracy of 61.42% and a mean AUC of 0.8222.

Kulkarni and Sundari [ 121 ] utilized five transfer learning architectures, AlexNet, VGG16, ResNet18, ResNet50, and GoogLeNet, to classify benign and malignant brain tumors from the private dataset collected by the authors, which only contained 200 images (100 benign and 100 malignant). In addition, data augmentation techniques, including scaling, translation, rotation, translation, shearing, and reflection, were performed to generalize the model and to reduce the possibility of overfitting. The results show that the fine-tuned AlexNet architecture achieved the highest accuracy and sensitivity values of 93.7% and 100%.

Despite many studies on CADx systems demonstrating inspiring classification performance, the validation of their algorithms for clinical practice has hardly been carried out. External validation is an efficient approach to overcome the problems caused by data mismatch and to improve the generalization, stability, and robustness of classification algorithms. It is the action of evaluating the classification model in a new independent dataset to determine whether the model performs well. However, we only found two studies that used an external clinical dataset to evaluate the effectiveness and generalization capability of the proposed scheme, which is described in below.

Decuyper et al. [ 73 ] proposed a 3D CNN model to classify brain MR volumes collected from the TCGA-LGG, TCGA-GBM, and BraTS 2019 databases into HGG and LGG. Multiple MRI sequences, including T 1 w, ceT 1 w, T 2 w, and FLAIR, were used in this research. All of the MR data were co-registered to the same anatomical template and interpolated to 1 mm 3 voxel sizes. Additionally, a completely independent dataset of 110 patients acquired at the Ghent University Hospital (GUH) was used as an external dataset to validate the efficiency and generalization of the proposed model. The resulting validation accuracy, sensitivity, specificity, and AUC for the GUH dataset were 90.00%, 90.16%, 89.80%, and 0.9398.

In [ 120 ], Gilanie et al. presented an automatic method using a CNN architecture for astrocytoma grading between AST-I, AST-II, AST-III, and AST-IV. The dataset consisted of MR slices from 180 subjects, including 50 AST-I cases, 40 AST-II cases, 40 AST-III cases, and 50 AST-IV cases. T1w, T2w, and FLAIR were used in the experiments. In addition, the N4ITK method [ 80 ] was used in the preprocessing stage to correct the bias field distortion present in the MR images. The results were validated on a locally developed dataset to evaluate the effectiveness and generalization capabilities of the proposed scheme. The proposed method obtained an overall accuracy of 96.56% for the external validation dataset.

In brain tumor classification, it is often necessary to use image co-registration to preprocess input data when images are collected from different sequences or different scanners. However, we found that this problem has not yet been taken seriously. In the surveyed articles, six studies [ 73 , 76 , 98 , 118 , 135 , 136 ] used data from multiple datasets for one classification target, while only two studies [ 73 , 76 ] performed image co-registration during the image preprocessing process.

The authors in [ 76 ] proposed a 2D Mask RCNN model and a 3DConvNet model to distinguish between LGG (Grades II and Grade III) and HGG (Grade IV) on multiple MR sequences, including T 1 w, ceT 1 w, T 2 w, and FLAIR. The TCIA-LGG and BraTS 2018 databases were used to train and validate these two CNN models in this research work. In the 2D Mask RCNN model, all of the input MR images were first preprocessed by rigid image registration and intensity inhomogeneity correction. In addition, data augmentation was also implemented to increase the size and the diversity of the training data. The performance measures accuracy, sensitivity, and specificity achieved values of 96.3%, 93.5%, and 97.2% using the proposed 2D Mask RCNN-based method and 97.1%, 94.7%, and 96.8% with the 3DConvNet method, respectively.

In the study conducted by Ayadi [ 98 ], the researchers built a custom CNN model for multiple classification tasks. They collected data from three online databases, Radiopaedia, the dataset provided by Cheng, and REMBRANDT, for brain tumor classification, but no image co-registration was performed to minimize shift between images and to reduce its impact on the classification performance. The overall accuracy obtained for tumorous and normal classification reached 100%; for normal, LGG, and HGG classification, it reached 95%; for MEN, glioma, and PT classification, it reached 94.74%; for normal, AST, OLI, and GBM classification, it reached 94.41%; for Grade I, Grade II, Grade III, and Grade IV classification, it reached 90.35%; for AST-II, AST-III, OLI-II, OLI-III, and GBM classification, it reached 86.08%; and for normal, AST-II, AST-III, OLI-II, OLI-III, and GBM classification, it reached 92.09%.

The authors in [ 118 ] proposed a 3D CNN model for brain tumor classification between GBM, AST, and OLI. A merged dataset comprising data from the CPM-RadPath 2019 and BraTS 2019 databases was used to train and validate the proposed model, but the authors did not perform image co-registration. The results show that the classification model has very poor performance during brain tumor classification, with an accuracy of 74.9%.

In [ 135 ], the researchers presented a CNN-PSO method for two classification tasks: normal vs. Grade II vs. Grade III vs. Grade IV and MEN vs. glioma vs. PA. The MR images used for the first task were collected from four publicly available datasets: the IXI dataset, REMBRANDT, TCGA-GBM, and TCGA-LGG. The overall accuracy obtained was 96.77% for classification between normal, Grade II, Grade III, and Grade IV and 98.16% for MEN, glioma, and PA classification.

Similar to the work conducted in [ 135 ], Anaraki et al. [ 136 ] used MR data merged from four online databases: the IXI dataset, REMBRANDT, TCGA-GBM, and TCGA-LGG, and from one private dataset collected by the authors for normal, Grade II, Grade III, and Grade IV classification. They also used the dataset proposed by Cheng [ 55 ] for MEN, glioma, and PA classification. Different data augmentation methods were performed to further enlarge the size of the training set. The authors in these studies did not co-register the MR images from different sequences from different institutions for the four-class classification task. The results show that 93.1% accuracy was achieved for normal, Grade II, Grade III, and Grade IV classification, and 94.2% accuracy was achieved for MEN, glioma, and PA classification.

Despite the high accuracy levels reported in most studies using CNN techniques, we found that in several studies [ 102 , 117 , 118 , 137 ], the models demonstrated very poor performance during brain tumor classification tasks.

The authors in [ 102 ] explored transfer learning techniques for brain tumor classification. The experiments were performed on the BraTS 2019 dataset, which consists of 335 patients diagnosed with brain tumors (259 patients with HGG and 76 patients with LGG). The model achieved a classification AUC of 82.89% on a separate test dataset of 66 patients. The classification performance obtained by transfer learning in this study is relatively low, hindering its development and application in clinical practice. The authors of [ 117 ] presented a 3D CNN model developed to categorize adult diffuse glioma cases into the OLI and AST classes. The dataset used in the experiment consisted of 32 patients (16 patients with OLI and 16 patients with AST). The model achieved accuracy values of 80%. The main reason for the poor performance probably lies in the small dataset, with only 32 patients being used for model training. That is far from enough to train a 3D model.

In another study [ 137 ], two brain tumor classification tasks were studied using the Lenet, AlexNet, and U-net CNN architectures. In the experiments, MR images from 11 patients (two metastasis, six glioma, and three MEN) obtained from Radiopaedia were utilized to classify metastasis, glioma, and MEN; the data of 20 patients collected from BraTS 2017 were used for HGG and LGG classification. The results show poor classification performance by the three CNN architectures on the two tasks, with an accuracy of 75% obtained by AlexNet and an accuracy of 48% obtained by Lenet for the first task and an accuracy of 62% obtained by AlexNet and an accuracy of 60% obtained by U-net for the second task. The poor performance of Lenet is probably due to its simple architecture, which is not capable of high-resolution image classification. On the other hand, the U-net CNN performs well in segmentation tasks but is not the most commonly used network for classification.

Even though CNNs have demonstrated remarkable performance in brain tumor classification tasks in the majority of the reviewed studies, their level of trustworthiness and transparency must be evaluated in a clinic context. Of the included articles, only two studies, conducted by Artzi et al. [ 122 ] and Gaur et al. [ 127 ], investigated the Black-Box nature of CNN models for brain tumor classification to ensure that the model is looking in the correct place rather than at noise or unrelated artifacts.

The authors in [ 122 ] proposed a pre-trained ResNet-50 CNN architecture to classify three posterior fossa tumors from a private dataset and explained the classification decision by using gradient-weighted class activation mapping (Grad-CAM). The dataset consisted of 158 MRI scans of 22 healthy controls and 63 PA, 57 MB, and 16 EP patients. In this study, several preprocessing methods were used to reduce the influence of MRI data on the classification performance of the proposed CNN model. Image co-registration was performed to ensure that the images become spatially aligned. Bias field correction was also conducted to remove the intensity gradient from the image. Data augmentation methods, including flipping, reflection, rotation, and zooming, were used to increase the size and diversity of the dataset. However, class imbalance within the dataset, particularly the under-representation of EP, was not addressed. The proposed architecture achieved a mean validation accuracy of 88% and 87% for the test dataset. The results demonstrate that the proposed network using Grad-CAM can identify the area of interest and train the classification model based on pathology-related features.

Gaur et al. [ 127 ] proposed a CNN-based model integrated with local interpretable model-agnostic explanation (LIME) and Shapley additive explanation (SHAP) for the classification and explanation of meningioma, glioma, pituitary, and normal images using an MRI dataset of 2870 MR images. For better classification results, Gaussian noise was introduced in the pre-processing step to improve the learning for the CNN, with mean = 0 and a standard deviation of 10 0.5 . The proposed CNN architecture achieved an accuracy of 94.64% for the MRI dataset. The proposed model also provided a locally model-agnostic explanation to describe the results for ordinary people more qualitatively.

5. Discussion

Many of the articles included in this review demonstrate that CNN-based architectures can be powerful and effective when applied to different brain tumor classification tasks. Table 4 b shows that the classification of HGG and LGG images and the differentiation of MEN, glioma, and PT images were the most frequently studied applications. The popularity of these applications is likely linked to the availability of well-known and easily accessible public databases, such as the BraTS datasets and the dataset made available by Cheng [ 55 ]. Figure 7 reveals that there is an increase in the overall accuracy achieved by CNN architectures for brain tumor classification from 2018 to 2022. It is observed that from 2019 onwards, the overall classification accuracy achieved in most studies reached 90%, with only few works obtaining lower accuracies, and in 2020, the extreme outlier accuracy was 48% [ 137 ]. It is also apparent from this figure that the proportion of papers with an accuracy higher than 95% increases after 2020.

Classification accuracy by publication year.

In order to discuss the technical differences and points of similarity between the papers included in the present review, we decided to proceed thematically. Wherever possible, it is more useful to make comparisons between studies containing as few differences as possible. The most commonly reported metric, and the only one that will be employed here, is the accuracy. There are several studies that allow us to make such comparisons across only one factor. In other cases, several studies employ a similar methodology, and we can perform across-study comparisons. Finally, accuracy data can be plotted for single factors to allow for a simple visual comparison without attempting to separate confounding factors.

5.1. The Importance of the Classification Task

Three papers [ 24 , 97 , 98 ] investigated the effect of splitting a dataset into different numbers of categories. They all showed the expected monotonic decrease in accuracy as the number of classes increased, with the caveat that the “normal” image category is relatively easy to distinguish from the others and does not decrease accuracy when added as an additional category. The pattern is also apparent in Figure 8 —the maximum accuracy for two-class problems was 100%; for four-class problems, it was 98.8%; and for six-class problems, it was 93.7%.

Classification accuracy by classification task.

Two papers employed a single architecture to perform different classification tasks [ 30 , 138 ] while keeping the number of classes constant. The results in [ 30 ] showed little difference between the accuracy obtained for two different problems, which could be explained by differences in the datasets. The results of [ 138 ] showed slightly larger variation between four two-class problems. Curiously, nets trained on larger datasets yielded worse accuracy values, suggesting that results obtained from smaller samples have an inflated accuracy (100% for a problem based on 219 images, 96.1% for a problem based on 2156 images). With reference to Figure 8 , the classification task seems to have a larger effect than the class number on the accuracy. Note that the categories that group various specific tasks (two-class, three-class) together show much greater heterogeneity than those with the same number of classes for specific comparisons.

Further evidence regarding the importance of the task comes from a comparison of the accuracy in the papers comparing tumor grade (LGC vs. HGC) and those seeking to differentiate different types of tumors (MEN vs. glioma vs. PT); although the latter task involves more classes, the median accuracy is 97.6 (against 94.4 for the former). We compared the articles that studied the classification of HGG and LGG and found that the classification performance varies widely, even between the articles published in 2021 that utilized state-of-the-art CNN techniques. One of the key factors that significantly affects the performance of CNN models for brain tumor classification lies in the size of the datasets. The authors of [ 40 , 78 ] both proposed custom CNN models to classify HGG and LGG images of 285 MRI scans from the BraTS 2017 dataset. The overall accuracy values were 90.7% and 94.28%, respectively. The authors of [ 137 ] utilized AlexNet for the same task, but MRI data of only 20 patients from the same dataset were studied. The model in this study yielded a poor classification accuracy of 62%, the lowest value among the articles on this classification task.

Figure 8 presents the overall accuracies achieved by the reviewed studies that worked on different classification tasks. What stands out in the figure is that with the exception of the five-class tasks, which achieved accuracies lower than 90%, the CNNs achieved promising accuracies on different brain tumor classification tasks, especially in three-class classification tasks distinguishing between MEN, glioma, and PT. We also noticed that the accuracies of the three-class classification tasks fluctuated widely, with the lowest accuracy being 48% in [ 137 ] for the metastasis vs. glioma vs. MEN classification. More research attention should be paid to improving the accuracies of these classification tasks.

5.2. The Effect of the Dataset

A few studies applied the same network architecture to two different datasets. For He et al. [ 78 ], the results demonstrating a higher accuracy (94.4% against 92.9%) were based on a training set that was both larger and more unbalanced. The first factor would have improved the training process, while the latter made the classification task easier. Several papers derive different subgroups from different datasets (for example, healthy subject data from IXI and tumors from other sets). This is poor practice, as there are likely to be non-pathological differences between the sets acquired from different centres, and this can artificially inflate classification accuracy [ 139 ].

As was mentioned in the Results section, dataset size is considered a critical factor in determining the classification performance of a CNN architecture. Some studies report the dataset size in terms of the number of subjects included, and others report it in terms of the number of images. Typically, several images are included from each subject, but this number is not specified.

Figure 9 and Figure 10 sum up the classification accuracies obtained according to each of the factors; Figure 9 shows that there is a marked increase in the overall accuracy achieved with more training subjects The improvement gained by increasing the image number seems more modest.

Classification accuracy by number of patients.

Classification accuracy by number of images.

Another interesting aspect of the datasets used is the choice of MRI sequence. This may provide a hint as to the features being used for classification. Comparing the articles that focused on the same classification task, of the sequences listed in Table 3 , only ceT 1 w was associated with studies showing a higher classification accuracy than those that excluded it for MEN vs. Glioma vs. PT classification, while all of the sequences contributed to an improvement in LGG vs. HGG classification. As a consequence, studies using multiple sequences were associated with higher accuracy in the LGG vs. HGG task but not in MEN vs. Glioma vs. PT classification.

5.3. The Effect of CNN Architecture

Three studies present comparisons of different architectures trained on the same problems (Yang et al. [ 45 ], Kulkarni et al. [ 121 ], Wahling et al. [ 137 ]).

In a study conducted by Yang et al. [ 45 ], GoogLeNet and AlexNet were both trained from scratch and fine-tuned from pre-trained models from the ImageNet database for HGG and LGG classification. When both were trained from scratch, GoogLeNet proved superior to AlexNet for the task. The test accuracies were 0.909 and 0.855, respectively. Fine-tuning pre-existing nets resulted in better performance in both cases, with accuracies on the test set of 0.945 and 0.927, respectively. In [ 121 ], five nets were used to distinguish benign from malignant tumors. The reported accuracies were surprisingly variable; from worst to best, the results were VGG16 (0.5) and ResNet50 (0.68). In [ 137 ], AlexNet and LeNet were both used to distinguish three classes.

The overall accuracies achieved by the different CNN architectures that have been used extensively for brain tumor classification are summarized in Figure 11 . It shows that the majority of CNN models have achieved high performance for brain tumor classification tasks, in which transfer learning with ResNet, VGG, and GoogleNet showed more stable performance than other models, such as 3D CNN. Among the reviewed articles, five articles utilized 3D CNN for brain tumor classification, and the classification accuracy of those studies fluctuates wildly. The highest accuracy was 97.1%, achieved by Zhuge et al. [ 77 ], who trained a 3D CNN architecture with a dataset of 315 patients (210 HGG, 105 LGG). The lowest accuracy of 75% was obtained by Pei et al. [ 118 ], who used 398 brain MR image volumes for GBM vs. AST vs. OLI classification. In another study [ 117 ], the authors explored a 3D CNN model for OLI and AST classification using a very small dataset of 32 patients (16 OLI, 16 AST) and obtained a low accuracy of 80%. It seems that 3D CNN is a promising technique for realizing patient-wise diagnosis, and the accessibility of a large MRI dataset can hopefully improve the performance of 3D CNNs on brain tumor classification tasks.

Classification accuracy by CNN architecture.

5.4. The Effect of Pre-Processing and Data Augmentation Methods

Researchers have paid increasing amounts of attention to enhancing input image quality by conducting different preprocessing steps on brain MRI datasets before propagating them into CNN architectures. No studies have systematically tested the number and combination of operations that optimize classification accuracy. Figure 12 presents the overall accuracy obtained with different numbers of preprocessing operations. It shows that the studies that pre-processed input MR images collectively obtained higher classification accuracies than the studies that performed no preprocessing methods. However, it is not obvious that more steps led to better performance.

Classification accuracy by number of preprocessing operations.

As previously stated, data augmentation can create variations in the images that can improve the generalization capability of the models to new images, and different data augmentation techniques have been widely explored and applied to increase both the amount and the diversity of training data. Figure 13 illustrates the overall accuracy obtained with different numbers of data augmentation operations. It can be seen that studies that performed five data augmentation techniques achieved higher and more stable classification performance than the studies that performed fewer operations.

Classification accuracy by number of data augmentation operations.

The accuracy data do not support the use of any single data augmentation method. It is interesting to ask whether data augmentation techniques were implemented specifically in those studies that lacked training data. However, on average, there is little difference between the 59 studies including or the 27 omitting a data augmentation step. On average, the former included 233 cases or 4743 images, and the latter included 269 cases or 7517 images. Curiously, the number of studies employing data augmentation has fallen as a proportion among those published in 2022, both compared to the total and compared to those using pre-processing methods.

Figure 14 indicates the cumulative impact of factors that are not fully reported or considered in the studies reported in Table 4 . Articles with multiple analyses for which factors differed were scored 1 (i.e., missing). Data are derived from Table 4 , with the following exceptions: “Explainability considered” means that there was some analysis within the article on the information used to come to a diagnosis. Out-of-cohort testing occurred when CNN testing was performed on a cohort that was not used in the training/validation phase (i.e., different hospital or scanner). Author affiliations were derived from the author information in the DOI/CrossRef listed in the bibliography. An author was considered to have a clinical affiliation if their listed affiliations included a department of radiology, clinical neurology, neurosurgery, or oncology.

Histogram (left scale) and cumulative distribution (right scale) of factors not fully reported or considered in the studies reported in Table 4 .

From the figure, the category other performance criteria performed means that performance criteria other than accuracy were reported. Validation was considered to be not properly reported if it was not performed or if the methods used in the validation step were not clearly described. Training patients/images properly reported means that the number of patients/images in each category used for training/validation is explicitly defined. Both factors are relevant as separate images from the same patient and are not fully independent. Public data used means that the data used are available to other researchers. In practice, all of the public data used were gathered in other studies, and no non-public data were made available by any of the studies identified.

5.5. The Effect of Other Factors

Beyond showing accuracy gains, the surveyed articles rarely examined their generalization capability and interpretability. Only very few studies [ 73 , 120 ] tested their classification models on an independent dataset, and only one study [ 122 ] investigated the Black-Box characteristic of CNN models for brain tumor classification to ensure that the model they obtained was looking in the correct place for decision-making rather than at noise or unrelated artifacts.

A limitation of this survey arises from the challenge of making comparisons in an objective manner between studies to analyze how each degrading factor affects the classification performance. One reason is that some studies worked on the same classification task but utilized different datasets, preprocessing methods, or classification techniques. Another reason lies in the variety of performance metrics reported. While accuracy was the most popular performance metric, it was not universally reported. Based on the difficulties encountered in the preparation of the present review, we suggest that at the very least, all deep learning studies for classification clearly report the classification accuracy of the models constructed and the numbers of images/subjects of each class used for training, validation, and testing purposes.

5.6. Future Directions

It is clear from the comparative analysis presented in Table 4 b that CNN techniques and algorithms have great power and ability to handle medical MR data, but so far, but none of them are at the point of clinical usability. The challenges we have identified here must be appropriately addressed if CNN research is to be translated into clinic practice. This review has identified some common performance-degrading factors and potential solutions.

5.6.1. The Training Data Problem

An exorbitant number of training cases are required to train a deep learning algorithm from scratch. With a limited number of training data, transfer learning with fine-tuning on pre-trained CNNs was demonstrated to yield better results for brain tumor classification than training such CNNs from scratch [ 45 , 116 ]. This is an efficient method for training networks when training data are expensive or difficult to collect in medical fields. In addition, high hardware requirements and long training times are also challenges that CNN-based CADx brain tumor classification systems face in clinical applications today. The continued development of state-of-the-art CNN architectures has resulted with a voracious appetite for computing power. Since the cost of training a deep learning model scales with the number of parameters and the amount of input data, this implies that computational requirements grow at the rate of at least the square of the number of training data [ 140 ]. With pre-trained models, transfer learning is also promising to address the difficulties caused by high hardware requirements and long training times when adopting CNN-based CADx systems for brain tumor classification in clinical practice. There are many issues related to optimizing transfer learning that remain to be studied.

5.6.2. The Evaluation Problem

CADx systems are mainly used for educational and training purposes but not in clinical practice. Clinics still hesitate to use CADx-based systems. One reason for this is the lack of standardized methods for evaluating CADx systems in a realistic setting. The performance measures described in Section 4.2 are a useful and necessary baseline to compare algorithms, but they are all highly sensitive to the training set used, and more sophisticated tools are needed. It would be useful to define a pathway towards in-use performance evaluation, such as what was recently proposed for quantitative neuroradiology [ 141 ]. It is notable that many of the papers reviewed did not include any authors with a clinical background and that the image formats used to train the models were those typical of the AI research community (PNG) and not those of the radiology community (DICOM, NIfTI).

5.6.3. Explainability and Trust

The Black-Box nature of deep CNNs has greatly limited their application outside of a research context. To trust systems powered by CNN models, clinicians need to know how they make predictions. However, among the articles surveyed, very few addressed this problem. The authors in [ 142 ] proposed a prototypical part network (ProtoPNet) that can highlight the image regions used for decision-making and can explain the reasoning process for the classification target by comparing the representative patches of the test image with the prototypes learned from a large number of data. To date, several studies have tested the explanation model proposed in [ 142 ] that was able to highlight image regions used for decision making in medical imaging fields, such as for mass lesion classification [ 143 ], lung disease detection [ 144 , 145 ], and Alzheimer’s diseases classification [ 146 ]. Future research in the brain tumor classification field will need to test how explainable models influence the attitudes and decision-making processes of radiologists or other clinicians.

The lack of physician training on how to interact with CADx systems and how to interpret their results to make diagnostic decisions is a separate but related technical challenge that can reduce the performance of CADx systems in practice, something that is not addressed in any of the papers included in the review. A greater role for physicians in the research process may bring benefits both in terms of the relevance of research projects and the acceptance of their results.

In summary, the future of CNN-based brain tumor classification studies is very promising and focusing on the right direction with references to the challenges mentioned above would advance these studies from research labs to hospitals. We believe that our review provides researchers in the biomedical and machine learning communities with indicators for useful future directions for this purpose.

6. Conclusions

CADx systems may play an important role in assisting physicians in making decisions. This paper surveyed 83 articles that adopted CNNs for brain MRI classification and analyzed the challenges and barriers that CNN-based CADx brain tumor classification systems face today in clinical application and development. A detailed analysis of the potential factors that affect classification accuracy is provided in this study. From the comparative analysis in Table 4 b, it is clear that CNN techniques and algorithms have great power and ability to handle medical MR data. However, many of the CNN classification models that have been developed so far still are still lacking in one way or another in terms of clinical application and development. Research oriented towards appropriately addressing the challenges noted here can help drive the translation of CNN research into clinical practice for brain tumor classification. In this review, some performance degrading factors and their solutions are also discussed to provide researchers in the biomedical and machine learning communities with indicators for developing optimized CADx systems for brain tumor classification.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics12081850/s1 , Table S1: Article Screening Recording.

Funding Statement

This research was funded by China Scholarship Council (grant number: 202008320283). And The APC was funded by a voucher belonging to author L.R.

Author Contributions

Conceptualization, C.T. (Claudia Testa), D.N.M., F.Z., L.R., Y.X.; methodology, C.T. (Claudia Testa), D.N.M., F.Z., L.R., Y.X.; formal analysis, C.T. (Caterina Tonon), C.T. (Claudia Testa), D.N.M., F.Z., L.R.; investigation, C.T. (Claudia Testa), D.N.M., F.Z., L.R.; re-sources, C.T. (Caterina Tonon), R.A., R.L.; data curation, D.N.M., Y.X.; writing—original draft preparation, Y.X.; writing—review and editing, C.T. (Caterina Tonon), C.T. (Claudia Testa), D.N.M., F.Z., L.R.; supervision, C.T. (Caterina Tonon), C.T. (Claudia Testa), D.N.M.; funding acquisition, C.T. (Caterina Tonon), R.A., R.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Informed consent statement, data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.