Enhancing object detection in aerial images

Abbreviations: DMNet: Density-map guided object detection network; DOTA: Dataset of object deTection in aerial images; MCNN: Multi-column convolution neural network; MSCOCO: Microsoft common objects in context; R-CNN: Region-based convolution neural network; RPN: Region proposal network; SSD: Single-shot detector; VOC: Visual object classes; YOLO: You only look once

1.   Introduction

In the upcoming years, the large scale infrastructural changes and development of smart cities will see a rise in the use of drones for various applications such as package delivery, traffic monitoring, policing, pollution management, etc. ^[1,2]. Object detection in such situations will be a crucial part of the solution ecosystem. Object detection in aerial images is necessary for various critical applications in computer vision, such as pedestrian detection, autonomous vehicles, surveillance, crowd counting and scene classification. Object detection approaches can broadly be divided into two categories, namely region-based methods and single-shot methods.

Region-based methods include Fast R-CNN ^[3], Mask R-CNN ^[4], Faster R-CNN ^[5]. Fast R-CNN ^[3] uses an exhaustive selective search to generate region proposals, and based on those proposals, it performs feature extraction followed by classification. Overcoming the exhaustive selective search, Faster R-CNN ^[5] uses convolution network to generate Region Proposal Network (RPN). Mask R-CNN ^[4] further extends Faster R-CNN ^[5] by introducing an additional branch of predicting segmentation masks in a pixel-to-pixel manner.

Single-shot methods include YOLO ^[6], SSD ^[7] and many other methods proposed in recent times. These methods view the problem of object detection as a regression problem. YOLO ^[6] views the entire image at once and simultaneously predicts bounding box and class probabilities. It makes use of objectness scores for prediction. It is not recommended for small objects that account for larger aerial datasets percentages. SSD ^[7], on the other hand, uses multi-sized convolutional feature maps and has a smaller trade-off between speed and accuracy.

Since the advent of deep learning architectures, general object detection has witnessed state-of-the-art performances by these methods on natural datasets MSCOCO ^[8] and Pascal VOC ^[9]. Zhang et al. ^[10] and Chu et al. ^[11] have implemented state-of-the-art object detection on MSCOCO ^[8] dataset and KITTI dataset ^[12] respectively which are natural datasets where the images have been taken from the ground in a first person view. However, these methods have appeared to be less than satisfactory on datasets of aerial or satellite images such as the VisDrone dataset ^[13], DOTA ^[14]. Despite the good results in other application areas, state-of-the-art techniques often fail in performance when applied to aerial images.

Aerial images are generally captured via satellites, drones, and aeroplanes equipped with cameras that capture images from the top-view and larger field view. Since the images contain a larger field of view and usually the top-view of the images, this contributes to several challenges in aerial images compared to general object detection. Some of these challenges are mentioned below.

a. Scale Variation: The angle of photos or camera viewpoint varies significantly, resulting in the variance of the scale of different objects per image and class.

b. Imbalanced Data: Highly imbalanced and non-uniform distribution of objects corresponding to various categories. Bigger size objects are easier to detect by the detector than small size objects.

c. Occlusion: Occlusion refers to one object getting hidden by another one. Object occlusion issues between objects are very common in aerial images.

d. Low Resolution: Since images are taken from a particular height, resolution may vary specifically for small objects.

e. Lightning variation: Nighttime pictures pose a difficulty in detecting the accurate coverage of an object.

f. The dominance of small objects: A large percentage of the dataset comprises small objects; therefore, it is necessary to give importance to small objects for better performance.

These issues contribute to making aerial image object detection more challenging. However, addressing these challenges can help significantly in improving the existing methods and contributing to some very critical applications.

Multiple researchers like Jiang et al. ^[15], Yang et al. ^[16], Behera et al. ^[17], Santosh et al. ^[18], and Balamurugan et al. ^[19] focus on addressing the IOT challenges around drones applicable in smart cities like traffic monitoring and communication. Another important application of using drones in smart cities is scene text recognition ^[20,21]. In the object detection domain, past works Li et al. ^[22] have focused on the strategy of simply using architectures for general objects with putting some emphasis on small objects and improving detection. Unel et al. ^[23] suggested tiling, which is the use of cropping images for boosting small object detection. Extending the tiling technique ^[23], Yang et al. ^[24] have proposed the idea of clusters to crop dense object regions and using ScaleNet to balance the shape of crops of image. Wang et al. ^[25] have addressed the issue of invariant viewpoint by focusing on leveraging the spatial information with the help of increased receptive fields, which can efficiently cover all objects. Li et al. ^[26] addressed the use of density maps for the tiling process. They used Multi-column CNN (MCNN) proposed by Zhang et al. ^[27] to generate the density maps and then use those maps to crop the image further. Yu et al. ^[28] have introduced the concept of dilated convolutions.

Aerial images contain objects of various scales; therefore, networks with better receptive fields are desired. Dilated convolutions help in expanding the receptive fields. Zhang et al. ^[27] and Shen at al. ^[29] in their work have addressed the effects of using dilated convolutions and how it can contribute to the detection of small objects. The class-imbalance problem refers to the scenario where the minority class has fewer examples as compared to the majority class. This leads to a drop in the overall performance as the detector tends to over classify the majority class. Hensman et al. and Masko et al. ^[30] showed that an imbalanced classification task could be handled by the data augmentation process in the training data. Nemoto et al. ^[31] in their work addressed the idea of data augmentation via rotation and inversion for minimising the class imbalance. In recent times, two-stage detectors under the category of R-CNN have been adopted for the task of object detection. They are preferred for their computation speed and accuracy.

However, considering the aerial images and the challenges like scale variation, occlusion, and small and densely located objects, these detectors often witness a performance drop when applied directly to the aerial datasets. Lin et al. ^[32] have proposed a method, RetinaNet, which is a single-stage detector, but it can perform similar results as compared to the two-stage detector approach by R-CNN. RetinaNet combines the features of both Region Proposal Networks and Feature Pyramids which are used for multi-scale object detection.

Aerial images often contain background as the objects to be detected are relatively smaller in size. Suppose the whole image is fed directly into the detector. In that case, the computation time is unnecessarily added without any results. The detector has to look for objects in the region proposals of the background where there might be no object present. Many recent approaches have adopted the methodology of first cropping out the sub-regions where the object is absent and then performing the final detection. Inspired by this methodology, we have adopted a similar approach. We view the problem as a composition of three modules. First, in order to address the class imbalance problem, we have augmented the rare, occurring class instances with rotation of 90,180 and 270 degrees. Second, we have used an improved CNN to generate density maps which then help in cropping that image. Third, in order to accurately give better results on aerial images we have adopted RetinaNet ^[32] with improved anchor sizes. All the experiments have been performed on the VisDrone ^[13] dataset.

The contributions of this paper can be summarised as follows:

1) We have tried to address the class imbalance problem in the VisDrone dataset by rotation augmentation of infrequent classes. We have also demonstrated the impact of this augmentation in improving the final results.

2) We have proposed an improved version of MCNN with better receptive fields achieved using dilated filters considering the size variation of objects in the dataset.

3) We have suggested effective anchors for RetinaNet to detect smaller objects specific to aerial datasets accurately. We have also demonstrated the performance of anchor optimised RetinaNet on various backbone architectures.

The rest of the paper is organised as follows. Section 2 explains the dataset and the three-step methodology proposed for object detection in aerial images. Section 3 presents the analysis of the results obtained from the proposed methods as compared to existing methods. Section 4 discusses a summary of the results and their impact, along with limitations and possible future directions for the study.

2.   Materials and methods

This section describes the dataset used in this study along with the three step proposed methodology and the procedure of the experiment employed by the study.

2.1.   Dataset description

With the high demand of practical applications like surveillance, there have been various datasets available for analysis, comparison and setting up benchmarks to identify the best performance acquired so far. Since, natural images significantly differ from aerial images, special aerial image datasets have gained more focus. We have experimented our methodology and tested its performance against the VisDrone-2019 ^[13] dataset. Figure 1 shows some sample images from the dataset with ground-truth bounding boxes. VisDrone 2019 is one of the most popular datasets used for object detection in aerial images. The total 8599 images are divided into three sets where the training set includes 6471 images, validation set includes 548 images and the remaining 1580 images are being included into the test set. The dataset contains images with occlusion, lightning variation and densely packed objects encapsulating the various challenges as depicted in Figure 2. Since the images are taken from an altitude, the camera viewpoint changes. The dataset contains ten classes/categories which includes human beings and transport vehicles. Human being classes are 'pedestrian' which refer to those standing while class 'person' refer to human who is standing, walking, running or any other pose. Transport vehicle classes include 'bicycle' 'car' 'van' 'truck' 'tricycle' 'awning tricycle' 'bus' and 'motor'.

Figure 1.  Sample images from VisDrone ^[13] dataset with ground-truth bounding boxes.

DownLoad: Full-Size Img PowerPoint

Figure 2.  Images depictingchallengessuchas illumination variation, sizevariation, occlusion.

DownLoad: Full-Size Img PowerPoint

2.2.   Proposed method

We have adopted a three step methodology that utilises the advantages of these architectures. First step deals with the process of addressing the class-imbalance problem. In the second step, we have proposed an improved MCNN with dilated filters or convolutions in order to gain higher receptive fields and generate better image crops out of the original image. We have performed decomposition of original image followed by object detection in order to minimise the background noise. Third step performs the detection using RetinaNet and analyses the performance of RetinaNet with default anchors and also proposes adjusted anchor sizes in RetinaNet to boost the detection of small objects. The framework of our approach is illustrated in Figure 3.

Figure 3.  Proposed methodology.

DownLoad: Full-Size Img PowerPoint

2.2.1.   Data augmentation of infrequent classes

VisDrone dataset suffers from the class-imbalance problem. As illustrated in Figure 4, the classes 'car' and 'pedestrian' account for a larger portion of the dataset while classes 'bus' and 'awning tricycle' occur in small proportions. We performed Data Augmentation to minimise this problem. We performed augmentation of the infrequent categories by picking out images with rarely occurring objects present from the dataset and adding those entries back into the dataset. In order to avoid the case of overfitting of oversampled images upon direct augmentation, we augmented the images by rotating them at angles 90,180,270. This method helps in reducing the difference between minority classes and majority classes.

Figure 4.  Visualization of class distribution in VisDrone dataset in terms of percentage.

DownLoad: Full-Size Img PowerPoint

2.2.2.   Image cropping using improved MCNN with dilated convolutions

Before feeding images for the detection, we divide the image into multiple crops. We have adopted the methodology of image cropping where input image is decomposed into several crops or chips by leveraging the advantages of density maps. Segmentation followed by detection helps to minimise the background noise. We used the concept of density-based maps used in crowd counting which is one of the extensive practical applications, to the domain of aerial imagery object detection. For the purpose of density map generation, we have adopted improved MCNN which is a multi-column architecture where we have involved four columns in order to cover all the scales and capture the features properly. Conventional MCNN ^[27] used three columns for small, medium and large size objects but for an effective receptive field for smaller objects we have incorporated four columns with adjusted filters. In addition to it, to increase the receptive field of columns we have introduced dilated convolutions Instead of standard convolution-pooling layers. Dilation factor helps in looking at larger areas while preserving spatial resolution by expanding the receptive field of kernels. Figure 5 is the proposed architecture of improved MCNN.

Figure 5.  Improved MCNN with 4 columns for various scales. 'D' represents Dilation factor.

DownLoad: Full-Size Img PowerPoint

This way features are adaptive and robust towards perspective distortion, varying resolution, and scale variation. In this case, MCNN consists of four columns or four parallel CNNs with different filter sizes to capture features corresponding to different scales. After density maps, the cropping module generates image crops for the forward process of detection. Cropping module is able to produce better crops, as can be seen in Figure 6, due to distribution of objects as background area can be filtered out effectively and the number of objects belonging to each crop can also be reduced. The cropping module has been used similarly as mentioned in ^[26]. The images are cropped by accumulating the pixel intensities and comparing their sum with a given threshold value. This is achieved with the use of a sliding window that slides over the density map and gathers the information to generate a density mask. The windows with intensities higher than threshold are combined together to break the image into crops. Then the final detection is performed. This facilitates the effective recognition of small objects.

Figure 6.  Visualization of (a) uniform cropping vs. (b) density crops achieved by our methodology. (a) corresponds to region proposals/ crops suggested by uniform cropping and (b) corresponds to cropped chips corresponding to MCNN with dilated convolutions. The uniform crops contain mostly background pixels. The crops made by density based methodology are more useful with respect to our task.

DownLoad: Full-Size Img PowerPoint

2.2.3.   Object detection by RetinaNet with better anchors

The final step is to perform detection on the crops obtained in the previous step. For the detection, any object detection architecture can be used. Algorithms such as YOLO ^[6], Fast R-CNN ^[3], Faster R-CNN ^[5] etc. can be chosen as the baseline architecture keeping in mind the trade-offs and challenges. We have adopted RetinaNet for this purpose due to its recent promising results in detecting small objects. RetinaNet which is a single-stage detector but it can perform similar results as compared to the two-stage detector approach by R-CNN. RetinaNet combines the features of both Region Proposal Networks and Feature Pyramids which are used for multi-scale object detection. However, we analyse that the default configuration of RetinaNet anchors is inadequate for detection in aerial images. The default anchor sizes are 32, 64,128,512 which work fine for normal objects but often miss out the smaller objects. We adjusted the anchor sizes to 16, 32, 64,128 which improves the detection as there is dominance of small objects and it is then able to capture the instances that it missed earlier. Further, we demonstrate the effect of adjusted anchors with respect to different backbone architectures. We have used ResNet50, ResNet101 to analyse the effect of RetinaNet.

2.3.   Experiment implementation

We have first tried to implement and replicate the results of ^[26]. In order to understand their approach and look for scope of improvements, we have followed their procedure of implementation. Their work uses MCNN for generating the crops and Faster R-CNN ^[5] for detection. As suggested, we have used MMDetection toolbox ^[33] for this purpose. MMDetection toolbox is a rich toolbox which is open-source and contains a number of popular detector algorithms. It also provides the flexibility of hyperparameter tuning and provides inference and ability to train on standard datasets or customizable datasets. Before applying final detection, there is a need to convert text annotations of the images into annotations of MSCOCO ^[8] format as the detector algorithms are designed to run on these formats containing natural images. For the backbone of the network, the ResNet-50 and ResNet-101 architectures were used.

Inspired by ^[26], we have implemented our suggested methodology on the VisDrone-2019 dataset ^[13], with data augmentation as the first step. The classes with total number of instances less than 10,000 in the training set were 'bus' 'tricycle' 'awning tricycle' with 5926, 8058, and 3246 instances respectively. We picked out the images containing these classes and augmented their 90,180, and 270 degree rotations back to the dataset. We have implemented improved MCNN in PyTorch as the underlying framework for creating density maps. Our MCNN uses four parallel CNNs with local receptive fields of varying sizes 7, 5, 3, and 3. Filters of varying sizes enable looking at objects of different scales, for example, filters with small receptive fields will help in creating maps for smaller objects. In addition to it, we have used dilated convolutions instead of normal (conv-pooling-conv-pooling) in the structure. The dilation filter values (1, 2, 4, 8) vary for each column corresponding to the kernel size. The resultant from each column are then stacked together to generate the density map. We have used the Stochastic Gradient Descent optimizer and Mean Absolute Error for the generation of density maps similar to the setting in ^[26].

Image crops are generated by cropping the connected neighbouring region as specified by the density maps. The cropping module implementation has been referred from ^[26]. For the purpose of detection, any state-of-the-art object detector can be used, here in this case we have implemented RetinaNet using Keras as per ^[32]. Before applying final detection, we converted annotations into the format of RetinaNet. RetinaNet has default anchor configurations in which we included anchor size 16 as some of the objects span over very less pixel area in the image and omitted the highest size 512. In addition to it, we increased the scales from 3 to 5 to increase the number of anchor boxes. As per the standard setting in ^[32], we have used Adam optimiser, Smooth L1 loss and focal loss with standard hyperparameters. We trained the model with default anchor sizes and then adjusted the size to boost the performance. For the backbone of the network, we have used ResNet-50 and ResNet-101 architectures and trained them separately to observe the effect of depth of backbone architectures.

3.   Results and discussion

Average Precision has been used to evaluate and compare performance. It refers to precision across various threshold values and evaluates the overlap between the ground-truth bounding boxes and the detected boxes.

Initially, we have implemented the approach presented in ^[26]. It is important to note that those experiments were performed on only half of the data which resulted in decrease of performance. The results are summarised in Table 1.

Table 1.  Results of standard vs. our implementation of ^[26].

Method	Backbone Architecture	AP
DMNet [26]	ResNet-50	28.2
DMNet [26]	ResNeXt-101	29.4
DMNet (Our implementation of [26])	ResNet-50	27.1
DMNet (Our implementation of [26])	ResNeXt-101	27.9

---

 | Show Table

DownLoad: CSV

Further we have run experiments and evaluated the performance of our methodology on VisDrone ^[13] Test-Dev dataset. We have tried to analyse the impact of every step by including them in the methodology in a step-wise manner. We have introduced data augmentation keeping rest of the structure in their default configurations, that is, MCNN and RetinaNet. Then we introduced improved MCNN keeping RetinaNet still same with default parameters. Afterwards, we added the improved RetinaNet to assess the overall performance. The results are illustrated in Table 2.

Table 2.  Quantitative results of our methodology on VisDrone ^[13] dataset.

Method	Backbone	AP
DMNet [26]	ResNet-50	28.2
DMNet [26]	ResNet-101	28.5
Data Augmentation(Ours) + MCNN + RetinaNet	ResNet-50	28.1
Data Augmentation(Ours) + Improved MCNN(Ours) + RetinaNet	ResNet-50	28.7
Data Augmentation(Ours) + Improved MCNN(Ours) + RetinaNet	ResNet-101	28.9
Data Augmentation(Ours) + Improved MCNN(Ours) + Improved RetinaNet(Ours)	ResNet-50	29.6
Data Augmentation(Ours) + Improved MCNN(Ours) + Improved RetinaNet(Ours)	ResNet-101	29.9

---

 | Show Table

DownLoad: CSV

We can infer from the table that our results outperform the standard ^[26] across both the backbone architectures ResNet 50 and ResNet 101 respectively. The significance of each step can be clearly inferred from the results. Improved MCNN helps in increasing receptive fields of the filters, RetinaNet with adjusted anchor parameters helped in detecting smaller objects efficiently. Further, experiments done with ResNet-101 gave better results as compared to ResNet-50. Instances belonging to classes such as 'people' 'bicycle' 'awning tricycle' often go undetected which leads to performance drop.

Based on the results, we were able to gain understanding of the effect of various modules. Our methodology focuses on density map based cropping of images using improved version of MCNN that generates better image crops for detection. We can see the performance gain of 0.6 points when improved MCNN was used along with the default RetinaNet architecture over ResNet-50. This increase occurs due to dilated filters used in MCNN that are able to expand the receptive fields at various scales. This performance further improves by a factor of 0.2 when ResNet-101 is used.

Further, it also suggests an improved RetinaNet with adjusted anchors and ratios that increase the detection of smaller objects. After replacing the default RetinaNet parameters by our suggested parameters, we were able to witness a further increase of 0.7 with ResNet-50. This can be accounted for by the fact that anchors were now able to capture the small objects with smaller pixel coverage which went undetected earlier. Similar to earlier experiments, ResNet-101 boosted it to more by a factor of 0.3. We can also infer that the deeper backbone architectures deliver better results.

Additionally, we also adopted data augmentation of infrequent classes as a pre-processing step that helped in increasing the rare-occurring instances. This pipeline is able to outperform the other density based approaches in terms of AP due to the robust properties of the improved architectures.

4.   Conclusions

Object Detection in aerial images has become an interesting and demanding area of research particularly due to its practical applications in the upcoming development of smart cities. It has emerged as an area with a wider range of possibilities and wide spectrum of approaches that have been adapted for improving the performance. This article discusses the challenges due to which detectors used in general object detection designed for natural images fail in achieving state-of-the-art performance in this domain. We have also discussed some of the architectures and methodologies proposed in recent times considering the challenges of aerial images. We have adopted data augmentation of infrequent classes as a part of pre-processing to increase the instances of less occurring classes. Further, expanding the ideology of density based image cropping, our approach generates density maps with improved MCNN with dilation factor, then crops images and finally performs detection with RetinaNet with default as well as adjusted anchor parameters across different backbone architectures. We have emphasised on the importance of every contribution suggested in this work with observations. The performance of the proposed methodology is a significant improvement over the existing state-of-the-art performance. However, due to the three step pipeline, our methodology becomes slightly computationally expensive. Future work can be done towards improvement in the computations. Further, Ensemble Learning methodology has always been proven to exhibit better performances as compared to the stand-alone detectors. Therefore, ensemble methodology also provides a good scope to try different combinations of models on aerial images. In the field of aerial imagery object detection, the scenario of heavy overlapping and extremely thin objects demands new approaches for robust object detection. Additionally, minimizing the effect of perspective distortion, occlusion are open avenues for further work.

Acknowledgments

This research was conducted under and supported by a project from the Life Sciences Research Board, Defence Research and Development Organization titled 'Automated Cognitive Metrics Classification Framework to Improve Human Machine Collaboration' (Project number: DRD-1659-CSE) to Prof. Partha Pratim Roy.

Conflict of interest

The authors declare there is no conflict of interest.