A full convolutional network based on DenseNet for remote sensing scene classification

1.   Introduction

Remote sensing images has been applied to a broad range of fields such as land use classification ^[1,2], geographic image retrieval ^[3,4] and remote sensing scene classification. Remote sensing scene classification is an important work, which can provide some decision-making basis for the follow-up work of urban planning and environmental monitoring. Remote sensing scene classification divides remote sensing images into various categories such as forests, cities and rivers according to the semantic information of remote sensing images. Remote sensing images we focus on are obtained by aerial photography. These remote sensing images have a high spatial resolution, highly complex geometric structures and spatial patterns. However, there is high similarity between categories of remote sensing while images in the same scene category may have great differences. This makes it difficult to classify remote sensing scenes.

Multimedia information processing and communication are popular in our daily lives, and security and privacy protection techniques are becoming more and more important ^[5]. Remote sensing images are almost saved in the digital form, therefore they are possible to be tampered with. It should be noted that for some remote sensing scene images, modifying a small number of pixels can affect the classification results greatly, which may cause some risks. This has become another challenge in the classification of remote sensing scenes. Because of these challenges mentioned above, the classification models should be supposed to extract the semantic information from the image very well to achieve the better robustness.

Researchers pay more attention to the design of handcrafted features that are used for remote sensing scene classification before the development of deep learning. Those low-level features designed by the human is hard to represent semantic information of remote sensing scenes. The classification accuracy is unsatisfactory in the methods using artificial features. With the development of deep learning, deep learning has been applied to many research field ^[6,7,8,9]. The convolution neural network (CNN), as the most popular technology of deep learning, is introduced to remote sensing scene classification naturally. In general, the neural networks can be trained very well with a large amount of training data while they are often over-fitting and poor at generalization with a small amount of training data. To avoid or alleviate this situation, on the one hand, data enhancement technology can be used to expand the training data ^[10,11], on the other hand, a specially designed network also can make the training more efficient ^[12].

Some neural network architectures are applied to remote sensing scene classification, such as CaffeNet, GoogleNet, and VGGNet. Although many methods based on those networks improve classification accuracy significantly, they still have a bottle-neck. For these networks, we observe that:

1) These networks that are trained with small amounts of data have a large number of parameters. Note that this often results in over-fitting in convolutional neural networks.

2) These networks are shallow. Because of the complexity of remote sensing images, to better classify remote sensing scenes, classification network is supposed to extract high-level semantic features of images well. It is worth noting that deeper networks often can extract more abstract semantic information.

In view of the above observations, we propose a deeper network with fewer parameters to improve the classification accuracy of remote sensing scene. Based on the DenseNet, we built a network with deeper layers but fewer parameters than those networks mentioned above. Table 1 shows the comparison of parameters and layers of different networks. To increase the diversity of training data, we use data augmentation technology on training dataset, such as brightness, color, contrast, sharpness and rotation when training the network. All of the parameters of those augmentation transformations are generated randomly. In addition, we utilize pre-training parameters trained on ImageNet ^[13], which can accelerate training and overcome over-fitting. Moreover, we fix the length of the final feature vector that is used for classification by an adaptive average 3D pooling operation. Thus, the network can accept images of various sizes. To avoid the flattening operation and make the network operation more concise, we design a convolutional layer instead of fully connected layer to classify the features. Unlike many methods that use Support Vector Machines (SVM) as a classifier, we use the softmax as the probability mapping function, so our network is an end-to-end process without additional training on SVM.

Table 1.  Comparison of different network parameters and layers.

Network	Parameters (millions)	Convolution layers	Fully connected layers
CaffeNet	61.1	5	3
GoogleNet	6.7	22	1
VGG16	138.3	13	3
VGG19	143.6	16	3
ours	7.5	121	0

---

 | Show Table

DownLoad: CSV

2.   Related works

The methods of remote sensing scene classification are mainly based on three aspects at present: low-level visual features, middle-level visual features and high-level visual features ^[14].

Low-level visual features are used in many tasks, such as object tracking ^[15]. There are some frequently-used low-level visual features in the field of classification of remote sensing scenes, such as spectral feature, texture feature and structure feature. Those methods based on low-level visual features either extract feature vector from these low-level visual features or use a combination of feature vectors extracted from different low-level visual features to describe remote sensing images ^{[16,17,18,19]}. This means that such methods are based on the premise that remote sensing images can be described by a feature vector or a combination of different kinds of feature vectors. For some images whose scene are simple, one kind of feature can be used as a descriptor of remote sensing images ^[16,20]. Unlike using only one kind of feature, literatures ^[17,18,19] show that the combinations of different kind of features can improve the results. Specifically, ^[17] combine six different kinds of feature to form a multiple-feature representation, which describes the scene better. However, low-level visual features are only applicable to simple scenarios that have no complex structure. For complex scenarios, their expressive ability is limited ^[21].

To overcome the limitations of low-level visual features, many methods based on middle-level visual features are proposed. The main idea of such methods is to encode the extracted features such as SIFT, LBP and color histograms. Then further extract the more discriminant feature representation. Bag of Word (BoVW) model ^[22] and its related variants ^{[23,24,25,26,27]} are typical algorithms based on the middle-level visual features. The BoVW regards an image as a collection of visual words, and to construct new features that are used for image classification with the frequency of visual words. Specific steps are as follows: it first extracts SIFT features from all images, next constructs a dictionary using a clustering algorithm, and then counts the occurrence frequency of each word of the dictionary in the image. The features used as a descriptor of an image for classification is composed of these occurrence frequencies of each word of the dictionary. The features obtained by such methods are a further abstraction of low-level visual features, lacking flexibility and adaptability for different scenarios ^[14].

The third category of methods are based on high-level visual features. Such methods achieve the best accuracy comparing with the above two kinds methods, even far than algorithms based on low-level visual features ^{[28,29,30,31,32,33,34,35]}. As mentioned above, neither low-level visual features nor middle-level visual features can represent remote sensing images well and flexibly, which means that the features obtained by these two methods lack abstract semantic information. As we all know, CNN can extract abstract semantic information from images. Thus, those features extracted by CNN are made to be a good descriptor of remote sensing images. In addition, CNN can learn how to extract the high-level semantic information of an image by itself, which is very flexible and avoids complex design of artificial features.

At present, CNN models used to the classification of remote sensing scene mainly include CaffeNet, GoogleNet and VGGNet ^{[33,36,37,38,39]}. Usually, these networks are shallow with no more than 30 layers, and the number of their parameters is large, such as VGG16 that has about 138 million parameters. Chaib et al. ^[37] use a VGGNet to extract the features from remote sensing images, then the features from different layers are transformed by discriminant correlation analysis (DCA), the transformed features are fused to construct a new feature that is classified by SVM that is used as a classifier in other fields ^[40]. The main idea of literature ^[39] is also features fusion. Different from ^[37], Yu et al. ^[39] proposed two schemes. One is that using the original RGB network and the mapped LBP coded network extracts the features from RGB images and LBP feature maps respectively. The features from different network are fused by concatenation layer, and then the obtained features are classified after being processed by the fully connected layers. The steps of the other one are similar. The difference is that a saliency coded network is used instead of mapped LBP coded network. Wang Q et al. ^[38] introduced the attention mechanism into the classification of remote sensing scene by constructing a recurrent attention structure with recurrent neural networks (RNN). Different from the above methods, Cheng G et al. ^[36] focus on the loss function, and propose a new loss function by applying metric learning to CNN features. By combining proposed metric learning loss with cross-entropy loss, features extracted from the same scene class are as close as possible, while features extracted from different categories are as far as possible.

3.   Method description

As mentioned above, the CNN model generally used in remote sensing scene classification have huge number of parameters, or the depth of network layer is too shallow, which is easy to cause over-fitting of the model and inadequate feature extraction ability of the model. Aiming at the problem of network applied in remote sensing scene classification, we apply the DenseNet ^[41] to our architecture and construct a full convolution network for remote sensing scene classification. Based on cross-layer connectivity, our full convolutional network is a deep network architecture with only a few parameters. In addition, we introduce an adaptive average 3D pooling operation, which fixes the sizes of the output feature maps from 7 × 7 to 1 × 1 and decrease the number of channels from 1024 to 512. Our method can accept images that are different sizes. Furthermore, our network exploits a convolutional layer instead of full connection layer as a classification layer. Thus, features can be classified without flattening, which makes the network simpler and more efficient.

3.1.   The main idea of DenseNet

In 2017, Huang et al. ^[41] proposed DenseNet derived from ^[42]. In general, the network structures are progressively hierarchical. In such a network structure, the feature maps from the $(i - 1)$th layer are the input of the $i$th layer. Experiments ^[42] have shown that the input of the $i$th layer can be not only the output of the $(i - 1)$th layer, but also the $(i - {\text{2}})$th layer, or the $(i - n)$th layer ($n$ less than the number of layers). This may lead to a more generalizable network. The basic idea of DenseNet is that each layer in the network is directly connected to the front layers. The connection strategy in DenseNet is shown in Figure 1. It is worth noting that in order to ensure that the feature maps can be concatenated, the sizes of feature maps need to be consistent. This means that the outputs of the convolutional layer are of the same size as the input.

Figure 1.  Densely concatenated convolution. The input of each layer comes from the output of all previous layers, and the final output of the network fuses all feature maps produced by front layers.

DownLoad: Full-Size Img PowerPoint

The input of each layer is the output of all front layers in the densely concatenated convolutions:

${H_l}$ represents the $l$th layer, ${X_l}$ represents the output of the $l$th layer. In DenseNet, Batch Normalizing (BN) ^[43] is used to normalize the input of layer, which reduces the absolute difference between data and considers the relative difference more. The BN algorithm is described as the Algorithm 1.

Table Algorithm 1.  Batch Normalizing Transform.

Input: Values of x over a batch: $B = \{ {x_{1...m}}\}$; Parameters to be learned: $\gamma$, $\beta$
Output: $\{ {y_i} = B{N_{\gamma, \beta }}({x_i})\}$
(1) ${\mu _B} \leftarrow \frac{1}{m}\sum\limits_{i = 1}^m {{x_i}}$ (2) ${\sigma ^2}_B \leftarrow \frac{1}{m}\sum\limits_{i = 1}^m {({x_i}} - {\mu _B}{)^2}$ (3) $\mathop {{x_i}}\limits^ \wedge \leftarrow \frac{{{x_i} - {\mu _B}}}{{\sqrt {{\sigma ^2}_B + \varepsilon } }}$ (4) ${y_i} \leftarrow \gamma \mathop {{x_i}}\limits^ \wedge + \beta \equiv B{N_{\gamma, \beta }}({x_i})$

---

 | Show Table

DownLoad: CSV

The more details about BN you can get from ^[43]. In order to increase the nonlinearity of network, ReLU is used as the activation function, which is described as:

Compared with other networks, the network composed of Densenet has the following advantages:

1) Fewer parameters. The input of current layer of the network is composed of the feature maps from front layers, which results in the continuous accumulation of the number of feature maps. Thus, many reusable feature maps can be learned with very few convolutional kernels.

2) Strong ability to prevent over-fitting. Short paths are created from early to late layers because of the dense connection. Each layer receives additional supervision from loss function, which greatly reduces the gradient disappearance. Therefore, dense connection has a very good ability to resist over-fitting, which makes it especially suitable for applications where training data is relatively scarce.

3) Deeper layers. The network can be designed very deep by the dense connection, which only have a few parameters. For example, a network with 96 convolution layers can be designed while it only has 5.34 million parameters.

Experiments on ImageNet ^[13] demonstrate the efficiency of the dense connection ^[41]. Therefore, densely concatenated convolution can extract high-level features of remote sensing images. Based on this idea, we construct a full convolution network based on DenseNet ^[41] for remote sensing scene classification.

3.2.   Our network architecture

Our full convolutional network based on DenseNet is shown in Figure 2. In the whole network, it is divided into an initial layer, three transition layers and four densely connected blocks (DenseBlock). After the last DenseBlock, we exploit an adaptive average 3D pooling operation to fix number and size of feature maps and then classify them by a convolution with kernel size of 1 × 1. We describe our architecture in detail in session 3.3.

Figure 2.  Our full convolutional network for remote sensing scene classification. In our network, each densely connected block (DenseBlock) consists of several Tuple Operations (TO). Each TO is composed of Batch Normalizing (BN), ReLu, Conv(1 × 1) and Conv(3 × 3). Conv(1 × 1) represents a convolutional operation with kernel size of 1 × 1 while Conv(3 × 3) represents a convolutional operation with kernel size of 3 × 3. DenseBlock1 has 6 TOs, DenseBlock2 has 12 TOs, DenseBlock3 has 24 TOs and DenseBlock4 has 16 TOs.

DownLoad: Full-Size Img PowerPoint

As shown in Figure 2, the whole network architecture does not strictly follow the rule of dense connection, it mainly consists of densely connected blocks instead. The reason for this is that the whole network has several down-sampling operations on feature maps, which results in the inconsistency of sizes between feature maps in different stages, so that feature maps with different sizes cannot be spliced in the dimension of channel. Moreover, concatenation operation requires feature maps to be kept in memory. With the increase of network layers, feature maps kept in memory are superimposed, which causes huge memory consumption. If every layer of the network strictly follows the dense connection theory, the network will consume many resources, which makes the network stop working.

3.3.   Architecture detail

In this section, we detail the various parts of our network architecture.

Initial layer. We first extract 64 features using 64 convolutional kernels with kernel size of 7 × 7. We set the convolutional stride as 2, which reduces the sizes of the feature maps and reduces the computational complexity of subsequent operations. After the convolution, the operations of BN, Relu and MaxPool are performed in turn. For example, when the size of the input image is 224 × 224, the size of the image after padding is 230 × 230 and convolution kernels of size 7 × 7 are used for convolution operation, then we will get feature maps of size 112 × 112. Subsequently, these feature maps will go through the operations of BN, Relu and MaxPool in turn. Finally, the obtained feature maps are size of 56 × 56. The generated feature maps are used as input of the subsequent module of the network.

Densely connected block. Our full convolutional network mainly consists of densely connected blocks (DenseBlock). Every DenseBlock acts as one stage for extracting features and consists of several Tuple Operations (TO). As shown in the Figure 2, a TO is composed of Batch Normalizing (BN), ReLU, Conv(1 × 1) and Conv(3 × 3). In particular, in all densely connected blocks, the stride of Conv(1 × 1) is 1 and the number of the convolutional kernels is 128. Moreover, the stride of Conv(3 × 3) is 1 and the number of convolutional kernels is 32. To ensure the same size of the output and input, the padding of Conv(3 × 3) is set to 1. It is worth mentioning that the dense connection operations are strictly enforced between every TO in DenseBlock. The structure of dense connection is shown as Figure 1.

Transition layer. It is observed that, this layer consists of BN, Relu, Conv(1 × 1) and AvgPool. Feature maps pass through these operations in turn. After a DenseBlock, the output of each layer in this DenseBlock is kept in memory, resulting in a large number of feature maps. Note that if the number of these feature maps does not decrease, the network will be difficult to deepen and may stop working due to too much computation. Therefore we reduce the number of the feature maps to half by using a convolutional layer with a convolution kernel size of 1 × 1 after a DenseBlock, which can not only reduce the number of feature maps but also increase the depth and improves fitting ability of the network.

Adaptive average 3D pooling. After 3 transition layers and 4 densely connected blocks, the obtained feature map size is 1/32 of the original image, and the number of feature maps is 1024. To further reduce the computational consumption, we need to reduce the number and size of the feature map. Through the adaptive average 3D pooling operation, the number of feature maps is reduced to 512 and the size becomes 1 × 1 without increasing the network parameters. As shown in Figure 2. In addition, this operation ensures that the size of output feature maps is 1 × 1 and the number is 512, which provides a fixed size feature map for the later convolutional classifier and makes the network accept images with different sizes. As shown in Figure 3.

Figure 3.  Our network can accept images of different sizes. After extracting the features from input images of sizes 3 × 224 × 224, 3 × 280 × 280, 3 × 300 × 300 by previous layers, the obtained feature map sizes are 1024 × 7 × 7, 1024 × 8 × 8, 1024 × 9 × 9 respectively. The sizes of the feature maps will be 512 × 1 × 1 after the adaptive average 3D pooling operation. Adaptive average 3D pooling operation produces a fixed size feature map for the later convolution classifier.

DownLoad: Full-Size Img PowerPoint

Convolution classifier. After adaptive average 3D pooling, a 512-dim feature representation is obtained. Then we exploit a 1 × 1 convolution to classify the 512-dim features and get the final category feature representation. Finally, we exploit the softmax function to perform the probability mapping.

Softmax function. This function maps the final category feature representation into a probability for every class. The function is defined as:

${X_c} \in v$, ${X_c}$ represents the ${c_{th}}$ number of vector $v$ that comes from convolution classifier.

3.4.   Loss function

In many classification tasks, the loss function is designed as:

In equation (4), ${h_\theta }$ is defined as mapping function of the network with parameters $\theta$, $P(c|{h_\theta }({x_i}))$ indicates the probability of real category for the $i$th sample, and $n$ is the number of samples used for one epoch. In order to prevent over-fitting, we normalize the network parameters by L2-norm.

In equation (5), ${n_\theta }$ represents the number of parameters $\theta$ of the network. Our ultimate loss function with L2-norm becomes:

In equation (6), $\lambda$ is defined as weight decay rate. In our experiment, we set $\lambda$ to 5×10^-4.

3.5.   Parameter optimization strategy

There are many algorithms that can be used to optimize the parameters of neural network, such as genetic algorithm ^[44] and gradient descent algorithm. It is worth noting that the gradient descent algorithm, the most commonly used algorithm in deep learning, is used to optimize the parameters of the network in our experiments. For gradient descent algorithm, there are various optimizers can be selected, such as Adam and SGD. According to our experience, SGD usually takes much more time to train, but the results are more reliable in the case of a good initialization and learning rate scheduling scheme. Because we use pre-training parameters, which satisfies the condition having a good initialization, so we use SGD as the optimization strategy for training network. For general SGD, the parameters $\theta$ of the network are updated as follows:

In equation (7), the gradient ${\nabla _\theta }J(\theta)$ is defined as:

In equation (7), $\eta$ is defined as the learning rate. One disadvantage of this type of SGD is that its update is very unstable because its update direction is completely dependent on the current batch. A simple way to solve this problem is to introduce the momentum. Momentum simulates the inertia of an object when it is moving, that is, it retains the direction of the previous update to a certain extent when updating, and uses the gradient of the current batch to fine tune the final update direction. In this way, stability can be increased to a certain extent, so that the learning is faster, and there is a certain ability to get rid of the local optimum. For SGD with momentum, the parameters $\theta$ of the network are updated as follows:

In equation (8) and (9), $m$ and $v'$ denote momentum and velocity respectively. In our experiment, we set $m$ to 0.9.

4.   Evaluation

4.1.   Evaluation indicators

We adopt confusion matrix and overall classification accuracy, which are widely used in classification tasks to evaluate the effectiveness of our algorithm.

4.1.1.   Confusion matrix

In the confusion matrix, each row represents the actual category, and each column represents the predicted category. The number of classes correctly classified and the number of classes misclassified can be clearly seen by the confusion matrix.

4.1.2.   Overall accuracy

Overall accuracy is the ratio of the number of models correctly predicted on all test sets to the total number of datasets. The overall accuracy can generally characterize the classification accuracy of model.

$TP$ represents true samples that are correctly classified by the model. $FN$ represents positive samples that are misclassified by the model. $FP$ represents negative samples that are misclassified by the model. $TN$ represents negative samples that are correctly classified by the model.

4.2.   Evaluation dataset

In order to verify the effectiveness of our algorithm, we have done a lot of experiments. We experiment on four data sets: UC Merced Land Use Dataset ^[22], Aerial Image Dataset ^[14], OPTIMAL-31 Dataset ^[38] and NWPU-RESISC45 ^[45] Dataset. These datasets contain a lot of images and cover a lot of categories. In addition, images of these datasets are pretty complex and have some similarities among different scenes, which make these datasets challenging. Some pictures of these datasets are shown in Figure 4.

Figure 4.  Some samples of AID, UCM, OPTIMAL-31, NWPU-RESISC45 datasets. From these samples, we can see that remote sensing scene image scene is complex and contains a lot of interference information, so remote sensing scene classification is a challenging task.

DownLoad: Full-Size Img PowerPoint

4.2.1.   UC Merced Land-Use Dataset ^[22]

UC Merced Land-Use Dataset (UCM) is a scene classification dataset containing 21 categories. This dataset includes agricultural, golf course, medium density residential, airplane, baseball diamond, beach, buildings, chaparral, storage tanks, dense residential, overpass, freeway, harbor, intersection, mobile home park, parking lot, river, runway, forest, sparse residential, and tennis courts. There are 100 RGB color images of the size 256 × 256 and 30 cm spatial resolution for each category.

4.2.2.   Aerial Image Dataset ^[14]

Aerial Image Dataset (AID) includes bare land, dense residential, baseball field, bridge, center, church, railway station, commercial, viaduct, desert, storage tanks, farmland, forest, industrial, meadow, medium residential, beach, mountain, park, parking, playground, pond, port, resort, river, school, square, stadium, sparse residential and airport. This data set has 30 categories and 10000 images in all. Pixel resolution of each picture is 600 × 600.

4.2.3.   OPTIMAL-31 Dataset

A remote sensing scene classification dataset named OPTIMAL-31 constructed by Qi Wang et al ^[38]. The dataset consists of 31 classes, each of which consists of 60 images with 256 × 256 pixels. It has 1860 pictures in total. The categories of the whole dataset are playground, airplane, airport, crossroads, runway, basketball court, bridge, parking lot, lake, bushes, church, beach, mountain, round farmland, business district, roundabout, dense houses, desert, forest, freeway, golf field, harbor, factory, island, meadow, medium houses, mobile house area, overpass, railway, square farmland, and baseball field. The whole dataset contains many categories, but the number of each category is small, so the dataset is more challenging.

4.2.4.   NWPU-RESISC45 Dataset ^[45]

The NWPU-RESISC45 dataset covers 45 scene classes, and each category has 700 images. The 45 scene classes include chaparral, airplane, airport, baseball, diamond, golf course, basketball court, beach, freeway, bridge, church, circular farmland, cloud, commercial area, rectangular farmland, river, dense residential, roundabout, desert, forest, palace, ground track field, harbor, sparse residential, industrial area, thermal power station, intersection, sea ice, island, lake, meadow, medium residential, mobile home park, storage tank, mountain, overpass, parking lot, railway, railway station, runway, ship, snowberg, stadium, tennis court, terrace and wetland.

5.   Experiments

In order to evaluate the effectiveness of our network, we train and test our method on UCM, AID, OPTIMAL-31 and NWPU-RESISC45. Each dataset is divided into two parts in our experiments, one for training and the other for testing. We design seven experiments in all to evaluate our network. The ratios of training and testing on each data set are as shown in Table 2. In each experiment, we randomly generate training set and testing set to ensure the validity of the experiment. Note that the training set is only used for training, while the testing set is only used for testing.

Table 2.  The setting of training and testing scales on each data.

Dataset	Train	Test
UCM	80%	20%
	50%	50%
AID	20%	80%
	50%	50%
NWPU-RESISC45	10%	90%
	20%	80%
OPTIMAL-31	80%	20%

---

 | Show Table

DownLoad: CSV

5.1.   The hyper parameter setting and the environment in experiments

In our experiments, we set train epoch to 400, each epoch will train all images of a training set. The batch size of each epoch is set to 80. After each two epochs, we will evaluate our network on the testing set once. To reduce memory usage, we resize the images used for training to the size of 224 × 224. Ones more, we initialize our network with pre-trained parameters trained on the ImageNet. Further, we use data augmentation technology to train a better model. All experiments are performed on a GPU server. The configurations of server are as follow: CPU Intel (R) Xeon (R) E5-2670 v3,128 GB of memory, GeForce GTX TITAN X, 12GB of display memory. The experimental framework is pytorch0.4.1.

5.2.   Reports of confusion matrixes

The following four obfuscation matrices are obtained by testing our network on AID, UCM, OPTIMAL and NWPU-RESISC45, respectively.

From Figure 5–8, we can see that most images can be correctly classified in our experiments. Especially in UCM dataset, only two images are misclassified. Among these experiments, the overall accuracy of NWPU-RESISC is the worst, only 94.98%. This is because there are many categories in the dataset while samples for training are few. For the case of misclassification, there are three main reasons.

Figure 5.  The confusion matrix with AID, 50% of dataset for training and 50% of the dataset for testing. The overall accuracy is 97.44%.

DownLoad: Full-Size Img PowerPoint

Figure 6.  The confusion matrix with UCM, 80% of the dataset for training and 20% of the dataset for testing. The overall accuracy is 99.50%.

DownLoad: Full-Size Img PowerPoint

Figure 7.  The confusion matrix with OPTIMAL, 80% of the dataset for training and 20% of the dataset for testing. The overall accuracy is 95.89%.

DownLoad: Full-Size Img PowerPoint

Figure 8.  The confusion matrix with NWPU-RESISC45, 20% of the dataset for training and 80% of the dataset for testing. The overall accuracy is 94.98%.

DownLoad: Full-Size Img PowerPoint

1) The appearance is similar. The appearance of the two images shown in Figure 9(a) is very similar.

Figure 9.  The cases of misclassification. In our experiment, the palace is misclassified into church easily, rectangular farmland is misclassified into terrace and wetland is misclassified.

DownLoad: Full-Size Img PowerPoint

2) Texture is similar. As shown in Figure 9(b), two images are divided by roads, which causes a similar texture structure.

3) One category may contain information of other categories. As shown in Figure 9(c), there is a lake-like area in the wetland, it is disturbed when wetland image is classified.

5.3.   Reports of experimental comparisons

To evaluate the performance of our algorithm, we compare it with some algorithms in recent years. From Table 3 to 6, we report the comparison results on UCM, AID, OPTIMAL-31 and NWPU-RESISC45 respectively.

Table 3.  Overall accuracies and standard deviations (%) of different feature fusion methods on the UCM dataset.

Method	Training Ratios (%)
	50%	80%
MS-CLBP + FV [46]	88.76 ± 0.76	93.00 ± 1.20
GoogLeNet [14]	92.70 ± 0.60	94.31 ± 0.89
CaffeNet [14]	93.98 ± 0.67	95.02 ± 0.81
VGG-VD-16 [14]	94.14 ± 0.69	95.21 ± 1.20
SalM3LBPCLM [47]	94.21 ± 0.75	95.75 ± 0.80
TEX-TS-Net(feature fusion model) [39]	97.55 ± 0.46	98.40 ± 0.76
SAL-TS-Net(feature fusion model) [39]	97.79 ± 0.56	98.90 ± 0.95
ARCNet-VGG16 [38]	96.81 ± 0.14	99.12 ± 0.40
ours	98.67 ± 0.42	99.50 ± 0.25

---

 | Show Table

DownLoad: CSV

Table 4.  Overall accuracies and standard deviations (%) of different feature fusion methods on the AID dataset.

Method	Training Ratios (%)
	20%	50%
CaffeNet [14]	86.86 ± 0.47	89.53 ± 0.31
VGG-VD-16 [14]	86.59 ± 0.29	89.64 ± 0.36
GoogLeNet [14]	83.44 ± 0.40	86.39 ± 0.55
ARCNet-VGG16 [38]	88.75 ± 0.40	93.10 ± 0.55
D-CNN with GoogLeNet [36]	88.79 ± 0.10	96.22 ± 0.10
D-CNN with VGGNet-16 [36]	90.82 ± 0.16	96.89 ± 0.10
TEX-TS-Net(feature fusion model) [39]	93.31 ± 0.11	95.17 ± 0.21
SAL-TS-Net(feature fusion model) [39]	94.09 ± 0.34	95.99 ± 0.35
ours	95.37 ± 0.32	97.19 ± 0.23

---

 | Show Table

DownLoad: CSV

Table 5.  Overall accuracies and standard deviations (%) of different feature fusion methods on the OPTIMAL-31 dataset.

Method	Training Ratios (%)
	80%
Fine-tuning AlexNet [38]	81.22 ± 0.19
Fine-tuning GoogLeNet [38]	82.57 ± 0.12
Fine-tuning VGGNet16 [38]	87.45 ± 0.45
VGG-VD-16 [14]	89.12 ± 0.35
ARCNet-Alexnet [38]	85.75 ± 0.35
ARCNet-ResNet34 [38]	91.28 ± 0.45
ARCNet-VGGNet16 [38]	92.70 ± 0.35
ours	95.41 ± 0.61

---

 | Show Table

DownLoad: CSV

Table 6.  Overall accuracies and standard deviations (%) of different feature fusion methods on the NWPU-RESISC45 dataset.

Method	Training Ratios (%)
	10%	20%
TEX-TS-Net(feature fusion model) [39]	84.77 ± 0.24	86.36 ± 0.19
SAL-TS-Net(feature fusion model) [39]	85.02 ± 0.25	87.01 ± 0.19
D-CNN with AlexNet [36]	85.56 ± 0.20	87.24 ± 0.12
D-CNN with GoogLeNet [36]	86.89 ± 0.10	90.49 ± 0.15
D-CNN with VGGNet-16 [36]	89.22 ± 0.50	91.89 ± 0.22
ours	92.90 ± 0.17	94.95 ± 0.04

---

 | Show Table

DownLoad: CSV

From the Table 3–6, we can get the following conclusions.

1) CNN regard as an advanced feature extractor is suitable for classification of remote sensing scenes, and it has achieved a good accuracy. As shown in Table 3, with 80% of dataset for training, the accuracy of ARCNet-VGG16 ^[38] is 99.12 ± 0.40, while our algorithm achieves 99.50 ± 0.25, which improves by about 0.38. It is worth noting that the accuracy of those methods based on CNN are very close to 100% in many datasets. This fully demonstrates the efficiency of convolutional neural networks.

2) In contrast to various algorithms, our algorithm gets a state-of-art recognition rate. As shown in Table 3, we are 0.88 and 0.38 higher than the best algorithms at the training ratios 50% and 80% respectively. As shown in Table 4, we are 1.28 and 0.3 higher than the best algorithm at the training ratios 20% and 50% respectively. As shown in Table 5, we are 2.71 higher than the best algorithm at the training ratio 80%. As shown in Table 6, we are 3.68 and 3.06 higher than the best algorithm at the training ratios 10% and 20% respectively. This fully demonstrates the superiority of our algorithm and the excellent ability to extract features by dense connections.

3) For CNN, the more data used for training, the better results for test. The results are reported in Table 3 show that whether it is our method or other deep learning models, the training ratio used is different, and the accuracy of the test is different. It is worth noting that as the training ratio increases, the test accuracy is also improved.

5.4.   Experimental skills

Literature ^[33] shows that making full use of the pre-training model can improve the classification accuracy of the network. In our work, one of the keys to improve the classification accuracy of our network is to initialize network with the pre-training parameters trained on ImageNet ^[13]. In our experiments, the parameters of the whole network (except the parameters of the convolution classifier) are trained on ImageNet first, and then are further fine-tuned on the remote sensing dataset. The other key point for our network to classify remote sensing images well is using data augmentation technology. As we all know, big data is the key to the success of deep learning. However, in the field of remote sensing, the number of images in the dataset is often small, because the images are difficult to obtain, and it is time-consuming to manually mark the corresponding scene categories. To solve this problem, data augmentation technology is introduced. In our experiments, we performed color transformation, brightness transformation, contrast transformation, and sharpness transformation on images. Besides those transformations, we rotated the image at random angles. The effect of data augmentation is shown in Figure 11. To demonstrate the effectiveness of pre-training models and data augmentation technique, we conduct experiments on various datasets. The report is shown in Figure 10. Note that when the network does not be initialized with pre-training parameters and does not use the data augmentation technology the classification accuracy of the network is very poor. But the accuracy will be greatly improved when using the pre-training parameters and the data augmentation technology. Thus, using pre-training parameters and data augmentation techniques is effective for classification and may lead to improved classification accuracy.

Figure 10.  The experimental results of whether to use the pre-training parameters and data augmentation technology on dataset. PMN_DEN denotes that pre-training parameters and data augmentation technique are not both used. PMN_DEY denotes that pre-training parameters is not used but a data augmentation technique is used. PMY_DEN denotes that pre-training parameters is used but a data augmentation technique is not used. PMY_DEY denotes that pre-training parameters and data augmentation technique are both used. The datasets are split as follows: For OPTIMAL, 80% of dataset are treated as training set, while 20% are treated as testing set. For AID 50% of dataset are treated as training set, while the 50% are treated as testing set. For NWPU-RESISC45, 10% of dataset are treated as training set, while the 90% are treated as testing set. For UCM, 80% of dataset are treated as training set, while the 20% are treated as testing set.

DownLoad: Full-Size Img PowerPoint

Figure 11.  The category of the original picture is a forest. The pictures A, B, C, D and E obtained are changed by brightness, color, contrast, rotate and sharpness, respectively.

DownLoad: Full-Size Img PowerPoint

6.   Conclusion

In this paper, we exploit the advantages of densely concatenated convolution to propose a full convolutional network for remote sensing scene classification. Our network is deep, but the parameters are few. As shown in Figure 2, these TOs directly contribute to network depth. The numbers of convolutional kernels of Conv(3 × 3) and Conv(1 × 1) are 32 and 128 respectively in DenseBlock, which produces few parameters, this is the reason that our network has few parameters. We use an adaptive average 3D pooling operation to down-sample the feature maps of the network output. The number of channel decrease to 512, which reduces the calculated consumption for the latter operations. And the size of the feature map is fixed, so that network can accept images with different sizes.

The experiment results on the several common datasets show that our fully convolutional network gets a state-of-art recognition rate.

Acknowledgments

The research work was funded by the National Natural Science Foundation of China grants nos. 61772454, 61811540410 and 61811530332, the Scientific Research Fund of Hunan Provincial Education Department Grant no. 16A008, the Industry-University Cooperation and Collaborative Education Project of Department of Higher Education of Ministry of Education Grant no. 201702137008, the Postgraduate Scientific Research Innovation Fund of Hunan Province Grant no. CX2018B565, the Undergraduate Inquiry Learning and Innovative Experimental Fund of CSUST Grant no. 2018-6-119, and the Postgraduate Training Innovation Base Construction Project of Hunan Province Grant no. 2017-451-30.

Conflict of interest

The authors declare no conflict of interest.