Perceptual loss function for generating high-resolution climate data

1.   Introduction

A wide range of research areas can benefit from climatology data on a variety of spatial and temporal scales ^[1,2]. For example, in terms of temporal scales, short-term precipitation data can be used for flood forecasting, and long-term precipitation patterns can be utilized for urban planning and policy development. For spatial scales, it is possible to understand the characteristics of temperature distribution on a global scale from large-scale temperature data, whereas small-scale solar data can provide information on the pattern of solar energy resource distribution at the regional level ^[3]. Analyzing small-scale climate data is particularly challenging since the data needs to be at a high resolution. Global climate model (GCM) is one of the most commonly used sources of climate data in current studies ^[4], from which climate data is provided at a coarse spatial scale. Additionally, because scale-specific physics, computational resources, and time frames are customized into numerical models for each application, simulating all desired scales is generally intractable.

There are various types of interpolation methods available for the generation of high-resolution (HR) climate data from corresponding low-resolution (LR) data, but interpolation tends to smooth out the resulting high-resolution data, which removes texture and sharp edge information on small scales ^[5]. For this, recently interest in applying deep learning models, which originally were developed for computer vision, to generate high-resolution climate data has increased. Two examples of such models are super-resolution convolutional neural network (SRCNN) ^[6] and super-resolution generative adversarial network (SRGAN) ^[7]. While many studies have demonstrated superior performances by SRCNN and SRGAN to solve climate-related problems ^[8,9,10], recent developments in deep neural networks suggest new thinking and a huge potential in climate research. However, despite their impressive results, the loss function in such networks is pixel-wise where the mean squared error (MSE) is used to measure the difference between a network output and the ground truth. One disadvantage of pixel-wise loss functions is that they lead to blurry and over-smoothed HR data by forcing the model to generate a result which is the average of all plausible solutions ^[11]. When an algorithm tries to minimize per pixel MSE, it overlooks subtle textures crucial to human perception.

In comparing two images, human vision does not compare pixels by pixels, but rather extracts and compares features. Inspired by this idea, more recently, perceptual loss functions have shown significant improvements in computer vision for dealing with HR image data ^[7,12,13]. Pixel-wise loss is calculated by directly comparing the high-resolution results generated by the model with the true values. In contrast to per-pixel loss functions, perceptual loss calculates the loss between the simulated and true values in the feature space, which are extracted by using pre-trained feature extractors such as very deep convolutional networks, for example, VGGs ^[14]. A pre-trained feature extractor is the basis of perceptual loss. In the early layers of the network, each neuron is connected to a smaller receptive field resulting in the extraction of predominantly low-level spatial information, which is primarily the detailed information of the input. As the network goes deeper, the receptive field gradually increases, and the information extracted in the deeper layer changes from low-level spatial information to global semantic information, while fine-grained spatial details are discarded ^[15,16]. By comparing model results with high-level information of ground truth, such as content and global structure, perceptual loss model results and ground truth are perceptually close. As a result, the problem of over-smoothing caused by per-pixel loss functions is resolved.

The idea of perceptual loss has been applied not only to image problems, but also to solve audio pattern representation ^[17]. In terms of generating high-resolution climate data, the application of a perceptual loss function is, however, limited. Climate data are typically presented as raster type of data, with each raster representing the average value of a particular climate variable within the raster. Since feature extractors used in perceptual loss are usually trained with natural images, it is unknown whether they are suitable for extracting features from raster type of climate data ^[8]. The deep layer of the neural network in pre-trained feature extractors contains all the semantic information in the input ^[18]. This may not be reliable for climate data. However, recent studies have demonstrated that pretrained models have more extraction power based on the network structure than on the trained weights. A multilayer CNN may be used to capture the dependencies between statistics of variables at multiple levels without the need for any learning ^[19]. Due to this insight, the limitation of pre-training and specific network structures (typically VGG), for which perceptual loss has been used, is weakened, thereby allowing a broader range of applications to be analyzed.

Perceptual loss, especially VGGs-based perceptual loss, has been widely demonstrated in computer vision research to produce more realistic results compared to per-pixel loss, especially in solving image smoothing issues. However, the application of perceptual loss is currently limited to the image domain, and to our knowledge there are no relevant studies to verify the performance of perceptual loss for raster types of climate data. To fill this gap, in this study, we investigate whether perceptual loss which is trained on natural images using a pre-trained feature extractor could be used to generate high-resolution climate data. For this, we first verify the impact of using different layers of feature maps to calculate perceptual loss in generating high-resolution climate data and investigate its performance in CNN and GAN both qualitatively and quantitatively. According to the results of the study, low- and medium-level feature maps can provide comparable results compared to per-pixel loss with greater detailed information and relatively larger root mean square error (RMSE). Based on the understanding of the results from different levels of feature maps, we propose a new method to benefit from the traditional perceptual loss. Using the proposed method, we demonstrate that it can obtain good detailed information while losing less general accuracy than calculating perceptual loss directly for the whole sample.

The structure of the paper is as follows: Section 2 introduces some related works about generating HR climate data and perceptual loss. Section 3 presents the methodology, and section 4 presents the experimental setup. Section 5 describes the results. The paper concludes with a discussion in Section 6 and concluding remarks in Section 7.

2.   Related works

In this section, we review relevant approaches that are used to generate HR climate data and perceptual loss in image related studies.

Climate data are downscaled by means of dynamical and statistical approaches to mitigate the lack of spatial resolution. The process of dynamic downscaling, also called regional climate models (RCMs), is driven by low-resolution data with large-scale data as boundary conditions and result in high-resolution data with small-scale climate projections. Dynamic downscaling is not transferable across regions and is computationally demanding. On the contrary, statistical downscaling obtains HR climate data by establishing a statistical relationship between HR and LR pairs. In past studies, linear regression models ^[20], support vector machines ^[21], and random forests ^[22] have been applied to establish statistical relationships between HR and LR data. Considering that climate data and images are both in raster form, statistical downscaling is similar to image super-resolution issues. It is becoming increasingly common to use models designed to handle super-resolution image in order to generate high-resolution climate data. For example, super-resolution CNN (SRCNN) was the first deep learning model to be applied to generate high-resolution climate data ^[23]. ^[10] also proposed a deep CNN, which successfully improved the resolution of precipitation data by a factor of 5. By referencing the residual dense block (RDB) to the super-resolution network, ^[24] was able to improve the performance of the HR data generated. In these works, the models were optimized by using pixel-wise loss functions. Adversarial training has also been applied in some studies. For example, ^[25] studied to determine whether Wasserstein generative adversarial networks (WGANs) could generate realistic weather conditions when trained by climate data of a general circulation model (GCM). The result of super-resolution generative adversarial network (SRGAN) showed an increase in resolution of climate data by a factor of 50 ^[8].

To generate realistic HR images, perceptual loss has been proposed as a solution to the problem of over-smoothing and lack of details in HR images based on the idea of perceptual similarity ^[7,12,26]. Perceptual loss, as opposed to pixel-wise loss, focuses on comparing the output with the ground truth in the feature domain, which is extracted by pre-trained networks such as VGG16 and VGG19 as commonly used pre-trained models ^[12,14]. Typically, perceptual loss is specified as the loss of one or several specific feature layers in a VGG network, and the loss depends on the application ^[27]. Visualizations of deep CNNs indicate that convolutional features at different levels provide different perspectives on objects and their surroundings ^[28]. For example, ReLU5_4 is chosen as a high-level feature layer in ^[7] and ^[29], while ReLU2_2 is selected as low-level feature layer in ^[30]. Instead of relying on one or multiple identified layers, some studies calculate the perceptual loss based on all feature layers ^[27]. A number of other studies divide the target image into several parts and calculate the loss of each part by using different feature layers ^[13,31]. Perceptual loss has also been applied to studies related to remote sensing images ^[32,33].

3.   Methodology

3.1.   Structure of SRGAN

We require neural networks for this work in order to transform low-resolution climate data into high-resolution climate data. In light of the fact that previous studies have demonstrated that adversarial loss can lead to improved results when applied to high-resolution climate data ^[8], we compare pixel-wise loss and perpetual loss by using the SRGAN and validate the proposed climatological binary perceptual loss (using a mask of 1 and 0). SRGAN follows the same structure as ^[7] where the networks are deep and fully convolutional neural networks with 16 residual blocks and skip connections. All convolutional kernels are 3 × 3 and are followed by rectified linear unit activation functions. In contrast to the three-layer RGB channel of image data, as we only have one channel of climate input data, the input to the model is modified to one channel. The network consists of two parts: 1) a generator G and 2) a discriminator D. Generator G converts LR input into HR output. Discriminator D attempts to distinguish between real and fake HR output. Learning between the two networks forces G to produce more realistic HR output, whereas D is able to distinguish between real and generated HR data more accurately. In the discriminator network, the loss function is defined as follows:

where ${X}_{LR}$ is the LR input data and ${Y}_{HR}$ is the HR target. Generator's loss function consists of two main components:

where $\alpha$ is a weight. The adversarial loss can be calculated as follow:

This adversarial loss captures how effective the generator network is at fooling the discriminator. By comparing the generated data with the actual HR data, the content loss ${\cal{L}}_{content}$ is effectively conditioning the output of the generator network and we calculate it in the following three forms: ${\cal{L}}_{MSE}$, ${\cal{L}}_{VGG}$, and ${\cal{L}}_{CTPL}$.

3.2.   Content loss

3.2.1.   Pixel-wise loss

The mean squared error (MSE) is used in this study to calculate the pixel-wise loss. This is currently the most widely used optimization target in generating high-resolution climate data. The pixel-wise MSE loss is calculated as follows:

where $W$ and $H$ are the width and height of ${Y}_{HR}$. Pixel-wise loss calculates the distance between ${Y}_{HR}$ and $G\left({X}_{LR}\right)$ in the data field. With the MSE loss, the network attempts to find pixel-wise averages of plausible solutions, resulting in poor perceptual qualities and a lack of high-frequency details.

3.2.2.   Perceptual loss

Instead of comparing ${Y}_{HR}$ and $G\left({X}_{LR}\right)$ directly from the data field, perceptual loss aims to compare ${l}_{2}$ distance (Euclidean Distance) between the two in the feature domain. In this study, the feature domain is constructed by using the VGG19 network. VGG19 consists of 19 hidden layers, including 16 convolutional layers and 3 fully connected layers. As mentioned earlier, early layers can return low-level detailed spatial information, such as edges and speckles. Deeper layers of the network can return mid-level information, such as texture, and the final layers can return high-level features, which are indicative of global semantic understanding. The perceptual loss is calculated as follows:

where ${\phi }_{i, j}$ indicates the feature map obtained by the ${j}_{th}$ convolution (before activation) within the VGG19 network prior to the ${i}_{th}$ max pooling layer. Within the VGG network, ${W}_{i, j}$ and ${H}_{i, j}$ describe the dimensions of the corresponding feature maps. In order to investigate the effect of different levels of information, i.e., different depths of feature maps on the result, perceptual loss is calculated by using the feature maps obtained by conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1, respectively.

3.2.3.   Climatological targeted perceptual loss

Perceptual loss is computed by selecting different depths of feature maps. In image-related studies, most studies choose one single-layer feature maps to calculate the perceptual loss of the whole image. Application of this technique to an entire image without considering semantic information limits its utility. Calculation of the perceptual loss of the whole image by a low-level feature map may cause the network to give unnecessary penalties for medium-level information, such as texture, resulting in a loss of informative features. In contrast, by using high-level feature maps to calculate perceptual loss, the model may lose some fine-grained spatial details, such as sharp edges. In contrast to images, samples of climate data often contain limited information. In the case of the precipitation data used in this paper, most regions in the sample have no precipitation values or relatively small precipitation values, which means that most regional grid values remain stable and do not show a gradient. Only a few areas in the sample receive a significant amount of precipitation, and these areas have a gradient. It is possible for the network to add useless information to those stable regions if the perceptual loss is calculated for the entire sample. In addition, these stable regions may have an impact on the feature map of regions with precipitation. On this basis, our climatological binary perceptual loss is calculated as follows:

where $\circ$ denotes element-wise multiplication.

Step 1. The first step is to calculate the MSE for the entire sample in order to ensure that the generated HR results and the ground truth remain similar.

Step 2. A certain sample of climate data usually has stable values across larger areas without gradient variation. As an example, in the precipitation data used in the paper, most regions in the sample have 0 or very low values of precipitation. Calculating the perceptual loss for these regions may insert details into the generated HR that do not match the actual situation, which increases the error. Meanwhile, using perceived loss for those regions with higher precipitation helps to include more details in the generated HR. There is a general requirement for different loss functions in different regions. In this step, a mask ${M}_{1}$ is generated to which the target region corresponds a raster value of 1 and the other regions correspond to a raster value of 0. The target region can be defined as one of the following three scenarios: 1) a threshold lower limit, the region in the original sample that is larger than this value is the target region; 2) a threshold upper limit, the region in the original sample that is smaller than this value is the target region; 3) a threshold interval, the region in the original sample that is within this interval is the target region. In our experiment, most areas in the sample have relatively small precipitation, and only a limited number of areas have large precipitation. Therefore, ${M}_{1}$ represents scenario 1 in the experiment, with a threshold of 75^th percentile for precipitation values in each sample.

Step 3. In the third step, we first multiply the ground truth and the generated HR data with mask separately using element-wise multiplication. After multiplication only the target region retains the corresponding value in the obtained result, while the other regions have the value of zero. Then the ground truth and the generated HR data after multiplication are used as input to calculate the perceptual loss. As the same mask is used on both the ground truth and the generated HR data, the feature distance only reflects the difference between target regions. VGG has three input channels, as the ground truth and the generated HR data are duplicated in three layers as inputs for perceptual loss.

4.   Experimental setup

In this work, we first study the effect of perceptual loss on the generation of high-resolution climate data by using Global Precipitation Measurement (GPM) data. GPM is an international network of satellites in collaboration with the National Aeronautics and Space Administration (NASA) and the Japan Aerospace Exploration Agency (JAXA), which provides the next generation of global observations of precipitation and snow ^[34,35]. GPM can provide precipitation data with a maximum resolution of 0.1° and has different versions, and in this study we used GPM_3IMERGDF, which is daily precipitation data that combines microwave-IR estimates with gauge calibration. We chose three patches (106.96°W-126.95°W, 25.05°N-45.05°N; 86.85°W-106.85°W, 25.05°N-45.05°N; 66.75°W-86.75°W, 25.05°N-45.05°N).

Figure 1 shows the location of each patch. Each patch has daily precipitation data from 06/01/2000 to 08/31/2021, with a total of 7756 samples. This provides us with a total of 23268 samples with a dimension of 200 × 200. Since ground truth is required when we evaluate different models, we consider the raw resolution of GPM precipitation data as our HR data and ground truth. We study the results of perceptual loss with improved resolution of $\times$4 and $\times$8, where we downsample the raw GPM data by a factor of 4 and 8, respectively, using an average sample pooling of 4 × 4 rasters and 8 × 8 rasters. The coarsened 0.4° × 0.4° resolution data (50 × 50) and 0.8° × 0.8° resolution data (50 × 50) are used as the LR input for all the experiments; 10% of the training examples are held out for testing in each case.

Daily precipitation dataset, by the GFDL-ESM4 model from Coupled Model Intercomparison Project (CMIP6), was obtained to verify the results of the proposed method applied to GCM data ^[36]. We used daily precipitation starting from January 2022 to January 2023 with a spatial resolution of 1° latitude ×1.25° longitude under the scenario of SSP1-2.6. For the purpose of minimizing distortion near the poles, we only use data from latitudes $\pm$60°. For validation, we used the RMSE and Pearson's correlation to evaluate the overall quantitative performance of the model.

For training, we first trained the generator network alone for 1000 loops. This step, on the one hand, enabled the generator to learn the mapping relationship from LR to HR roughly before adversarial learning, which is a warm-up process before adversarial learning and helps to improve the results of GANs. In addition, the loss function of this process did not include adversarial loss, and we were able to compare the difference between pixel-wise loss and perceptual loss in the CNN training process. After the warm-up of the generator network iwas completed, we continued the training of GANs. The generator and discriminator were trained sequentially with 500 loops with 0.001 as the value of $\alpha$. The loss functions used in each network are summarized in Table 1.

5.   Results

5.1.   VGG19 feature extractor

Given that the VGG19 network is trained on natural images, its performance on climate features extracted from other raster types is unknown. For comparison, the feature maps obtained by VGG19 are shown in Figure 2 along with the feature maps obtained by one natural image in corresponding layers. It is difficult to conduct a quantitative analysis of these feature maps, however, we can see that, similar to the results on natural images, conv1_1 pays more attention to detailed climate information, such as boundaries and edges. As the layers deepen, the information presented by the feature maps becomes more and more abstract, implying that the deeper layers pay attention to the global semantic information. Additionally, we find that the average percentage of neurons that are activated in each layer is similar in both climate data and natural images. As an example, the activation percentages for conv1_1 are 64% and 59%, respectively; these percentages are 23% and 26% for conv5_1. The VGG network is therefore considered to be a reliable feature extractor for climate data.

5.2.   Impact of different layers on perceptual loss

We used various layers of feature maps to calculate the perceptual loss and examined the effect of different levels of information on the resulting HR climate data. Quantitative results are summarized in Table 2 and Table 3. As can be seen, the RMSE of ${Pretrained\;CNN}_{RMSE}$ is the smallest for both $\times$4 and $\times$8 results, which are 0.67 and 1.19, respectively. With the deepening of the applied feature maps, the RMSE of perceptual loss increases and both $Pretrained\;CNN$ and $GANs$ exhibit the same trend. Particularly, when the perceptual loss is calculated using conv5_1, the error increases significantly. As can be seen from the quantitative results, except for conv5_1, the results produced by all layers are not significantly different from those produced by using RMSE as the loss. We compared the results qualitatively in order to further illustrate the performance of perceptual loss.

Figure 3 shows the results of ${Pretrained\;CNN}_{RMSE}$ and $GAN{s}_{RMSE}$, with the first row representing the result of $\times$4 and the second row representing the result of $\times$8. In the training process of ${Pretrained\;CNN}_{RMSE}$, the loss is calculated by per-pixel loss (RMSE), while the loss in the training process of $GAN{s}_{RMSE}$ includes both pixel-wise loss and adversarial loss. In spite of the fact that ${Pretrained\;CNN}_{RMSE}$'s results are slightly better than $GAN{s}_{RMSE}$'s in terms of metrics, we can see from the figure that the result of $Pretrained\;CNN$ is over-smoothed, resulting in a lack of detail information. As an example, in ${Pretrained\;CNN}_{RMSE}$ corresponding to Ground Truth (20051216), there is a lack of detailed information in the lower right corner of the region. Nevertheless, when the adversarial loss is included, we can see that the results of $GANs$ are clearly superior to those of ${Pretrained\;CNN}_{RMSE}$ with respect to detailed information. In contrast, Figure 4 shows $Pretrained\;CNN$ and $GANs$ computed by using perceptual loss. In spite of the absence of adversarial loss, perceptual loss contributes some detailed information to the results of $Pretrained\;CNN$. In the training of $GANs$, the addition of adversarial loss forces the detailed information generated by perceptual loss to be closer to the real results.

The per-pixel loss (RMSE) used as the loss function is consistent with our final evaluation metric in Table 1 and Table 2. Accordingly, a model trained to minimize per-pixel loss would always outperform one trained to minimize perceptual loss or adversarial loss, which encourages the generated HR data to be perceptually similar to the target HR data, but does not require their exact match. This explains why ${Pretrained\;CNN}_{RMSE}$ is best in terms of the metric. It is specifically $Pretrained\;CNN$ to optimize the content-based loss, resulting in safer HR fields, i.e., over-smoothed ground truth predictions. According to the difference between the results of $Pretrained\;CNN$ under per-pixel loss and perceptual loss, we find that, although the parameters of VGG19 are trained on natural images, the medium- and low-level feature maps from VGG19 remain effective for climate data. Even though perceptual loss achieves worse evaluation metrics, it makes more aggressive predictions, inserting significantly more small-scale features that better reflect the nature of real precipitation than the results under per-pixel loss.

Figure 5 compares the results of $GANs$ for different loss conditions. We also show the results of Bicubic whose results are evidently over-smoothed when compared to the ground truth. By adding the adversarial loss, $GAN{s}_{RMSE}$ weakens the smoothing compared to ${Pretrained\;CNN}_{RMSE}$ and adds more detailed information. It should be noted, however, that $GAN{s}_{RMSE}$ provides more smoothed result when compared to the results of $GANs$ using perceptual loss; for example, see the area with more precipitation on the right side of the region in Figure 5. For the $GANs$ using perceptual loss, we can observe that as the feature maps become deeper, i.e., from Conv1_1 to Conv5_1, the results are less smooth, and the HR data are produced with more detailed information. The negative effect of this very aggressive generation is the increase of RMSE, i.e., the deviation from the ground truth. It is important to observe from Table 1 and Table 2 that Conv5_1 produced significantly worst results metrics. In reviewing the results of Conv5_1, we identified two issues that contributed to this outcome. For some samples, the results of Conv5_1 deviate significantly from those of the ground truth. Secondly, some HR results are similar to the ground truth but contain significant checkerboard artifacts. In view of the large error and instability of the results generated by Conv5_1, we do not include Conv5_1 in subsequent analyses.

As previously stated, for climate data, we are more concerned with the performance of those regions with extreme events in the generated HR results. To compare the results of different loss functions in these regions, different quantitative thresholds are used to test the ability of each method to capture extreme events. At each grid cell in our study area, all precipitation values above the percentile threshold were selected first, and then the RMSE was calculated based on the selected data. This was performed in a range of percentages between 75 and 99.9 and averaged over all locations and all samples. As can be seen from Figure 6, in both the $\times$4 and $\times$8 models, the RMSEs corresponding to $GAN{s}_{1}$, $GAN{s}_{2}$, and $GAN{s}_{3}$ are smaller than those in the region of the extreme event. Specifically, for the $\times$4 model, $GAN{s}_{4}$ has the worst RMSE. There is a close relationship between the curves obtained in $GAN{s}_{2}$, $GAN{s}_{3}$ and $GAN{s}_{RMSE}$, and the best result is obtained in $GAN{s}_{1}$. Compared with $GAN{s}_{4}$ and $GAN{s}_{RMSE}$, the results of $GAN{s}_{1}$, $GAN{s}_{2}$, and $GAN{s}_{3}$ are very close in the $\times$8 model. We can conclude that the results of $GAN{s}_{1}$, $GAN{s}_{2}$, and $GAN{s}_{3}$ are comparable to $GAN{s}_{RMSE}$ for the generation of extreme events region. In addition, we examine the spatial distribution of errors in the region of extreme events for $GAN{s}_{1}$, $GAN{s}_{2}$, $GAN{s}_{3}$ and $GAN{s}_{RMSE}$. In the case of $GAN{s}_{1}$ and $GAN{s}_{RMSE}$, the error distribution is wider, but the error values are relatively small. In the case of $GAN{s}_{2}$ and $GAN{s}_{3}$, the errors are concentrated in fewer grid cells, but the error values are relatively large.

Figure 7 shows the precipitation distribution of the six samples, where the x-axis is the variation of precipitation in each raster, and the y-axis refers to the proportion of the corresponding raster. The first row is the result of the $\times$4 model and the second row is the result of the $\times$8 model. As can be seen from the figure, the distribution of $GAN{s}_{4}$ is significantly different from the distribution of the ground truth. The distributions of $GAN{s}_{1}$, $GAN{s}_{2}$, $GAN{s}_{3}$ and $GAN{s}_{RMSE}$ are gradually approaching the distribution of the ground truth as precipitation quantity increases. In contrast, the distributions of $GAN{s}_{RMSE}$ and $GAN{s}_{1}$ for the smaller precipitation are closer to the ground truth's distribution than the distributions of $GAN{s}_{2}$ and $GAN{s}_{3}$. As an example, in sample 6, the density of $GAN{s}_{2}$ and $GAN{s}_{3}$ are greater than the ground truth's distribution in the interval of 0-3 mm. The same observation is also made for sample 1, sample 2, and sample 5.

In order to better qualitatively evaluate the results corresponding to different feature maps, we set up a user experiment. In the experiment, we had a total of 40 participants. There were 20 samples of precipitation data provided to each participant. Each data sample consisted of HR ground truth, and generated results from $GAN{s}_{1}$, $GAN{s}_{2}$, $GAN{s}_{3}$, $GAN{s}_{4}$, and $GAN{s}_{5}$. The order of the different results in each sample was randomized. User experiment results are presented in Figure 6. We collected a total of 790 votes. Among all the votes, the votes corresponding to $GAN{s}_{2}$ occupy 32.4%, slightly higher than the 31.6 corresponding to $GAN{s}_{3}$. The results of $GAN{s}_{2}$ and $GAN{s}_{3}$ are significantly higher than the results of the other models. As shown in Figure 8, the best results for each sample are determined by the principle of the largest number of votes, where 7 best results correspond to $GAN{s}_{3}$ and 8 best results correspond to $GAN{s}_{2}$.

In summary, although low- and medium-level feature maps have slightly worse RMSE metrics than per-pixel loss, they can provide more detailed information for HR results, and with such detailed information, HR results perform well in regions of extreme events. In contrast to per-pixel loss, there is a difference between the distribution of perceptual loss and the ground truth in regions with less precipitation. By qualitatively and quantitatively comparing the results corresponding to different layers of feature maps, we use $GAN{s}_{2}$ and $GAN{s}_{3}$ to test the proposed perceptual loss, since we consider $GAN{s}_{2}$ and $GAN{s}_{3}$ as a balance of qualitative detail information and quantitative evaluation metrics.

5.3.   Using climatological binary perceptual loss

In this section, we examine the performance of the results obtained by training the GAN via climatological binary perceptual loss. Table 4 shows the results for $\times$4 and $\times$8. In the table, although some metrics are still poor when compared to the $GAN{s}_{RMSE}$, they are optimized when compared to the direct calculation with $GAN{s}_{2}$ and $GAN{s}_{3}$. In the case of $\times$4, $GAN{s}_{bpl2}$ and $GAN{s}_{bpl3}$ improve by 0.11 and 0.10, respectively, and in the case of $\times$8, $GAN{s}_{bpl2}$ and $GAN{s}_{bpl3}$ improve by 0.17 and 0.11, respectively. Some distributions of HR results based on climatological binary perceptual loss are shown in Figure 10. The results obtained with the proposed loss are closer to the ground truth in the interval with lower precipitation levels than the direct use of Conv2_1 and Conv3_1, which retain the advantage of calculating the loss by using per-pixel loss. In addition to improving the distribution of the samples, the proposed climatological binary perceptual loss is also optimized for the regions of extreme events. In Figure 9, the results obtained by using Conv2_1 and Conv3_1 are directly compared with those obtained by using the proposed loss. It can be seen that the proposed loss provides detailed information in the area of extreme events that is closer to the ground truth.

5.4.   Application to the GCM data

In this section, we discuss the application of the proposed climatological targeted perceptual loss to the CMIP6 data. The original raster size is 120 × 288 with a resolution of 1° × 1.25°. By increasing the resolution by a factor of 4 and 8, we increase the size of the raster to 480 × 1152 and 960 × 2304. Figure 10 shows the distribution of precipitation before and after the resolution improvement. To further illustrate the results of the model, we calculated the average precipitation of the raw CMIP6 data and the generated high-resolution data in the range of 25° N to 49° N, 70° W to 130° W. This region roughly covers the contiguous United States range. The obtained time series are also presented in Figure 11. We combine the daily data into the corresponding monthly data, and there are 132 time points in the time series from January 2023 to January 2033. As we can see from the figure, the results of $\times$4 are very close to the precipitation process obtained from the original resolution. The results of $\times$8 have more error than the results of $\times$4 in some months, such as July 2024, but it also basically follows the precipitation process as the original resolution. From the samples, we can also see that the proposed climatological targeted perceptual loss increases the detail information of the extreme events regions.

6.   Discussion

Obtaining high-resolution climate data from low-resolution climate data is an ill-posed problem, as for a given LR input, there may be multiple possible high-resolution outputs corresponding to it. On the one hand, we want the generated HR data to be quantitatively similar to the LR input. Additionally, we want the generated HR data to contain detailed information. There is often a conflict between the two during the training process. In general, qualitative similarity forces the model to provide smoother results, which, on average, are close to the input. The disadvantage, however, is that detailed information is lost. Including rich detailed information requires the model to insert possible high-resolution results, and this aggressive approach can lead to an increase in overall error. Our study examines both qualitative and quantitative aspects of perceptual loss in the generation of high-resolution climate data. In terms of multiplicity, we consider both cases of increasing the original resolution by a factor of 4 and 8. In terms of training mode, we examine the results of perceptual loss in the training process of CNN and the training process of GAN. Despite the fact that the parameters of the feature extractor used for perceptual loss are usually trained by using natural images, it still achieves promising results compared with the traditional pixel-wise loss for climatic data. Perceptual loss can provide more realistic HR results than pixel-wise loss during CNN training, though there is a trade-off in the overall RMSE. As the feature map used for perceptual loss becomes deeper, the results become less smooth. The addition of adversarial loss to the GAN training process can solve the problem of pixel-wise loss results that provide less detailed information and are too smooth in nature. A comparison of the results of pixel-wise loss and perceptual loss in GAN indicates that the perceptual loss results are perceptually closer to those obtained by pixel-wise loss. In contrast, when deep feature maps, such as Conv4_1 and Con5_1, are used, the generated results are unstable. The results of Con5_1, in particular, show large errors as well as checkerboard artifacts. The VGG network is trained to solve classification problems, so the high-level semantic information expressed in the deep feature map is primarily used to distinguish between different labels. In the case of climate data, there may be zero or no significant change in most of the region, resulting in an insufficient amount of semantic information contained in the deep feature map. It may not be possible to accurately measure the difference between the generated results and the ground truth by comparing these feature maps.

As a result of comparing the results obtained from feature maps with different layers, we find that the feature maps of the low and middle layers, in our case Conv1_1, Conv2_1, and Conv3_1, provide good results for regions with extreme events. They have slightly smaller RMSEs in these regions than the HR results obtained by using pixel-wise loss, and they provide richer detailed information as well. Based on our qualitative and quantitative analyses, Conv2_1 and Conv3_1are the best balance between obtaining detailed information and preserving the overall error. To solve the problem that perceptual loss has large errors in regions with small raster values and small gradient changes in the sample, we propose climatological targeted perceptual loss. In order to calculate the error for the whole sample, we first calculate the pixel-wise loss, and then for the targeted regions, that is, those containing extreme events, we further use the perceptual loss. In comparison to calculating perceptual loss directly for the entire region, our results show that climatological targeted perceptual loss can improve the detailed information of extreme events regions while reducing error.

7.   Conclusions

In this work, we adapt the perceptual loss technique developed for super-resolution in image processing to obtain high-resolution climatological data. We present the difference between perceptual loss and pixel-wise loss by increasing the resolution of raw rainfall data by $\times$4 and $\times$8 under two different training modes of CNN and GAN. We examine the difference in the perceptual loss calculated by using different layers of feature maps. Based on our results, it appears that low- and mid-level feature maps can yield comparable results to pixel-wise loss. Based on our qualitative and quantitative analyses, Conv2_1 and Conv3_1are the best balance between obtaining detailed information and preserving the overall error. In comparison with calculating perceptual loss directly for the entire region, our results show that the proposed climatological binary perceptual loss can improve the detailed information of extreme events regions. We also examine the performance of the climatological binary perceptual loss by using unseen GCM data.

Conflict of interest

All authors declare that there is no conflict of interests in this paper.