An adaptive preference retention collaborative filtering algorithm based on graph convolutional method

1.   Introduction

Recommender systems have attracted considerable attention from both the industry and academia ^[1,2]. Collaborative filtering (CF) is among the most important recommendation method and depends on a basic assumption that people with similar purchasing experiences in the past will make similar decisions in the future ^[3]. With the development of graph neural networks (GNN) ^[4,5], the application of GNN in recommender systems has become increasingly mature ^[6,7]. In recent years, the history of user-item interactions has been modeled as a user-item graph ^[8] and applied to knowledge graphs ^[9,10], heterogeneous graphs ^[11,12], and social recommendations ^[13]. In addition, the graph convolutional network (GCN) ^[14] has been applied extensively to recommender systems [15−19] and delivered promising performance. Compared with the traditional CF methods, the graph neural network has powerful feature extraction and learning capabilities. The graph convolution can capture the high-order neighborhoods information and the complex interaction relationship on the user-item graph, and generate the feature information of the local graph neighborhood of the user/item. First-order neighborhoods are the user's direct neighbors, second-order neighborhoods are the neighbors of the user's neighbors, and so on. Higher neighborhoods than first-order are called high-order neighborhoods. Meanwhile, GNN combines the target nodes own representation with it's neighborhood information to update the new representation of the target node. The graph convolutional method stacks multiple graph convolutional layers and can aggregate features of users and items. Furthermore, exploiting high-order relationships on the user-item graph can alleviate the problem of data sparsity in collaborative filtering ^[20].

Despite their enormous success, these methods assumed that all interactive items could be used to model user preferences, in which each item contributed the same in generating user preferences. However, the distinctions among the items determine the purchasing behaviors of users. For instance, different items have different attributes, such as functionality and appearance. When users purchase an item, some of them prefer functionality while others prefer its appearance. This prompts us to capture fine-grained preference. A more detailed and specific description of the users' preferences for the item, namely user fine-grained preferences. Similarly, item fine-grained preferences mean more granular features of items, for example, goods will be preferred by users because of their price or quality advantages.

Meanwhile, when the graph convolution operation generates the user's representation, it suffers from an over-smoothing effect on the high-order neighborhood. Therefore, when modeling user preferences, it is important to consider the difference between items to capture the fine-grained preferences of users/items and alleviate the over-smoothing on the user-item graph.

As is shown in Figure 1. Different ratings of items indicate their different contributions to user preferences. If items are similar in terms of importance, it is unclear whether the user assigns an item a high or low rating. However, if we consider differences between items, we may find that assign a high rating because the attributes are more consistent with the characteristics of a career. Hence, failing to discern the differences between items may limit the performance of the recommender system. It is therefore desirable to design a recommender system to adaptively identify differences among items to acquire fine-grained user preferences.

Although it is useful to distinguish the differences among items, three challenges are outstanding in this context. The first challenge stands in the large and complex user-item graph, which makes it difficult to capture the differences between items in the user-item graph. The second challenge involves ways to incorporate the differences between items into modeling user preferences. It is also difficult to identify items that are important for modeling user preferences from among the large number with which he/she interacts. Besides, the user-item graph usually suffers from the problems of data sparsity and over-smoothing due to the cold start issue inherent in recommender systems. Hence, the third challenge pertains to the means of solving these problems in the user-item graph.

In this paper, we proposed an Adaptive Preference Retention Graph Convolutional Collaborative Filtering Method (APR-GCCF), which is an end-to-end model that differentiates distinctions between items to model user preferences. To address the first two challenges, we proposed an adaptive preference retention mechanism. It contains two important elements. First of all, when users interact with different items, the contributions of different items to the user's representation are different. So, we adopted a trainable weight pair for different neighborhoods that generated corresponding scores and then assign them to capture neighborhoods to the user's local interest. Moreover, when we used GNN to aggregate neighborhoods of user nodes, different neighborhoods would carry different item node information. So, we modeled the user and item preferences through two trainable mapping vectors to capture fine-grained user and item preferences. To address the third, we exploited a residual preference prediction mechanism. Among them, to alleviate the problem of data sparsity, we used a multi-layer GNN to aggregate higher-order neighbors. However, similar to previous work ^[21], multi-layer GNNs can cause an over-smoothing problem. The essence of over-smoothing is that too much information about the other nodes suppress the information of the nodes themselves. By using the residual network, we connected its embedding and the aggregated new vector when updating the node embedding. This implemented the decoupling of neighborhoods and their information; thus the problem of over-smoothing could be relieved.

The contributions of this paper are as follows:

● We use a residual preference prediction mechanism that exploits high-order relationships and the residual network, which can alleviate the problems of data sparsity and over-smoothing generated by the user-item graph.

● To distinguish the different contributions made by different items to user embedding, we design an adaptive preference retention mechanism to adaptively identify different types of information provided by different items, thus capturing a fine-grained representation of the user's preference.

● We use L0 regularizers to address the overfitting problem and apply a weighted random sampling strategy to select items that are important for user preference modeling. We validate the effectiveness of the proposed framework (APR-GCCF) on three empirically acquired datasets: Amazon, FilmTrust, and Yelp.

The following parts of this paper were organized as follows: In Section 2, relevant works related to collaborative filtering and graph convolution neural networks were reviewed. In Section 3, we defined the formulations of the recommendation problem in our paper. The framework proposed was presented in Section 4 and three empirically acquired datasets were verified in Section 5. Finally, we summarize our work here and suggested directions for future research in the area in Section 6.

2.   Related works

This section focuses on two areas of research directly related to our study: collaborative filtering and graph neural networks.

2.1.   Collaborative filtering

Collaborative filtering (CF) assumed that users with similar behaviors would have similar preferences for items. Most collaborative filtering algorithms that have been proposed in recent years used matrix factorization (MF) for the recommendation. Matrix factorization (MF) involved mapping users and items into a shared latent space and learning vector information to make recommendations. The PMF ^[22] utilized Gaussian distribution to model latent factors related to users and items, and then used their product to predict item ratings. BiasedMF ^[3] improved the PMF by combining user/item bias and a global bias. SVD++ ^[23] utilized implicit feedback (such as the number of clicks and identification of items by users as their favorite) to model user preferences. Deep learning models have yielded impressive performance in recommender systems in recent years. ^[24] combined collaborative filtering and neural networks to solve the cold start problem. NCF ^[9] employed deep learning to learn complicated internal interactions between users and items. AutoRec ^[25] utilized an encoder layer to project the user's/item's observed ratings into a latent space and reconstructs them using a decoder layer. Another relevant method involved modeling the history of user-item interactions as a user-item graph ^[8] to predict user preferences. The NGCF ^[26] adopted graph neural networks to capture higher-order connectivity and extracted a collaborative signal for a user-item graph. The LR-GCCF ^[27] exploited the residual network to solve the problem of over-smoothing in the user-item graph, and eliminated the nonlinearity in the GNN to improve recommendation performance. The MCCF ^[28] decomposed the user's motivations for purchasing items. Meanwhile, MCCF used node-level and component-level attention mechanisms to generate user/item preferences.

2.2.   Graph neural networks

Graph neural networks were first proposed by [29−31]. The graph convolutional network (GCN) has lately achieved remarkable success in various tasks of graph-based analysis. It aggregated features derived from neighbors of the target node and propagated the information related to it in the graph to update new node embeddings. While different from the GCN, GraphSAGE ^[32] also aggregated features from local neighbors and used uniform sampling to generate a node embedding. The GAT ^[33] employed an attention mechanism ^[34] to adaptively assign weights to a node's neighbors when performing an aggregate operation. The GIN ^[35] operated through an injection-based neighborhood aggregation scheme. It applied a learnable mechanism of combining features to preserve the local graph structure and features of the nodes. The HAN ^[36] has considered the difference in importance among nodes in the convolution process, and the SGC ^[37] reduced the complexity of the GCN by removing feature transformations and nonlinear activations. It was used as a low-pass filter in the spectral domain to reduce the effects of over-smoothing as well. The developers of the DAGNN ^[38] claimed that the effects of entanglement of representational transformation and information propagation primarily affected the performance of the GNN and devised an adaptive fusion weight to adjust the effect of over-smoothing on it.

3.   Preliminaries

In this section, we will introduce the notation used in this paper. We classically modeled the history of user-item interactions as a user-item graph ${\rm{G}} = \{{\rm{U}}, {\rm{V}}, {\rm{R}}, {\rm{E}}\}$ ^[28]. Let ${\rm{U}} = \{{{\rm{u}}}_{1}, {{\rm{u}}}_{2}, \dots, {{\rm{u}}}_{{\rm{n}}}\}$ and ${\rm{V}} = \left\{{{\rm{v}}}_{1}, {{\rm{v}}}_{2}, \dots, {{\rm{v}}}_{{\rm{m}}}\right\}$ denoted the sets of users and items, respectively, where $n$ was the number of users and $m$ was the number of items. We assumed that ${\rm{R}}\in {\mathbb{R}}^{{\rm{n}}\times {\rm{m}}}$ was the user-item graph and may contain several ordinal rating levels $\left\{1, \dots, {\rm{R}}\right\}$. Each edge ${\rm{e}} = \left({{\rm{u}}}_{{\rm{i}}}, {{\rm{v}}}_{{\rm{j}}}, {{\rm{r}}}_{{\rm{i}}{\rm{j}}}\right)\in {\rm{E}}$ showed that user ${{\rm{u}}}_{{\rm{i}}}$ assigned rating ${{\rm{r}}}_{{\rm{i}}{\rm{j}}}$ to item ${{\rm{v}}}_{{\rm{j}}}$. If ${{\rm{u}}}_{{\rm{i}}}$ assigned a rating to ${{\rm{v}}}_{{\rm{j}}}$, ${{\rm{r}}}_{{\rm{i}}{\rm{j}}}$ is the rating, otherwise, ${{\rm{r}}}_{{\rm{i}}{\rm{j}}} = 0$. Generally, we let ${{\rm{R}}}_{{{\rm{u}}}_{{\rm{i}}}}$ be the set of items that have interacted with the user ${{\rm{u}}}_{{\rm{i}}}$, ${{\rm{R}}}_{{{\rm{v}}}_{{\rm{j}}}}$ be the set of users who ${{\rm{v}}}_{{\rm{j}}}$ have interacted with. We used an embedding vector ${{\rm{s}}}^{{{\rm{u}}}_{{\rm{i}}}}\in {{\rm{R}}}^{{\rm{d}}}$ and an embedding vector ${{\rm{h}}}^{{{\rm{v}}}_{{\rm{j}}}}\in {{\rm{R}}}^{{\rm{d}}}$ to denote a user ${{\rm{u}}}_{{\rm{i}}}$ and an item ${{\rm{v}}}_{{\rm{j}}}$, respectively, where ${\rm{d}}$ was the dimension of the embedding vector. The mathematical notations used in this paper are given in Table 1. Then the task of rating prediction can be formulated as:

Input: The user-item interaction matrix R.

Output: Each user's missing rating for the item.

4.   Adaptive preference retention collaborative filtering algorithm based on graph convolutional method

4.1.   Overview

In this part, we proposed Adaptive Preference Retention Graph Convolutional Collaborative Filtering Method (APR-GCCF) which was a general graph neural network based on the CF model for the recommendation. The general framework of the APR-GCCF was shown in Figure 2. APR-GCCF consisted of two major components. The first component, an adaptive preference retention mechanism, is shown in the red dotted line and blue dotted line of Figure 2 used the GNN to assemble the user's/item's neighborhood to learn the user/item embedding and took the user of a trainable mapping vector ${{\rm{m}}}_{1}/{{\rm{m}}}_{2}$ that was shared between layers of the GNN generated retain scores ${{\rm{a}}}^{0}, \dots, {{\rm{a}}}^{{\rm{k}}}$ to differentiate among users/items. The second component was a residual preference prediction mechanism as shown in the black dotted line of Figure 2. It utilized a residual network to concatenate user's/item's preferences representations from an adaptive preference retention mechanism. Finally, the user-item interaction ratings were predicted through MLP.

4.2.   Adaptive preference retention mechanism

In this paper, we aimed to learn the fine-grained preference ${{\rm{A}}}_{{{\rm{u}}}_{{\rm{i}}}}$ of user ${{\rm{u}}}_{{\rm{i}}}$ and representation ${{\rm{A}}}_{{{\rm{v}}}_{{\rm{j}}}}$ of item ${{\rm{v}}}_{{\rm{j}}}$ from user-item graph. It was claimed that user ${{\rm{u}}}_{{\rm{i}}}$ 's preference relied on his/her characteristics ${{\rm{s}}}^{{{\rm{u}}}_{{\rm{i}}}}$ and neighborhood ${{\rm{h}}}^{{\rm{a}}, {\rm{k}}}$, where ${\rm{a}}\in {{\rm{R}}}_{{{\rm{u}}}_{{\rm{i}}}}$. So, we combined ${{\rm{u}}}_{{\rm{i}}}$ 's characteristics with features of ${{\rm{u}}}_{{\rm{i}}}$ 's neighborhood through an aggregation function ${{\rm{A}}{\rm{g}}{\rm{g}}{\rm{r}}{\rm{e}}}_{{\rm{u}}{\rm{s}}{\rm{e}}{\rm{r}}}$. When aggregating the features of the user to generate item ${{\rm{v}}}_{{\rm{j}}}$ 's embedding, a similar aggregation function ${{\rm{A}}{\rm{g}}{\rm{g}}{\rm{r}}{\rm{e}}}_{{\rm{i}}{\rm{t}}{\rm{e}}{\rm{m}}}$ was used. The aggregation process as follows:

where ${{\rm{e}}}_{{{\rm{u}}}_{{\rm{i}}}}^{{\rm{k}}+1}$ denoted the user ${{\rm{u}}}_{{\rm{i}}}$ 's embedding of GNN at layer k that is generated by user aggregation function ${{\rm{A}}{\rm{g}}{\rm{g}}{\rm{r}}{\rm{e}}}_{{\rm{u}}{\rm{s}}{\rm{e}}{\rm{r}}}$. ${\rm{k}}\in \left(1, \dots, {\rm{K}}\right)$ indicated the number of layers in GNN. ${{\rm{R}}}_{{{\rm{u}}}_{{\rm{i}}}}$ was the set of items that user ${{\rm{u}}}_{{\rm{i}}}$ has interacted with. ${{\rm{R}}}_{{{\rm{v}}}_{{\rm{j}}}}$ represented the set of users that have interacted with the item ${{\rm{v}}}_{{\rm{j}}}$, ${{\rm{s}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ denoted the vector of user ${{\rm{u}}}_{{\rm{i}}}$ at k-th layer. ${{\rm{h}}}^{{{\rm{v}}}_{{\rm{j}}}, {\rm{k}}}$ was the vector of item ${{\rm{v}}}_{{\rm{j}}}$ at k-th layer. Then we will discuss how to define aggregation functions ${{\rm{A}}{\rm{g}}{\rm{g}}{\rm{r}}{\rm{e}}}_{{\rm{u}}{\rm{s}}{\rm{e}}{\rm{r}}}$ and ${{\rm{A}}{\rm{g}}{\rm{g}}{\rm{r}}{\rm{e}}}_{{\rm{i}}{\rm{t}}{\rm{e}}{\rm{m}}}$.

We adopted aggregation functions that is different from the traditional graph convolutional networks on the user-item graph to capture the user's/item's representations. To be specific, the attention mechanism was applied to aggregate the neighborhood of user/item, as below:

where ${{\rm{ \mathsf{ α} }}}_{{\rm{i}}, {\rm{a}}}$ denoted the attention coefficient of items in the user's neighborhood when characterizing user ${{\rm{u}}}_{{\rm{i}}}$ 's representation from the user-item interaction history ${{\rm{R}}}_{{{\rm{u}}}_{{\rm{i}}}}$. ${{\rm{ \mathsf{ α} }}}_{{\rm{j}}, {\rm{b}}}$ denoted the attention weight of users in the item ${{\rm{v}}}_{{\rm{j}}}$ 's neighborhood when characterizing item ${{\rm{v}}}_{{\rm{j}}}$ 's representation from the item-user interaction history ${{\rm{R}}}_{{{\rm{v}}}_{{\rm{j}}}}$. ${\rm{R}}{\rm{e}}{\rm{l}}{\rm{u}}(\cdot)$ was the Relu activate function. ${{\rm{W}}}_{{\rm{u}}{\rm{s}}{\rm{e}}{\rm{r}}{\rm{s}}}$ / ${{\rm{W}}}_{{\rm{i}}{\rm{t}}{\rm{e}}{\rm{m}}{\rm{s}}}$ and ${{\rm{b}}}_{{\rm{u}}{\rm{s}}{\rm{e}}{\rm{r}}}$ / ${{\rm{b}}}_{{\rm{i}}{\rm{t}}{\rm{e}}{\rm{m}}}$ were weight and bias of aggregation functions, respectively. ${\rm{♁}}$ is the concatenation operation. Take the aggregation function of user ${{\rm{u}}}_{{\rm{i}}}$ as an example, we took user ${{\rm{u}}}_{{\rm{i}}}$ 's representation ${{\rm{s}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ and features of item interacted with user ${{\rm{u}}}_{{\rm{i}}}$ as input, then use the Softmax function to produce the attention coefficient ${{\rm{ \mathsf{ α} }}}_{{\rm{i}}, {\rm{a}}}$ as:

where ${{\rm{W}}}_{{\rm{i}}, {\rm{j}}}$ and ${{\rm{W}}}_{{\rm{j}}, {\rm{i}}}$ were the attention weight of ${{\rm{ \mathsf{ α} }}}_{{\rm{i}}, {\rm{a}}}$ and ${{\rm{ \mathsf{ α} }}}_{{\rm{j}}, {\rm{b}}}$.

The GNN can capture information on the neighborhood of a given node through the feature propagation mechanism. The nodes were assumed to contribute similarly to the neighborhood aggregation process, which led to the inevitable loss of fine-grained user preference-related information. In practice, the purchasing behavior of the user was usually determined by differences among items. Therefore, different items should make different contributions to generate user embedding. This motivated us to design an adaptive preference retention mechanism to adaptively identify different kinds of information provided by different items.

By taking original user embedding generated by the GNN as input, we took advantage of a trainable mapping vector shared between layers of the GNN to learn the differences among items, and to output their preference retention scores, such as ${{\rm{a}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ and ${{\rm{a}}}^{{{\rm{v}}}_{{\rm{j}}}, {\rm{k}}}$. These preference retention scores were used for representations that carried information on the differences among items in neighborhoods spanning a variety of ranges, and also for measuring the feature-related information of the items to identify their contributions as follows:

where ${{\rm{a}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ was the preference retention scores of the user ${{\rm{u}}}_{{\rm{i}}}$ at k-th layer, and ${{\rm{a}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ was the preference retention scores of ${{\rm{v}}}_{{\rm{j}}}$ same as ${{\rm{u}}}_{{\rm{i}}}$. ${{\rm{m}}}_{1}$ and ${{\rm{m}}}_{2}$ indicated trainable mapping vectors shared by the processes of user and item preference modeling, respectively, and ${{\rm{n}}}_{1}$ and ${{\rm{n}}}_{2}$ were biases. ${{\rm{e}}}_{{{\rm{u}}}_{{\rm{i}}}}^{{\rm{k}}}$ and ${{\rm{e}}}_{{{\rm{v}}}_{{\rm{j}}}}^{{\rm{k}}}$ are user's/item's embeddings generated by GNN at the k-th layer.

Finally, we confirmed differences among the items by using the preference retention scores and preserving the differences in user embeddings automatically to model fine-grained user preferences. The generated user/item preference embeddings ${{\rm{A}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ / ${{\rm{A}}}^{{{\rm{v}}}_{{\rm{j}}}, {\rm{k}}}$ were as follows:

where $\odot$ represented element-wise multiplication, and ${{\rm{A}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ and ${{\rm{A}}}^{{{\rm{v}}}_{{\rm{j}}}, {\rm{k}}}$ were the preference embeddings of the user and the item in layer k.

When users interact with different items, the contributions of different items to the user's representation are different. As shown in Figure 1, the user's current preference is shoes, so shoes are critical to modeling user preferences, and the contribution of musical instruments to the user's representation will be much smaller, therefore, the user's rating of shoes is higher than that of musical instruments. We have added an attention mechanism to the aggregation process of the graph neural networks, namely Eqs (5) and (6), and model the user's representation by assigning corresponding preference scores ${{\rm{ \mathsf{ α} }}}_{{\rm{i}}, {\rm{a}}}$ and ${{\rm{ \mathsf{ α} }}}_{{\rm{j}}, {\rm{b}}}$ to different items. At the same time, we regard the user's neighbors of different levels of user interaction as the user's local interest. We believe that the local interest also plays a vital role in determining the final user's preferences, so we adopt a trainable weight pair for different neighborhoods generate corresponding scores, namely Eqs (7) and (8), and assign them to each neighborhood to generate the final user representation.

When we used GNN to aggregate the high-order neighborhoods of user nodes, different neighborhoods would carry different item node information. We modeled the user and item preferences through two trainable mapping vectors ${{\rm{m}}}_{1}$ and ${{\rm{m}}}_{2}$. We use the adaptively adjusted weight ${{\rm{m}}}_{1}$ / ${{\rm{m}}}_{2}$ shared between GNN layers was used to generate retention scores for different neighborhoods. These retention scores were applied to the user's/item's multi-level neighborhood to save the difference of the item/user nodes in each level of the user's/item's neighborhood. When modeling user preference, we considered the impact of the differences among items for user preference, i.e., the user's purchasing behavior was affected by the attributes of the items. Hence, we utilized ${{\rm{m}}}_{1}$ to adaptively identify the differences in items and retain their different features. Similarly, when modeling item preference, we thought that it was different decision-making procedures that determined a user's decision to buy an item. We thus used ${{\rm{m}}}_{2}$ to distinguish among these procedures to capture fine-grained item preferences.

With the adaptive preference retention mechanism, APR-GCCF enabled the adaptive identification of differences between items and generated a fine-grained representation of users'/items' preferences.

The preference embeddings for the user and the item have been introduced in the previous part. Next, we will show how the residual preference prediction mechanism was used to generate predictions.

4.3.   Residual preference prediction mechanism

With a predefined depth K, the traditional GNN-based recommender systems stop at the k-th layer, where output of the user embedding was regarded as ${{\rm{e}}}_{{\rm{i}}}^{{\rm{k}}}$ and that of the item embedding was ${{\rm{e}}}_{{\rm{j}}}^{{\rm{k}}}$. In practice, ${{\rm{e}}}_{{\rm{i}}}^{{\rm{k}}}$ and ${{\rm{e}}}_{{\rm{j}}}^{{\rm{k}}}$ can catch the fine-grained preferences of user and item. Their inner product was used to predict the ratings ${\widehat{{\rm{r}}}}_{{\rm{i}}{\rm{j}}}$ as follows:

where $\odot$ denoted element-wise multiplication.

Most GNN-based recommendation models won't achieve the best performance until k = 2 or k = 3 (He et al., 2020; Xiang Wang, He, Wang, et al., 2019; Ying et al., 2018). The performance dropped quickly when K continued to increase. We think that the reason was that the embedding of each user/item node was smoothed by higher-order neighbors in the user-item graph. Thus, it was the problem of over-smoothing. In the collaborative filtering task, there was few user-item interactions, and consequently rendered the effect of over-smoothing severe. To solve these problems, a residual preference prediction mechanism was proposed. We performed a concatenation operation on user and item preference embeddings ${{\rm{A}}}^{{{\rm{u}}}_{{\rm{i}}}, {\rm{k}}}$ and ${{\rm{A}}}^{{{\rm{v}}}_{{\rm{j}}}, {\rm{k}}}$ generated by the adaptive preference retention mechanism in layer k as follows:

where ${\rm{♁}}$ was the concatenation operation, k was the layer number of GNN. The residual preference prediction mechanism could help alleviate effect of over-smoothing by using deeper GNN layers to enhance the expressiveness of user preference. Hence, we used residual preference prediction mechanism to connects all layers of the results of prediction:

where ${{\rm{g}}}_{{\rm{i}}{\rm{j}}}$ denoted the concatenation of preference-related predictions in all layers, ${{\rm{g}}}_{{\rm{i}}{\rm{j}}}^{0}$ represented the initially predicted preference used as input for the user and the item, and ${\rm{♁}}$ denoted the concatenation operation of the embeddings. The previous process was reasonable in that each user's neighbors were different, and thus the practice that integrating each layer's representation to generate the final user preference was advantageous.

The residual preference prediction mechanism connected the user representations generated by the GNN of each layer with different neighborhood information to produce the final unified embedding of the user. At the same time, we obtained the final unified embedding of the item through the same operation, and combined the embedding of the user and the item to generate a prediction.

Equation (9) was equivalent to embedding the nodes with differences in each layer to generate a new embedding. This was quite reasonable because the user-item interaction subgraph was different. As a result, the nodes in its neighborhood were different, and the contribution to the final representation of the node was different as well. Therefore, we integrated the representations of each layer to generate the final user/item embedding which contained richer information.

Once the final preference has been obtained, we used it as input to the MLP to predict the rating ${{\rm{r}}}_{{\rm{i}}{\rm{j}}}$ from ${{\rm{u}}}_{{\rm{i}}}$ to ${{\rm{v}}}_{{\rm{j}}}$ as:

where ${{\rm{W}}}_{{\rm{l}}}$ was a weight vector, ${{\rm{b}}}_{{\rm{l}}}$ was a bias vector, and $l$ was the hidden layer index. As we can see, the layer-wise aggregation was the main operation. For the k-th GNN layer, the aggregation function has computational complexity ${\rm{O}}({\rm{n}}{\rm{m}}{\rm{ }}\times {\rm{ }}{\rm{d}})$, where $n$ was the number of users and $m$ was the number of items, and $d$ was the dimension of embedding vector.

We used a graph convolution operation in the user-item graph to aggregate the user's representation. By obtaining the user's high-level neighborhood information, we could mine hidden user-item interaction relationships, which can increase the density of the data to a certain extent. However, aggregating the high-level neighborhood information through multi-layer GNN will cause the problem of over-smoothing. The essence of over-smoothing is that too much information about the other nodes suppress the information of the nodes themselves. A residual network was used to inject the user's previous information into the high-order neighborhoods. Specifically, we connected its embedding and the aggregated new vector together when updating the node embedding. This implemented the decoupling of neighborhoods and their information; thus the problem of over-smoothing could be relieved.

In the following section, some optimization strategies will be discussed.

4.4.   Optimization

Objective function: We focused on the task of predicting ratings. The ultimate goal was thus to minimize the difference between the predicted rating and the ground-truth rating:

where ${\rm{O}}$ was the set of users and items, and ${\widehat{{\rm{r}}}}_{{\rm{i}}{\rm{j}}}$ and ${{\rm{r}}}_{{\rm{i}}{\rm{j}}}$ were the predicted ratings and the ground-truth ratings.

L0 regularization was used to alleviate the problem of overfitting. The final objective function was:

where λ is the L0 regularization parameter and ${\rm{ \mathsf{ θ} }}$ represented the set of model parameters.

Weighted random sampling strategy: To model the representation of user/item, the user's/item's neighborhoods were aggregated by applying GNN. However, a large number of interactions occurred between items and users. So, we usually understood user preferences using only some items with what he/she interacted. In Eq (1), we did not aggregate all of the items that interacted with the user because highly-rated items better-reflected user preferences. Hence, we adopted a weighted random sampling strategy to emphasize more on such items.

We regarded the average degree of the node ${{\rm{N}}}_{{\rm{i}}}^{}$ (${{\rm{N}}}_{{\rm{j}}}^{}$) of the user (item) as the threshold. When the number of nodes neighboring a given node exceeded the threshold, we chose neighborhoods using a random weighted sampling strategy; otherwise, all neighborhood nodes would be retained. The procedure was as follows:

where w was a random number in (0, 1), ${\rm{r}}$ was the rating, and s indicated a generated weighted sampling score. We sampled all nodes in the neighborhood, arranged them in descending order, and took the highest ${{\rm{N}}}_{{\rm{i}}}^{}$ (${{\rm{N}}}_{{\rm{j}}}^{}$) as the neighborhood of the user/item.

5.   The proposed method

We conducted experiments on three empirically acquired datasets to verify the effectiveness of our proposed model and answered the following questions:

● Q1: How does the performance of our model compare with state-of-the-art collaborative filtering models?

● Q2: How effective is the preference learned from the adaptive preference retention mechanism? Can it be used for residual preference prediction?

● Q3: How do different hyperparameters affect the model?

5.1.   Datasets

We used three empirically acquired datasets to validate our model: Amazon, FilmTrust, and Yelp. Their main properties were listed in Table 2. We took 80% of each dataset as the training set and the other 20% as the test set.

● Amazon: The Amazon is a widely used product recommendation dataset containing 65,170 ratings from 1000 users on 1000 items.

● FilmTrust: FilmTrust is a movie sharing and rating website containing 35,497 ratings from 1508 users on 2071 items.

● Yelp: This is a dataset to recommend local businesses containing 30,838 ratings from 1286 users on 2614 items.

5.2.   Baseline

To verify the effectiveness of our model, we used the following methods for comparison:

● PMF ^[22]: This is the most popular matrix factorization model for a CF-based recommender systems.

● BiasedMF ^[3]: This is a factorization model that considers biases to model user and item preferences.

● SVD++ ^[23]: It utilizes implicit feedback to capture fine-grained user preferences.

● AutoRce ^[25]: It utilizes an encoder layer to project the user's/item's observed ratings into a latent space, and then reconstructs them using a decoder layer.

● NGCF ^[26]: It is a deep learning method that uses the GNN to extract higher-order connectivity for recommendation.

● LR-GCCF ^[27]: The model uses a residual network to alleviate over-smoothing in the user-item grap.

● MCCF ^[28]: It decomposes the user's motivation for purchasing, and uses node-level and component-level attention mechanisms to generate user/item preferences.

5.3.   Implementation

We adopted two widely used evaluation metrics, respectively, the RMSE (root mean squared error) and MAE (mean absolute error) to assess the predicted ratings of items. We set the number of layers of the graph neural network to $\{1, 2, 3, 4, 5\}$ and the number of dimensions of the embedding ${\rm{d}}$ to $\{16, 32, 64,128,256\}$. We randomly initialized the model parameters using a Gaussian distribution N (0, 0.1) and took Adam as the optimizer. The batch used for training was $\{64,128,256,512\}$ and the learning rate was $\{0.00005, 0.0001, 0.0005, 0.001, 0.01\}$. The dropout rate was $\left\{0.1, 0.2, 0.3, 0.4, 0.5\right\}$. We used L0 regularization and the regularization parameters from ^[39].

5.4.   Baseline comparison (Q1)

We compared the performance of all methods. Table 3 shows the errors incurred by all of them in the experiments.

We have the following observations: 1) Our model, the APR-GCCF, generally outperformed all the other baselines in terms of the RMSE and MAE, which showed its superiority in the recommendation. 2) The APR-GCCF further improved recommendation performance compared with the NGCF and MCCF. This suggested that considering the differences among items when modeling user preferences helped improve recommendation performance. 3) Our method consistently outperformed the LR-GCCF. Although the LR-GCCF used a residual network to predict user interest, it neglected the varying contributions of items to this outcome. 4) The APR-GCCF achieved better performance than the PMF, BiasedMF, and SVD++. Because these three methods were based on the MF, whereas the APR-GCCF was based on the graph neural networks, and took the user-item graph as input to model the user preference. 5) Of the baselines, the NGCF, LR-GCCF, MCCF, and APR-GCCF belonged to graph neural network-based methods, which indicated that graph neural networks were powerful in terms of learning representations for the user-item graph.

5.5.   Ablation study (Q2)

We designed three variants of the proposed model: APR-all, APR-residual, and APR-retain. APR-residual and APR-retain involved the proposed method without the residual preference mechanism and the adaptive preference retention mechanism, respectively, and APR-all consisted of the proposed method without both. We conducted experiments using them on the three empirically acquired datasets and compared the results with those of our proposed model.

The experimental results were shown in Figure 3.

Figure 3 illustrated that APR-all, APR-residual, and APR-retain delivered worse performance than the proposed method in varying degrees on the three datasets. There were three reasons. First, when we removed the adaptive and the residual preference prediction mechanisms, our model degenerated into a normal GNN that yielded poorer performance in terms of collaborative filtering. Second, the GNN treated the contributions of all items as the same because it did not have the adaptive preference retention mechanism, and thus could not distinguish among the items. Finally, when the residual preference prediction mechanism was removed, the GNN could not handle the problem of over-smoothing, so this degraded its performance.

Effectiveness of adaptive preference retention mechanism: APR-residual delivered worse results than APR-GCCF, which indicated that considering the differences among items is important for learning user preferences and improving recommendation-related performance.

To distinguish the adaptive preference retention mechanism and the attention mechanism, we designed an attention-based variant of the proposed method called APR+ATT, i.e., we used an attention mechanism instead of the adaptive preference retention mechanism. We experimented it on three datasets and compared its results with those of the proposed APR-GCCF, as shown in Figure 4.

The APR+ATT was inferior to our model on both evaluation metrics on the three datasets. This is because the attention mechanism usually pays more attention to items with larger weights, and cannot effectively retain the difference information between items. Besides, using the attention mechanism to model user representations, items with higher weights were usually assigned more contributions to the user representation, while items with lower weights may be ignored. So, this cannot reflect the difference information between the items. It did not consider the differences among items, which was an important reason for the inefficiency of APR+ATT.

Effectiveness of residual preference prediction mechanism: Figure 3 showed that the APR-all delivered worse than APR-retain. This verifies the effectiveness of the residual preference prediction mechanism. We asserted that the traditional GNN suffered from data sparsity and over-smoothing during propagation in higher-order neighborhoods. We integrated the aggregated results of each layer of the GNN into the final prediction by using a residual network, and this helped avoid data sparsity and over-smoothing, thus improving performance in terms of recommendations.

5.6.   Effect of hyperparameters (Q3)

Since the number of GNN layers is critical to the model, we investigated its impact on performance. We also analyzed the effects of the learning rate and the number of embedding dimensions on the APR-GCCF.

The effect of the number of GNN layers: In the graph convolution clustering process, the number of layers of the graph convolution is 1, which means that the first-order neighborhoods are aggregated. The number of layers is k, which means that the kth-order neighborhoods are aggregated. To explore the effect of the number of layers on the model, we varied the number of GNN layers k in the range $\{1, {\rm{ }}2, {\rm{ }}3, {\rm{ }}4, {\rm{ }}5\}$ while keeping the other parameters constant. Figure 5 implied the performance obtained using different numbers of graph convolutional layers on the three datasets. When k = 3, the model achieved the best performance on all three datasets. As the number of graph convolutional layers increased, performance gradually worsened due to over-smoothing and overfitting.

The effect of the learning rate: We delved into the effects of different learning rates on the model on the three datasets, as shown in Figure 6. We set the initial learning rate lr to 0.01 and gradually reduced it. The performance of the model was optimal when lr = 0.001, and degraded as the learning rate decreased. We thus used an appropriate learning rate to reduce the complexity of the model.

The impact of the number of embedded dimensions: The number of embedded dimensions ${\rm{d}}$ was a key parameter controlling the APR-GCCF. Figure 7 showed that the proposed method attained the best performance on Amazon and FilmTrust at ${\rm{d}}$ = 64, and on Yelp at ${\rm{d}}$ = 32. This difference might have been acquired owing to the varying sparsity of the datasets. In general, as ${\rm{d}}$ increased, recommendation performance gradually improves, with more embedded dimensions yielding a stronger representation. But the model's performance worsened when the number of dimensions was larger than the optimal value.

6.   Conclusions and future work

We proposed a graph network model called the APR-GCCF to model collaborative filtering to predict the ratings assigned by users to items to make recommendations to them. The APR-GCCF was composed of two parts. An adaptive preference retention mechanism was used to adaptively distinguish the items. We retained differences among these items to model fine-grained user/item preferences. In addition, to reduce data sparsity and the effect of over-smoothing introduced by higher-order layers of the GNN, we designed a residual preference prediction mechanism that used a residual network to concatenate the user/item preference generated by each layer of the GNN. We conducted extensive experiments on three real-world datasets to validate the effectiveness of our approach.

Although the APR-GCCF has achieved success, information is collected from adjacent nodes without distinguishing which path the information comes from. However, this problem is necessary for improved model capability and interpretability. In future work, we will investigate this problem.

Acknowledgments

This study was supported by the Jilin Provincial Development and Reform Commission (contract number 2022C046-5) and the Scientific Research Project of The Education Department of Jilin Province in 2023, NO. JJKH20231514K.

Conflict of interest

The authors declare there is no conflict of interest.