Figure 1.
An example of knowledge graph with associated multimodal content.
Citation: Esther Menendez, Paula Garcia-Fraile, Raul Rivas. Biotechnological applications of bacterial cellulases[J]. AIMS Bioengineering, 2015, 2(3): 163-182. doi: 10.3934/bioeng.2015.3.163
| [1] | Hanming Zhai, Xiaojun Lv, Zhiwen Hou, Xin Tong, Fanliang Bu . MLSFF: Multi-level structural features fusion for multi-modal knowledge graph completion. Mathematical Biosciences and Engineering, 2023, 20(8): 14096-14116. doi: 10.3934/mbe.2023630 |
| [2] | Xiaowen Jia, Jingxia Chen, Kexin Liu, Qian Wang, Jialing He . Multimodal depression detection based on an attention graph convolution and transformer. Mathematical Biosciences and Engineering, 2025, 22(3): 652-676. doi: 10.3934/mbe.2025024 |
| [3] | Xiaodong Zhu, Liehui Jiang, Zeng Chen . Cross-platform binary code similarity detection based on NMT and graph embedding. Mathematical Biosciences and Engineering, 2021, 18(4): 4528-4551. doi: 10.3934/mbe.2021230 |
| [4] | Guoli Wang, Pingping Wang, Jinyu Cong, Benzheng Wei . MRChexNet: Multi-modal bridge and relational learning for thoracic disease recognition in chest X-rays. Mathematical Biosciences and Engineering, 2023, 20(12): 21292-21314. doi: 10.3934/mbe.2023942 |
| [5] | Yang Liu, Tianran Tao, Xuemei Liu, Jiayun Tian, Zehong Ren, Yize Wang, Xingzhi Wang, Ying Gao . Knowledge graph completion method for hydraulic engineering coupled with spatial transformation and an attention mechanism. Mathematical Biosciences and Engineering, 2024, 21(1): 1394-1412. doi: 10.3934/mbe.2024060 |
| [6] | Peng Wang, Shiyi Zou, Jiajun Liu, Wenjun Ke . Matching biomedical ontologies with GCN-based feature propagation. Mathematical Biosciences and Engineering, 2022, 19(8): 8479-8504. doi: 10.3934/mbe.2022394 |
| [7] | Paula Mercurio, Di Liu . Identifying transition states of chemical kinetic systems using network embedding techniques. Mathematical Biosciences and Engineering, 2021, 18(1): 868-887. doi: 10.3934/mbe.2021046 |
| [8] | Zhijing Xu, Yang Gao . Research on cross-modal emotion recognition based on multi-layer semantic fusion. Mathematical Biosciences and Engineering, 2024, 21(2): 2488-2514. doi: 10.3934/mbe.2024110 |
| [9] | Keyue Yan, Tengyue Li, João Alexandre Lobo Marques, Juntao Gao, Simon James Fong . A review on multimodal machine learning in medical diagnostics. Mathematical Biosciences and Engineering, 2023, 20(5): 8708-8726. doi: 10.3934/mbe.2023382 |
| [10] | Xinyu Lu, Lifang Wang, Zejun Jiang, Shizhong Liu, Jiashi Lin . MRE: A translational knowledge graph completion model based on multiple relation embedding. Mathematical Biosciences and Engineering, 2023, 20(3): 5881-5900. doi: 10.3934/mbe.2023253 |
Knowledge Graphs are a type of relational graphs that store the factual knowledge in real-world, in which the factual knowledge is in the form of triplets. Existing large-scale knowledge graphs projects include FreeBase [1], YAGO [2] and DBpedia [3], which are effective to support downstream applications such as medical question answering [4], named entity disambiguation [5] and dialogue systems [6] and so on. Therefore, it is an important problem to develop an effective method to represent and store Knowledge Graphs for different applications. In order to provide a numerical representation for knowledge graph, knowledge graph embedding (KGE) aims to translate the entities and relations to a continuous low dimensional vector space [7,8]. Then, the embedded representation can be used as the input for other applications.
Recently, KGE has attracted a great attention in natural language processing, and many KGE models have been proposed. Most of the existing KGE models mainly learn the representation for the relation, head entity and tail entity, based on the structural information of triples [9,10,11]. These models neglect the abundant content associated with the entities and relation, which affects the performance of the learned representation. Usually, many of the nodes in KG may be associated with different modalities of external data, such as text description and images, which provides details of the corresponding entities. These data are also valuable to specify the semantics of the nodes and predict the relation between entities, and hence improve the learning of KGE. For example, Figure 1 shows an example of knowledge subgraph, in which the nodes are associated with multi-modal contents. The image associated with the entity "Bill Gates" is helpful to predict that the "gender" of "Bill Gates" is "male". The text description associated with "Bill Gates" is helpful to predict that the "country" of "Bill Gates" is the "United States of America". Similar observations are the same for the entities "Microsoft" and "Melinda Gates". Therefore, effectively encoding the multimodal data into the learning of knowledge graph embedding provides new clue to improve KGE.
There are some works that attempt to improve the performance of KGE by exploiting the multimodal content. For example, Mousselly-Sergieh et al. [12] propose to align the features of structure, text and image to learn the representation of KG with a translation-based method. Veira et al. [13] replace or add the entity features with text features. Yao et al. [14] use the text associated with the KG triples to finetune the pre-trained model BERT [15] for knowledge graph completion. Although these methods have achieved a certain degree of success, they are not effective to learn the cross-modal correlation in the multimodal data. Compared with the unimodal data, different modalities of data content contained in KGs are heterogeneous and represented in different spaces. It is not appropriate to integrate the features in different spaces directly by element wise addition, multiplication or concatenation. Moreover, the correlation in multimodal data is more complex. There exist intra-modal and inter-modal correlation in them. For example, the different objects in an image are correlated with each other, and also the words in text description. As for the inter-modal correlation, some objects in an image are semantically similar with certain words in the corresponding text description. Therefore, it is nontrivial to encode the intra-modal and inter-modal correlation simultaneously to learn the features for the multimodal content. Meanwhile, the multimodal content and graph structure are also heterogenous, it is difficulty to combine the content and structural information for embedding learning. Finally, a large number of studies have shown that the interaction between entities and relations should not be ignored in KGE models [16,17]. It is desired to effectively encode the interaction between entities and relations in depth into the relation embedding learning.
To tackle these problems, we propose a multi-modal content fusion-based knowledge graph embedding model (MMCF), which encodes the multimodal content based on cross-modal correlation with the structural information to learn embedding representation. In particular, we investigate: 1) how to encode the intra-modal and inter-modal correlation of multimodal content into the embedding of knowledge graph; 2) how to fuse the structure information and data content for embedding learning. As shown in Figure 2, the model mainly contains three modules. The first module is entity embedding, which is proposed to learn the intra-modal and inter-modal correlation, in which the multimodal content is fused based on the cross-modal correlation to obtain a uniform representation. Then, the content representation is fused with the structural features to obtain entity embedding. The second module is relation embedding, which is used to fuse the interaction between entities and relation to obtain relation embedding. The third module is a decoder, which learn a scoring function to fuse the learning of entity embedding and relation embedding. Our model is different from existing fusion models that mainly learn the global features of attribute data and network structure for KGE, which is not effective to capture the latent semantics correlation between different modality of data. In addition, the proposed model can be directly extended to integrate more types of data with existing decoders. To evaluate the model, we supplement the entities with related text and images for two public datasets. Experimental results demonstrate the superiority of our approach. The main contributions are as follows:
• We propose to exploit the fine-grained semantics and correlation in the different modalities of data to improve the embedding of knowledge graph.
• We propose a novel Multi-modal content Fusion model (MMCF) for knowledge graph embedding, in which the text content and visual content are fused with the structure information to learning the embedding of entities and relation.
• Extensive experiments are conducted on three benchmark datasets, and the result demonstrates the superiority of our approach.
In the rest part of this paper, the related existing works is summarized in Section 2. Then, the problem of KGE with multimodal content is formulated in Section 3, followed by Section 4 which presents the detail of our model. The experiments and analysis are provided in Section 5. Finally, the paper is discussed in Section 6 and concluded in Section 7.
There are many knowledge graphs embedding models, which can be roughly divided into two categories: structure-based models, and models fused with external content.
The traditional structure-based models mainly learn the representation for the entities and relations from the triples, which defines a scoring or distance function on each fact to measure its plausibility. Some works propose translational distance models, including MuRE [10], TransE [18], TransR [19], etc. TransE is one of the first proposed models, which considers relation as a translation from the head entity to tail entity in the same vector space. By applying a relation-specific matrix, MuRE proposes a relation-specific distance measuring method. Some other works propose semantic matching models, which defines a similarity-based scoring function to calculate the probability that a triple is a golden triple. They mainly learn latent semantic representation for the entities and relations to measure the plausibility of each fact, such as TuckER [20], HolE [21], CrossE [22], etc. Recently, there are some works based on neural network, which proposes to apply the classic neural network models, i.e., CNN [23] and GNN [24], to learn the deep interaction information between the entities and relations. These approaches mainly include ReInceptionE [25], ConvKB [26], HypER [27], COMPGCN [28], etc. There are also some models that propose to use logical rules [29], and relation paths [30], to further learn the structure information.
These structure-based models exploit different properties of representation space to expect that the learned embedding is effective to preserve the structural information of the original knowledge graph, i.e., the representation is effective and efficient to infer the relationships between entities. Though these approaches achieve great success, they only learn the representation from the structural information of the triples, which can't be directly extended to encode the abundant and valuable content associated with the nodes of KG.
With the development of Web technology and social media, various data are produced for the entities and facts of knowledge graph. Therefore, there are some works that attempt to learn from the external data to improve the embedding of KG. One of the earliest works try to fuse the text information for knowledge base completion [31], in which only the text description is used to initialize the representation of entities. The text representations learned from text and knowledge graph structure are directly combined by a gate strategy in [9]. DKRL [25] propose to use both CBOW and CNN to learn entity representation by combining the text description, and then the objective function proposed by TransE is adopted for joint learning. Similarly, Veira et al. [13] propose to add text features to the entity features directly, which can be built on other KGE models. KG-BERT [14] uses the pre-trained language model BERT [15] to learn the representation of the text description associated with the entities and relations, and the triples are considered as textual sequences. However, these models are not effective to learn the latent correlation between different types of features since they are represented in different spaces.
There are also some works use Nonnegative Matrix Factorization (NMF) to combine different views of data or attribute with the affinity information. For example, the tensor singular value decomposition is used to learn the relation between different views in [32]. The structure information of co-expression and attribute data are fused by NMF in [33]. Li et al. [33] uses Nonnegative Matrix Factorization to fuse the structure information and attribute data, where the attribute contains one modality of content. Ma et al. [34] also uses the joint NMF and self-representation leaning to combine the structure and multi-view data, and [35] uses the joint NMF to combine different networks. These methods need the global network structure, which is not effective to handle new data because it needs a matrix built on the whole network.
Beside the text content or attribute, the visual content is also fused to learn knowledge graph embedding. Based on DKRL, IKRL is proposed to further combine image with the structure information for KGE [36] and the objective function of DKRL is also used for joint learning. Based on IKRL, Mousselly-Sergieh et al. [12] propose a multimodal translation-based approach to leverage both multimodal, i.e., visual and linguistic, and structural information for KG representation learning. These methods mainly align the structural information and other types of features instead of learn the latent correlation between them. More types of data, such as numbers, texts and images are included to learn the embedding of KG in [37], which directly concatenates different types of features into a high-dimensional vector. It is not effective to be applied to embed large-scale KGs.
Though introducing the external information has improved the performance of KGE, there are still some problems remained unsolved. First, these methods mainly regard different type of data as a whole, which neglects the fine-grained semantics and the cross-modal correlation. Second, most of the models mainly combine the text or image features with the entities, which is not effective to model the interaction between the content of entities and relations. However, it has been demonstrated that the interaction contributes greatly to the performance of KGE [38]. Finally, many of the fusion-based models are specifically designed, which can't be extended to other KGE models to include the external content.
Before the introduction of our model, we formulate the problem of KGE. A knowledge graph G is represented by a collection of golden triples denoted as (h, r, t), where h, t ∈ E denote the head and tail entity respectively, and r ∈ R denotes the relation between the head and tail entities. Beside the original structure information, each entity is also associated with other multi-modal content, i.e., the textual description e and image I.
Then, the problem of knowledge graph embedding can be formulated as: $ ({{\boldsymbol{z }}_h}, {{\boldsymbol{z }}_r}, {{\boldsymbol{z }}_t}) \leftarrow f(h, r, t) $, where f(.) is the embedding function which combines the triple structure information and the multimodal content associated with the entities to learn the embedding, $ {{\boldsymbol{z }}_h} $, $ {{\boldsymbol{z }}_r} $, $ {{\boldsymbol{z }}_t} $∈ Rd are the learned representations of the head entity h relation r and tail entity t respectively.
The framework of our embedding model MMCF is showed in Figure 2. As it is shown, MMCF mainly contains three modules, i.e., entity embedding, relation embedding and decoder. The entity embedding component is proposed to learning embedding of entity, in which the intra-modal and inter-modal correlation are encoded to fuse the multimodal content and structure information. The relation embedding module is used to learn the embedding of relation, which includes the features of the head entity and tail entity to supplement the relation feature. The decoder can be any of the existing decoders, such as MuRE [10] and InteractE [11], which learns a score for the input embeddings of a triple by a loss function.
As shown in Figure 2, our model MMCF is mainly comprised of three modules, i.e., the entity embedding module, relation embedding module and decoder. We detail the three modules in this section.
Most of the existing works of knowledge embedding learn the representation mainly based on the triple information [10,18]. Though some works try to include the other types of content for entity embedding, they are not effective to capture the latent correlation between different types of content [25,29]. In this module, we first learn the multimodal content features by exploiting the intra-modal and inter-modal correlation, and then they are fused with the structure features by a gating network to obtain the final representation of entity.
Given an image I, we use Faster R-CNN [39] initialized with ResNet-101 to extract the visual object proposals, which is represented by triple set (oi, li, ai), where oi is the feature vector extracted from the region of interest (ROI) pooling layer in the Region Proposal Network, li is a 4-dimensional representation of the bounding box location, and ai is a one hot representation of the attribute class. These vectors are then combined to formulate the representation of the visual object as follows:
| $ {o'_i} = concat({o_i},{{\boldsymbol{ W}}^l}{l_i},{{\boldsymbol{ W}}^a}{a_i}) $ | (1) |
where concat(.) is the concatenation operation, Wl and Wa are the parameter matrices. Then, a fully connected layer is added to transform the vector $ {o'_i} $ to match the textual features, i.e., $ {\boldsymbol{ v}} = \{ {v_1}, {v_2}, ..., {v_k}) $, where vi denotes the transformation of $ {o'_i} $.
The pre-trained language representation model BERT [15] is used to obtain word vectors for the text description. Given a text description document e, we extract the word vectors as e = {w1, w2, …, wm}, where m denotes the number of words in the document. Then, the image object features and textual word embeddings are further processed to learn the intra-modal and inter-modal correlation, as shown in Figure 2.
Usually, the objects in an image and words in a document are correlated with each other. By exploiting the intra-modal correlation, the important information in each modality can be enhanced for representation learning. As for the visual content, we use the self-attention mechanism [40] to learn the intra-modal correlation. Specifically, given a set of objects $ {\boldsymbol{ v}} = \{ {v_1}, {v_2}, ..., {v_k}{{\text{\}} }} $, the query, key and value are calculated: $ {{\boldsymbol{Q }}_v} = {\boldsymbol{v }}{{\boldsymbol{W }}^Q} $, $ {{\boldsymbol{K }}_v} = {\boldsymbol{v }}{{\boldsymbol{ W}}^K} $, $ {{\boldsymbol{V }}_v} = {\boldsymbol{ v}}{{\boldsymbol{ W}}^V} $, where $ {{\boldsymbol{ W}}^Q} $, $ {{\boldsymbol{ W}}^K} $, and $ {{\boldsymbol{W }}^V} $ are the matrices of parameters. Then, the weighted sum of the value is calculated as follows:
| $ Self - Attent{{\text{(}}}{{\boldsymbol{ Q}}_v},{{\boldsymbol{K }}_v},{{\boldsymbol{ V}}_v}) = softmax\left( {\frac{{{{\boldsymbol{ Q}}_v}{\boldsymbol{ K}}_v^T}}{{\sqrt {{d_k}} }}} \right){{\boldsymbol{V }}_v} $ | (2) |
where dk denotes the dimensionality of the visual object vector. We apply the multi-headed attention mechanism to calculate the self -attention h times, and then the values of all heads are concatenated. Then, the Add Norm layers are appended to smooth the result as follows:
| $ {\boldsymbol{O }}_v^{(l)} = Norm({\boldsymbol{ H}}_v^{(l - 1)} + {\boldsymbol{ M}}_v^{(l)}) $ | (3) |
| $ {\boldsymbol{H}}_v^{(l)} = max(0,O_v^{(l)}{\boldsymbol{W}}_v^{(l)} + {\boldsymbol{b}}_v^{(l)}){\boldsymbol{W}}_v^{{{(l)}^\prime }} + {\boldsymbol{b}}_v^{{{(l)}^\prime }} $ | (4) |
where $ {\boldsymbol{ H}}_v^{(l - 1)} $ is the input before self-attention process, and $ {\boldsymbol{ M}}_v^{(l)} $ denotes the output of self-attention process. The self-attention and Add Norm process literately to obtain the visual object vectors, and then the vectors are aggregated to obtain the image representation $ {{\boldsymbol{v }}^{_0}} \in {R^d} $ by an average pooling operation.
As for the textual content, the convolution neural networks [41] is used to learn the intra-modal correlation. Specifically, given the textual vectors input e = {w1, w2, …, wm}, the 1-dim CNN [41] is used to encode the context information. We use three window sizes, i.e., uni-gram, bi-gram and tri-gram, to learn the representation of the i-th word as follows:
| $ {w_{s,i}} = ReLU({{\boldsymbol{W }}_s}{w_{i:i + s - 1}} + {b_s}),s = 1,2,3 $ | (5) |
where $ {w_{s, i}} $ is the output of the i-th word using window size s, $ {{\boldsymbol{ W}}_s} $ is the parameter of filter matrix and $ {b_s} $ is the bias parameter. Then, all the word vectors corresponding to the window size s is aggregated using a max-pooling operation to obtain the text representation: $ {p_s} = max({w_{s, 1}}, {w_{s, 2}}, ..., + {w_{s, m}}) $. Finally, p1, p2 and p3 are concatenated to a fully connected layer with a l2 normalization to obtain the final text description embedding e0:
| $ {e^0} = Norm({{\boldsymbol{ W}}_e}concat({p_1},{p_2},{p_3}) + {b_e} $ | (6) |
where $ {{\boldsymbol{ e}}^0} \in {R^d} $ is the learned text representation which encodes the intra-modal correlation.
Beside the intra-modal correlation, each object in an image may also correlated with some words in the text description. The inter-correlation is important to supplement the learning of the representation for each other modalities. We use the cross-attention to capture the cross-modal correlation for representation learning. As shown in Figure 2, the input of the cross-attention are the stacked features of image objects $ {\boldsymbol{v }} = \{ {v_1}, {v_2}, ..., {v_k}{{\text{\}} }} $ and textual words e = {w1, w2, …, wm}. First, we obtain the query, key and value for the two modalities: $ {{\boldsymbol{K }}_v} = {\boldsymbol{v }}{{\boldsymbol{W }}^K} $, $ {{\boldsymbol{Q }}_v} = {\boldsymbol{v }}{{\boldsymbol{W }}^Q} $, $ {{\boldsymbol{V }}_v} = {\boldsymbol{v }}{{\boldsymbol{W }}^V} $, $ {{\boldsymbol{K }}_e} = {\boldsymbol{e }}{{\boldsymbol{W }}^K} $, $ {{\boldsymbol{Q }}_e} = {\boldsymbol{e }}{{\boldsymbol{W }}^Q} $, $ {{\boldsymbol{V }}_e} = {\boldsymbol{e }}{{\boldsymbol{W }}^V} $. Then, the visual object and textual word representation encoding the cross-modal correlation are calculated as follows:
| $ {\boldsymbol{v }}' = softmax({{\boldsymbol{Q }}_v}{\boldsymbol{K }}_v^T){{\boldsymbol{V }}_e} $ | (7) |
| $ {\boldsymbol{e }}' = softmax({{\boldsymbol{Q }}_e}{\boldsymbol{K }}_e^T){{\boldsymbol{V }}_v} $ | (8) |
where $ {\boldsymbol{v }}' $ denotes the set of visual object representation which captures the inter-modal cross-modal correlation, and $ {\boldsymbol{e }}' $ denotes the set of textual word representation. By these operations, we can obtain another representation for each textual word and visual object. To obtain the final representation $ {v^1} \in {R^d} $ for the whole image, the learned $ {\boldsymbol{v }}' $ is passed into an average pool layer. The learned $ {\boldsymbol{e }}' $ is passed into an 1d-CNN layer followed by a max pool layer to obtain the final representation for the whole text description $ {{\boldsymbol{e }}^1} \in {R^d} $.
As discussed above, we have learned multiple features from the multimodal content. Meanwhile, there is also another type of feature which encode the structure information of knowledge graph, which is also the main feature learned in other works [10,18,19]. To capture the structure information, we use the structure-based methods, such as TransE [18], to learn the raw structure feature $ {{\boldsymbol{s }}^0} \in {R^x} $ of each entity node. In the end, we obtain two types of visual feature $ {{\boldsymbol{v }}^0} $ and $ {{\boldsymbol{v }}^1} $, two types of textual feature $ {{\boldsymbol{e }}^0} $ and $ {{\boldsymbol{e }}^1} $, and the structural feature $ {{\boldsymbol{s }}^0} $.
Finally, all these features are fused to obtain a final representation of the entity node. The feature fusion operation is composed of several steps. First, the two types of textual features are fused using multi-modal factorized bilinear pooling (MFB) [42] as follows:
| $ {{\boldsymbol{e }}^2}{{\text{ = }}}MFB({{\boldsymbol{e }}^0},{{\boldsymbol{e }}^1}) = \sum\limits_{i = 1}^k {({\boldsymbol{U }}_i^T{{\boldsymbol{e }}^0} \circ {\boldsymbol{G }}_i^T{{\boldsymbol{e }}^1})} $ | (9) |
where ◦ denotes the element wise multiplication operation, k denotes the number of factors in MFB. Similarly, the two types of visual features are also fused with MFB to obtain the final representation $ {{\boldsymbol{v }}^2} $. Then, we fuse the visual feature $ {{\boldsymbol{v }}^2} $, the textual feature $ {{\boldsymbol{e }}^2} $ and the structural feature $ {{\boldsymbol{s }}^0} $ to obtain the final representation of the entity using gating network with softmax function as follows:
| $ (\alpha ,\beta ,\gamma ){\text{ = }}softmax(\frac{{{\boldsymbol{G }}}_{e}^{T}{v}^{2}}{\sqrt{d}},\frac{{{\boldsymbol{G }}}_{e}^{T}{e}^{2}}{\sqrt{d}},\frac{{{\boldsymbol{G }}}_{e}^{T}{s}^{0}}{\sqrt{d}}) $ | (10) |
| $ {{\boldsymbol{z }}_e}{{\text{ = }}}\alpha {{\boldsymbol{v }}^2} + \beta {{\boldsymbol{e }}^2} + \gamma {{\boldsymbol{s }}^0} $ | (11) |
where $ {{\boldsymbol{z }}_e} $ is the final representation of the whole entity, which can be a head entity $ {{\boldsymbol{z }}_h} $ or a tail entity $ {{\boldsymbol{z }}_t} $. As a result, the learned representation of an entity captures both the structure information and multimodal content, which encodes the content by exploiting the intra-modal and inter-modal correlation. When α and β are trained to be 0, the representation only contains the structural features, which is similar to the existing models [10,18,19]. Therefore, our approach is more effective to learn the representation.
Usually, the relation in knowledge graph exists between a head entity and tail entity, which rarely contains other content information. To improve the representation of relation, we use the multimodal features learned from the head entity and tail entity to enhance the semantics information of relation. Meanwhile, this process can also capture the interaction between entities and the corresponding relation, and thus further improves the representation learning of relation. The Bilinear Network is used to fuse the entity feature and relation structure feature as follows:
| $ {{\boldsymbol{z }}_r}{{\text{ = }}}\sigma {{\text{(}}}{\bf{W }}_h^T{{\boldsymbol{z }}_h}) \circ \sigma {{\text{(}}}{\bf{W }}_t^T{{\boldsymbol{z }}_t}) \circ \sigma {{\text{(}}}{\bf{W }}_r^T{{\boldsymbol{r }}_s}){{\text{ + }}}{{\boldsymbol{b }}_r} $ | (12) |
where σ is a nonlinear activation function, $ {{\boldsymbol{r }}_s} $ is the relation representation learned by other structure-based method [18], $ {{\boldsymbol{z }}_h} $ and $ {{\boldsymbol{z }}_t} $ are the fused representation of the head entity and tail entity respectively. This formulation is also used to learn the structural information of a triple.
| Algorithm 1 the training of MMCF |
| Input: triples (h, r, t) of a graph G; Output: $ {{\boldsymbol{z }}_h} $, $ {{\boldsymbol{z }}_r} $, $ {{\boldsymbol{z }}_t} $ 1: For each entity in G 2: Feature Extraction 2: Extract the visual representation $ {o'_i} $ of each object by Eq (1); 6: Extract the word representation wi; 3: Transform $ {o'_i} $ to vi by fully-connect layer; 4: Intra-modal Correlation Learning 4: Learn the intra-modal correlation between vi by Eq (2); 5: All the vi s are aggregated to obtain the image representation $ {{\boldsymbol{v }}^{_0}} \in {R^d} $; 6: Extract the word representation wi; 7: Use 1-dim CNN to encode the intra-modal correlation of wi by Eq (5); 8: Obtain text description embedding $ {{\boldsymbol{e }}^0} \in {R^d} $ by Eq (6). 9: Inter-modal Correlation Learning 9: Learn visual representation $ {v^1} \in {R^d} $ based on inter-modal correlation by Eq (7); 10: Learn textual representation $ {{\boldsymbol{e }}^1} \in {R^d} $ based on inter-modal correlation by Eq (8); 12: Cross-modal Feature Fusion 11: Obtain text representation $ {{\boldsymbol{e }}^2} $ by fusing $ {{\boldsymbol{e }}^0} $ and $ {{\boldsymbol{e }}^1} $ using Eq (9); 12: Obtain visual representation $ {{\boldsymbol{v }}^2} $ by fusing $ {{\boldsymbol{v }}^0} $ and $ {{\boldsymbol{v }}^1} $ using Eq (9); 13: Combine $ {{\boldsymbol{v }}^2} $, $ {{\boldsymbol{e }}^2} $, $ {{\boldsymbol{s }}^0} $ to obtain entity representation $ {{\boldsymbol{z }}_h} $ or $ {{\boldsymbol{z }}_t} $ by Eq (11); 14: end for 15: For each relation in G 16: Learn relation representation $ {{\boldsymbol{z }}_r} $ by Eq (12) 17: end for 18: Minimize $ \phi {{\text{(}}}h, r, t{{\text{)}}} $ by Eq (13). |
DownLoad:
CSV
In the decoder module, a scoring function is applied to learn the representation. Many of current score functions proposed by other works [10,18,19] can be used since our method can be directly extended to these models. For example, we use the MuRE [10] scoring function to calculate a score for a triple as follows:
| $ \phi {{\text{(}}}h,r,t{{\text{) = }}} - d{({\bf{R }}{{\boldsymbol{z }}_h},{{\boldsymbol{z }}_t} + {{\boldsymbol{r }}_f})^2} + {{\boldsymbol{b }}_h} + {{\boldsymbol{b }}_t} $ | (13) |
where $ d(.) $ is a Euclidean distance function, $ {\bf{R }} $ is a relation-specific matrix, $ {{\boldsymbol{b }}_h} $ and $ {{\boldsymbol{b }}_t} $ are the biases of the head and tail entities. The scoring function aim to give a high value to the positive triple, and a small value to the negative triple. With this function, we can train our model on the training dataset to learn the representation of entity and relation. Meanwhile, the other types of decoders proposed by other models can also be used to learn the scoring function, such as InteractE [11] and TransE [12]. The training process of MMCF is shown in Algorithm 1.
To evaluate the performance of our approach, extensive experiments are conducted to compare our approach with other approaches. Meanwhile, the effectiveness of each component in our model is also verified.
Three public datasets are used in the experiments, i.e., FB-IMG [12], WN18RR [43] and FB15k-237 [44]. The dataset FB-IMG has already included high-quality multimodal content to knowledge graph. It contains the embedding representation for entities and relations, and representation of the textual description and images associated with the entities. WN18RR is built on the base of WN18[18]. It removes the inverse relations, which makes the test triples can't be inferred from the inverse of training examples directly. FB15k-237 is a revision of FB15k [18], in which all the inverse relations are also removed. Though the two datasets are widely used in KGE evaluation, they contain only the structure information. To associate the datasets with external multimodal content, Yao et al. [14] download the text description for the triples in WN18RR and FB15k-237. Based on the work [14], we further extend the two datasets with text description and images. The names of entities are used as keywords to crawl the related images from the web search engines, such as Google and Bing. Then, the top-15 images and the text content, such as title, caption, abstract of each entity are downloaded. We manually select the image and the text content which is most related to the corresponding entity to supplement the multimodal content of the two datasets. Since the relation denotes the structure between entities, it is difficulty to be directly described by other multimodal content. Therefore, we use the data of the associated entities to enhance the learning of relation embedding as discussed above. We show the statistics information of these datasets in Table 1.
| Dataset | #entities | #relation | #train | #valid | #test |
| FB-IMG | 11,757 | 1,231 | 285,850 | 29,580 | 34,863 |
| WN18RR | 40,943 | 11 | 86,835 | 3,034 | 3,134 |
| FB15k-237 | 14,541 | 237 | 272,115 | 17,535 | 20,466 |
DownLoad:
CSV
We use the pre-trained language model BERT [13] to obtain a 300-dimensional vector for the text word. The whole text content is finally represented by a 1024-dimensional vector. As for the image, the top-15 objects extracted by the pretrained Fast R-CNN [39] with the highest accuracy are selected and each one is represented by a 2048-dimensional vector. The Adam et al. [45] optimizer is used in the experiments, whose initial learning rate is 1 × 10−4, and then decreases at a rate of 0.9 every 20 epochs. By adjusting the parameters, the model with the highest F1 value in the verification set is finally selected. All of the experiments are conducted on 2 NVIDIA RTX 3090 24 GB.
Usually, the task of link prediction is used to evaluate the quality of the representation learned by the embedding method. It is used to predict the missed facts of knowledge graph, which is also an effective way to solve the problem of incompleteness of KGs. The task of link prediction in KG is formulated as inferring the missed head entity given (−, r, t), or the missed tail entity given (h, r, −). The trained model calculates the plausibility scores of all possible triples in the test set, and then the ranking result of these triples is used for evaluation. In the experiment, we use the popular matrices of Mean Reciprocal Rank (MRR) and HNtoevaluatetherankingresult.MRRisthemeanreciprocalsofalltherankingresultofthetestsamples.HITSN denotes the hits occur at the N-th position, which denotes the average proportion of positive triples that rank less than N in the ranking list, N = 1/3/10.
To evaluate the performance of our approach, we compare it with two categories of models, i.e., the structure-based models and external information-fused models.
The structure-based models mainly learn the embedding of entity and relation based on the graph structure information, such as the triplet set. In the experiment, MuRE and MuRP [10], InteractE [11], ConvKB [26], HypER [27], DistMult [46] and M2GNN [47] are used as the baselines, and the link prediction result on WN18RR and FB15k-237 published by these models are directly used for comparison. As for the dataset FB-IMG, we also reproduce several baseline modes for comparison, such as MuRE [10], InteractE [11], ComplEx [35], ConvE [48]. There are also some models exploiting external information for knowledge graph embedding, such as the entity-related text descriptions or images. On the datasets WN18RR and FB15k-237, we adopt the baseline model KG-BERT [14] for comparison, which exploits the external text content for KGE and achieves the state-of-the-art performance. On the dataset FB-IMG, there are some other works exploiting the external multimodal content for KGE, such as TransE [12] and MKRL [36]. Accordingly, we compare with these baseline models on FB-IMG.
In the first experiment, we compare the performance of link prediction of our model MMCF with the baseline models on the datasets WN18RR and FB15k-237 which introduces the textual and visual content. We implement MMCF with two decoders MuRE [8] and InteractE [7]. The comparison result is shown in Table 2. From the table, several conclusions can be derived.
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MuRE | 0.475 | 0.436 | 0.487 | 0.554 | 0.336 | 0.245 | 0.370 | 0.521 |
| MuRP | 0.481 | 0.440 | 0.495 | 0.566 | 0.335 | 0.243 | 0.367 | 0.518 |
| InteractE | 0.463 | 0.430 | − | 0.528 | 0.354 | 0.263 | − | 0.535 |
| ConvKB | 0.249 | 0.057 | 0.417 | 0.524 | 0.243 | 0.155 | 0.371 | 0.421 |
| HypER | 0.465 | 0.436 | 0.477 | 0.522 | 0.341 | 0.252 | 0.376 | 0.520 |
| DistMult | 0.430 | 0.390 | 0.440 | 0.490 | 0.241 | 0.155 | 0.263 | 0.419 |
| M2GNN | 0.485 | 0.444 | 0.498 | 0.572 | 0.362 | 0.275 | 0.398 | 0.565 |
| ComplEx | 0.440 | 0.410 | 0.460 | 0.510 | 0.247 | 0.158 | 0.275 | 0.428 |
| ConvE | 0.430 | 0.400 | 0.440 | 0.520 | 0.325 | 0.237 | 0.356 | 0.501 |
| KG-BERT | − | − | − | 0.524 | − | − | − | 0.420 |
| MMCFMuRE | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
| MMCFInteractE | 0.483 | 0.448 | 0.495 | 0.570 | 0.367 | 0.273 | 0.402 | 0.563 |
DownLoad:
CSV
First, our model MMCF achieves the best performance of multiple metrices on the two datasets. The result demonstrates that learning the intra-modal and inter-modal correlation to fuse the multimodal content with the structure information is effective to improve the performance of knowledge graph embedding. Second, whichever of the two decoders MuRE [8] and InteractE [7] is applied, MMCF outperforms other models in most of the matrices. Therefore, it also demonstrates that including the multimodal content with cross-modal correlation can improve the performance of the existing KGE models. Meanwhile, MMCFMuRE performs better than MMCFInteractE on WN18RR, which is the same as that MuRE performs better than InteractE. The same observation can also be found in FB15k-237. Therefore, an effective decoder also contributes to the performance of MMCF. Third, compared with other models that include the related content for knowledge graph embedding, such as KG-BERT, our model still improves the performance. It demonstrates that exploiting the intra-modal and inter-modal correlation to fuse multimodal content is more effective than directly fusing the multimodal for KGE.
In the other experiment, we also compare MMCF with the baselines on the dataset FB-IMG. The comparison result is shown in Table 3, where MMCFTreansE denotes that MMCF adopts TransE [12] as the decoder. From the table, it is observed that our modal outperforms than these baselines. The result further demonstrates that the multimodal content of entities supplements the structural information for KGE, and exploiting the intra-modal and inter-modal correlation is also effective to learn the representation for multimodal content.
| Models | MRR | HITS@1 | HITS@3 | HITS@10 |
| TransE | − | − | − | 0.494 |
| MKRL | − | − | − | 0.645 |
| MuRE | 0.765 | 0.703 | 0.807 | 0.874 |
| InteractE | 0.813 | 0.762 | 0.849 | 0.895 |
| ConvKB | 0.449 | 0.337 | 0.513 | 0.621 |
| ComplEx | 0.525 | 0.392 | 0.618 | 0.754 |
| ConvE | 0.747 | 0.667 | 0.804 | 0.882 |
| MMCFTransE | 0.453 | 0.342 | 0.525 | 0.702 |
| MMCFMuRE | 0.820 | 0.768 | 0.852 | 0.899 |
| MMCFInteractE | 0.819 | 0.771 | 0.849 | 0.897 |
DownLoad:
CSV
To evaluate the effectiveness of each component in MMCF, we design a set of ablation experiments on multi-modal dataset FB-IMG. Meanwhile, a set of various versions of MMCF are designed as follows:
MMCF-intra. It removes the intra-modal correlation learning components for both the two modalities, and only the inter-modal correlation is encoded to learn the multimodal representation.
MMCF-inter. It removes the inter-modal correlation learning components, and only the intra-modal correlation is encoded to learn the multimodal representation.
MMCF-gating. It removes the gating network, and then the feature output by the cross-modal correlation learning module and the structure feature are added element-wise.
MMCF-text. It removes the text description from the multimodal content, and then only the image is used as the external content of the entities.
MMCF-image. It removes the image from the multimodal content, and then only the textual description is used as the external content of the entities.
MMCF-entity. The features of entities are not fused into the relation feature. It mainly tests whether the entity features are useful for relation representation learning.
Then, the ablation experiment is conducted on WN18RR and FB15k-237 with the decoder MuRE. Table 4 shows the experiment result. From the table, we can obtain several conclusions. First, MMCF with all components to exploit the multi-modal content obtains the best performance. Second, all the components contribute to the performance of MMCF, and the related multimodal content is useful to knowledge graph embedding. MMCF without image obtains the second-best performance, while the performance of MMCF without the text description decrease greatly. This is might because that the text content is more effective to reflect the semantics of entity than image. Third, including the entity features to the relation is effective to improve the learning of relation representation, since the performance of MMCF-entity also decreases greatly. The entity features can enrich the semantics of relation.
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MMCF-intra | 0.471 | 0.426 | 0.479 | 0.554 | 0.342 | 0.256 | 0.384 | 0.544 |
| MMCF-inter | 0.475 | 0.429 | 0.482 | 0.556 | 0.348 | 0.258 | 0.389 | 0.546 |
| MMCF-gating | 0.480 | 0.435 | 0.491 | 0.566 | 0.353 | 0.265 | 0.396 | 0.552 |
| MMCF-text | 0.458 | 0.421 | 0.471 | 0.549 | 0.335 | 0.252 | 0.380 | 0.538 |
| MMCF-image | 0.482 | 0.441 | 0.494 | 0.562 | 0.356 | 0.267 | 0.394 | 0.549 |
| MMCF-entity | 0.467 | 0.438 | 0.485 | 0.559 | 0.339 | 0.254 | 0.387 | 0.542 |
| MMCF | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
DownLoad:
CSV
In this section, we analyze the impact of embedding size and also give a case study. We recode the experiment result by setting the embedding of MMCF with different dimension size {20, 40, 60, 80,100,120,140,160} on WN18RR. From the table, it can be found that the performance is improved greatly with the embedding size increased in the early stage. Then the performance is maintained for a period, and decreases very slowly. Therefore, it demonstrates that a large vector is not always perform better than a small vector of the embedding. Moreover, our model is not very sensitive to the size of the embedding size when the size reaches a certain number.
To visualize the performance in detail, we present an example of ranking list for a query (peach, _hypernym, −). Table 6 shows the result of MMCF and MuRE, where MMCF use MuRE as the decoder. From the table, it can be observed that the correct answer is located in the first position of the ranking list returned by MMCF. MuRE ranks the correct answer in the third position. The other method TransE which also fuse the external multimodal content to embed knowledge graph ranks the correct item in the second position. Therefore, our method obtains the best result in this example. This is might because that stone fruit contain some text and visual content that describe the nature of peach. MMCF can learn the latent correlation between the multimodal content of different entities, which is then more effective to infer the related entity. From the experiment result, it is further demonstrated that the external multi-modal content is useful for knowledge graph embedding.
| Embedding Size | MRR | HITS@1 | HITS@3 | HITS@10 |
| 20 | 0.368 | 0.395 | 0.421 | 0.523 |
| 40 | 0.476 | 0.428 | 0.468 | 0.551 |
| 60 | 0.483 | 0.435 | 0.490 | 0.562 |
| 80 | 0.487 | 0.440 | 0.495 | 0.567 |
| 100 | 0.489 | 0.443 | 0.497 | 0.571 |
| 120 | 0.488 | 0.441 | 0.495 | 0.572 |
| 140 | 0.487 | 0.438 | 0.493 | 0.570 |
| 160 | 0.485 | 0.435 | 0.490 | 0.565 |
DownLoad:
CSV
| Top-N | MMCF | MuRE | TransE |
| 1 | stone fruit | structure | fruit tree |
| 2 | fruit | fruit tree | stone fruit |
| 3 | citrus fruit | stone fruit | veggie |
| 4 | fruit tree | seasoning | tree |
| 5 | structure | computer memory unit | nut |
| 6 | monocot genus | veggie | citrus fruit |
| 7 | veggie | citrus fruit | monocot genus |
| 8 | tree | monocot genus | structure |
| 9 | produce | tree | root |
| 10 | root | root | animal |
DownLoad:
CSV
Our model is built on the exiting KGE model to exploit the external multimodal knowledge to improve the embedding of knowledge graph. Since it learns the inter-modal and intra-modal correlation of different modalities of content, it needs more running time than the traditional KGE algorithm, such as TransE, MuRE and InteractE. The parameter size of our model is about 4.2M. The training of our model is conducted on 2 NVIDIA RTX 3090 24GB, which takes about 12 hours for one dataset. However, the algorithm can be optimized by distributed, parallel and cluster computing.
This study is designed to deeply exploit the external multimodal content for knowledge graph embedding. In reality, there is usually a great volume of different types of data related to entities and relation, such as text description, web pages, medical images, web images, audio and videos and so on. Many of the data can be easily obtained from different sources, such as Web sites, traditional databases and medical datasets, etc. Therefore, it is reasonable to exploit the multimodal data to improve the performance of traditional knowledge graph embedding methods. Accordingly, there are already some works [14,37] to fuse the external data for knowledge graph embedding, which has achieved certain success. However, these methods mainly regard the different types of data as a whole or directly fuse the features of external data with the features learned from knowledge graph. Therefore, these methods are not effective to learn from the multimodal data since different types of data are heterogeneous and there exists cross-modal correlation.
Our method first learns the representation of different modalities of data by exploiting the intra-modal correlation, and then the features of different modalities are fused by encoding the inter-modal correlation. Finally, the features learned from the multimodal content and graph structural information are fused by a gating network. Therefore, our method gives consideration to the characteristic of multimodal data, and thus it is more effective to fuse the multimodal content for knowledge graph embedding. The experiment result also demonstrates the superiority of our method, by comparing with the structure-based methods and multimodal content fusion-based methods. By using the same decoder, our model performs better than the original models MURE, TransE and InteractE. Though we mainly fuse the text description and image for knowledge graph embedding in this paper, the other types of data, such as video and audio, can also be fused by extending our method directly. Moreover, the framework of our method can also be used or revised in other domains which needs to handle multimodal data, such as network embedding, multimodal knowledge graph construction, visual question answering, multimodal data classification and so on. The limitation of our method is that it might be more complex than the structure-based methods and other multimodal content fusion-based methods, since it further learns the fine-granularity cross-modal correlation between different types of data. However, this problem can be alleviated by parallel computing.
In this paper, we propose to learn knowledge graph embedding by exploiting the cross-modal correlation between the multimodal content related to the entities. Specifically, a novel model is proposed to exploit the intra-modal and inter-modal correlation for multimodal representation learning, which then fused with the structure features for entity and relation representation learning. It is different from existing works which learn entity embedding mainly base on the structure information or include the external data as a whole. We evaluate the performance on three datasets, and the result demonstrate the superiority of the proposed model. Meanwhile, our model can be easily combined with other structure-based models, such as MuRE, TransE and InteractE.
In the future works, it is interesting to exploit the multi-modal pre-training models to more effectively learning the context semantics of entity. Moreover, this model can also be combined with other embedding models, such as network embedding.
The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.
This work is supported by State Grid Corporation of China Big Data Center Technology Project (SGSJ0000YFJS2200066).
All authors declare no conflicts of interest in this paper.
| [1] | Keshk SMAS (2014) Bacterial cellulose production and its industrial applications. J Bioproces Biotechniq 4: 2. |
| [2] |
Rajwade JM, Paknikar KM, Kumbhar JV (2015) Applications of bacterial cellulose and its composites in biomedicine. Appl Microbiol Biotechnol 99: 2491-2511. doi: 10.1007/s00253-015-6426-3
|
| [3] | Araujo S, Moreira da Silva F, Gouveia IC (2015) The role of technology towards a new bacterial-cellulose-based material for fashion design. J Ind Intell Inf 3: 168-172. |
| [4] |
Castellane TCL, Persona MR, Campanharo JC, et al. (2015) Production of exopolysaccharide from rhizobia with potential biotechnological and bioremediation applications. Int J Biol Macromol 74: 515-522. doi: 10.1016/j.ijbiomac.2015.01.007
|
| [5] |
Ross P, Weinhouse H, Aloni Y, et al. (1987) Regulation of cellulose synthesis in Acetobacter xylinum by cyclic diguanylic acid. Nature 325: 279-281. doi: 10.1038/325279a0
|
| [6] | Ude S, Arnolod DL, Moon CD, et al. (2006) Biofilm formation and cellulose expression among diverse environmental Pseudomonas isolates. Environ Microbiol 8: 1997-2011. |
| [7] |
Zogaj X, Nimtz M, Rohde M, et al. (2001) The multicellular morphotypes of Salmonella typhimurium and Escherichia coli produce cellulose as the second component of the extracellular matrix. Mol Microbiol 39: 1452-1463. doi: 10.1046/j.1365-2958.2001.02337.x
|
| [8] | Romling U (2002) Molecular biology of cellulose production in bacteria. ResMicrobiol 153: 205-212. |
| [9] |
Solano C, Garcia B, Valle J, et al. (2002) Genetic analysis of Salmonella enteritidis biofilm formation: critical role of cellulose. Mol Microbiol 43: 793-808. doi: 10.1046/j.1365-2958.2002.02802.x
|
| [10] | Mohite BV, Patil SV (2014) Physical, structural, mechanical and thermal characterization of bacterial cellulose by G. hansenii NCIM 2529. Carbohyd Polym106: 132-141. |
| [11] | Morgan JL, Strumillo J, Zimmer J (2013) Crystallographic snapshot of cellulose synthesis and membrane translocation. Nature 493: 181-186. |
| [12] | Amikam D, Benziman M (1989) Cyclic diguanylic acid and cellulose synthesis in Agrobacterium tumefaciens. J Bacteriol 171: 6649-6655. |
| [13] | Romling U (2012) Cyclic di-GMP, an established secondary messenger still speeding up. Environ Microbiol 14: 1817-1829. |
| [14] | Whiteley CG, Lee DJ (2014) Bacterial diguanylate cyclases: Structure, function and mechanism in exopolysaccharide biofilm development. Biotech Adv 33: 124-41. |
| [15] | Morgan JL, Mcnamara JT, Zimmer J (2014) Mechanism of activation of bacterial cellulose synthase by cyclic di-GMP. Nature Struc Biol 21: 489-496. |
| [16] | Dazzo FB, Truchet GL, Sherwood JE, et al. (1984) Specific phases of root hair attachment in the Rhizobium trifolii-clover symbiosis. Appl Environ Microbiol 48: 1140-1150. |
| [17] |
Mateos PF, Baker DL, Philip-Hollingsworth S, et al. (1995) Direct in situ identification of cellulose microfibrils associated with Rhizobium leguminosarum biovar trifolii attached to the root epidermis of white clover. Can J Microb 41: 202-207. doi: 10.1139/m95-028
|
| [18] | Flemming HC, Wingender J (2010) The biofilm matrix. Nat Rev 8: 624-633. |
| [19] |
McCrate OA, Zhou X, Reichhardt C, et al. (2013) Sum of the Parts: Composition and Architecture of the Bacterial Extracellular Matrix. J Mol Biol 425: 4286-4294. doi: 10.1016/j.jmb.2013.06.022
|
| [20] |
Koeck DE, Pechtl A, Zerlov VV, et al., (2014) Genomics of cellulolytic bacteria Curr Opin Biotech 29: 171-183. doi: 10.1016/j.copbio.2014.07.002
|
| [21] |
Lombard V, Ramulu HG, Drula E, et al. (2014) The carbohydrate-active enzymes database (CAZy) in 2013. Nucleic Acids Res 42: D490-D495. doi: 10.1093/nar/gkt1178
|
| [22] |
Davies G, Henrissat B (1995) Structures and mechanisms of glycosyl hydrolases. Structure 3: 853-859. doi: 10.1016/S0969-2126(01)00220-9
|
| [23] |
Bourne Y, Henrissat B (2001) Glycoside hydrolases and glycosyltransferases: families and functional modules. Curr Opin Struct Biol 11: 593-600. doi: 10.1016/S0959-440X(00)00253-0
|
| [24] |
Sadhu S, Maiti TK (2013) Cellulase production by bacteria: a review. British Microbiology Research Journal 3: 235-258. doi: 10.9734/BMRJ/2013/2367
|
| [25] | Fuji K, Satomi M, Fukui Y, et al. (2013) Streptomyces abietis sp. nov., a cellulolytic bacterium isolated from soil of a pine forest. Int J Syst Evol Microbiol 63: 4754-4759. |
| [26] | Cole JK, Gieler BA, Heisler DL, et al. (2013) Kallotenue papyrolyticum gen. nov., sp. nov., a cellulolytic and filamentous thermophile that represents a novel lineage (Kallotenuales ord. nov., Kallotenuaceae fam. nov.) within the class Chloroflexia. Int J Syst Evol Microbiol 63: 4675-4682. |
| [27] | Podosokorskaya OA, Bonch-Osmolovskaya EA, Novikov AA, et al. (2013) Ornatilinea apprima gen. nov., sp. nov., a cellulolytic representative of the class Anaerolineae. Int J Syst Evol Microbiol 63: 86-92. |
| [28] | Hatamoto M, Kaneshige M, Nakamura A, et al. (2014) Bacteroides luti sp. nov., an anaerobic, cellulolytic and xylanolytic bacterium isolated from methanogenic sludge. Int J Syst Evol Microbiol 64: 1770-1774. |
| [29] | Kusube M, Sugihara A, Moriwaki Y, et al. (2014) Alicyclobacillus cellulosilyticus sp. nov., a thermophilic, cellulolytic bacterium isolated from steamed Japanese cedar chips from a lumbermill. Int J Syst Evol Microbiol 64: 2257-2263. |
| [30] | Horino H, Fujita T, Tonouchi A (2014) Description of Anaerobacterium chartisolvens gen. nov., sp. nov., an obligately anaerobic bacterium from Clostridium rRNA cluster III isolated from soil of a Japanese rice field, and reclassification of Bacteroides cellulosolvens Murray et al. 1984 as Pseudobacteroides cellulosolvens gen. nov., comb. nov. Int J Syst Evol Microbiol 64: 1296-1303. |
| [31] | Bing W, Wang H, Zheng B, et al. (2015) Caldicellulosiruptor changbaiensis sp. nov., a cellulolytic and hydrogen-producing bacterium from a hot spring. Int J Syst Evol Microbiol 65: 293-297. |
| [32] | Koeck DE, Ludwig W, Wanner G, et al. (2015) Herbinix hemicellulosilytica, gen. nov., sp. nov., a thermophilic cellulose-degrading bacterium isolated from a thermophilic biogas reactor. Int J Syst Evol Microbiol [in press]. |
| [33] | Menéndez E, Ramírez-Bahena MH, Fabryová A, et al. (2015) Pseudomonas coleopterorum sp. nov., a cellulase producing bacterium isolated from the bark beetle Hylesinus fraxini. Int J Syst Evol Microbiol [in press]. |
| [34] | Huang S, Sheng P, Zhang H (2012) Isolation and Identification of Cellulolytic Bacteria from the Gut of Holotrichia parallela Larvae (Coleoptera: Scarabaeidae) Int J Mol Sci 13: 2563-2577. |
| [35] | Hameed A, Shahina M, Lai WA, et al. (2015) Oricola cellulosilytica gen. nov., sp. nov., a cellulose-degrading bacterium of the family Phyllobacteriaceae isolated from surface seashore water, and emended descriptions of Mesorhizobium loti and Phyllobacterium myrsinacearum. A van Leeuw J Microb 107: 759-771. |
| [36] | Sethi S, Datta A, Gupta L, et al. (2013) Optimization of Cellulase Production from Bacteria Isolated from Soil. ISRN Biotechnology 2013: 985685. |
| [37] | Howard JA, Hood E (2005) Bioindustrial and biopharmaceutical products produced in plants. Adv Agron 85: 91-124. |
| [38] | Yu LX, Gray BN, Rutzke CJ, et al. (2007) Expression of thermostable microbial cellulases in the chloroplasts of nicotine-free tobacco. J Biotechnol1 31: 362-369. |
| [39] |
Rastogi G, Muppidi GL, Gurram RN, et al. (2009) Isolation and characterization of cellulose-degrading bacteria from the deep subsurface of the Homestake gold mine, Lead, South Dakota, USA. J Ind Microbiol Biotechnol 36: 585-598. doi: 10.1007/s10295-009-0528-9
|
| [40] |
Trivedi N, Gupta V, Kumar M, et al. (2011) An alkali-halotolerant cellulase from Bacillus flexus isolated from green seaweed Ulva lactuca. Carbohyd Polym 83: 891-897. doi: 10.1016/j.carbpol.2010.08.069
|
| [41] |
DiPasquale L, Romano I, Picardello G, et al., (2014) Characterization of a native cellulase activity from an anaerobic thermophilic hydrogen-producing bacterium Thermosipho sp. strain 3. Ann Microbiol 64: 1493-1503. doi: 10.1007/s13213-013-0792-9
|
| [42] |
Dalmaso GZL, Ferreira D, Vermelho AB (2015) Marine extremophiles: a source of hydrolases for biotechnological applications. Mar Drugs 13: 1925-1965. doi: 10.3390/md13041925
|
| [43] |
Kinet R, Destain J, Hiligsmann S, et al. (2015) Thermophilic and cellulolytic consortium isolated from composting plants improves anaerobic digestion of cellulosic biomass: Toward a microbial resource management approach. Bioresource Technol 189: 138-144. doi: 10.1016/j.biortech.2015.04.010
|
| [44] |
Voget S, Steele HL, Streit WR (2006) Characterization of a metagenome-derived halotolerant cellulase. J Biotechnol 126: 26-36. doi: 10.1016/j.jbiotec.2006.02.011
|
| [45] | Duan CJ, Feng JX (2010) Mining metagenomes for novel cellulase genes. Biotechnol Lett 32: 1765-1775. |
| [46] |
Liu J, Liu W, Zhao X, et al. (2011) Cloning and functional characterization of a novel endo-β-1,4-glucanase gene from a soil-derived metagenomic library. Appl Microbiol Biotechnol 89: 1083-1092. doi: 10.1007/s00253-010-2828-4
|
| [47] | Adrio JL, Demain AL (2014) Microbial enzymes: tools for biotechnological processes. Biomolecules 4: 117-139. |
| [48] | Mori T, Kamei I, Hirofumi H, et al. (2014) Identification of novel glycosyl hydrolases with cellulolytic activity against crystalline cellulose from metagenomic libraries constructed from bacterial enrichment cultures. Springer Plus 3: 365. |
| [49] | Attri S, Garg G (2014) Isolation of microorganisms simultaneously producing xylanase, pectinase and cellulase enzymes using cost effective substrates. J Innovative Biol 1: 45-50. |
| [50] | Singh G, Singh AK (2014) Alternative substrates for the amylase and cellulase production with rhizobial isolates. Int J Avd Res Sci Technol 3: 79-85. |
| [51] | Teather RM, Wood PJ (1982) Use of Congo red-polysaccharide interactions in enumeration and characterization of cellulolytic bacteria from the bovine rumen. Environ Microbiol 43: 777-780. |
| [52] |
Kasana RC, Salwan R, Dhar H, et al. (2008) A rapid and easy method for the detection of microbial cellulases on agar plates using gram’s iodine. Curr Microbiol 57: 503-507. doi: 10.1007/s00284-008-9276-8
|
| [53] | Taha M, Kadali KK, Al-Hothaly K, et al. (2015) An effective microplate method (Biolog MT2) for screening native lignocellulosic-straw-degrading bacteria. Ann Microbiol [in press]. |
| [54] |
Zhang YHP, Himmel ME, Mielenz JR (2006) Outlook of cellulase improvement: screening and selection strategies. Biotechnol Adv 24: 452-481. doi: 10.1016/j.biotechadv.2006.03.003
|
| [55] | Sadhu S, Ghosh PK, Aditya G, et al. (2014) Optimization and strain improvement by mutation for enhanced cellulase production by Bacillus sp. (MTCC10046) isolated from cow dung. J King Saud Univ 26: 323-332. |
| [56] | Sangkharak K, Vangsirikul P, Janthachat S (2012) Strain improvement and optimization for enhanced production of cellulase in Cellulomonas sp. TSU-03. African J Microbiol Res 6: 1079-1084. |
| [57] | Maki M, Leung KT, Qin W (2009) The prospects of cellulose-producing bacteria for the bioconversion of lignocellulosic biomass. Int J Biol Sci 5: 500-516. |
| [58] |
Zhao H, Chockalingam K, Chen Z (2002) Directed evolution of enzymes and pathways for industrial biocatalysis. Curr Opin Biotechnol 13: 104-110. doi: 10.1016/S0958-1669(02)00291-4
|
| [59] | Cherry JR, Fidantsef AL (2003) Directed evolution of industrial enzymes: an update. Curr Opin Biotechnol 14: 438-443. |
| [60] |
Hasunuma T, Okazaki F, Okai N, et al. (2013) A review of enzymes and microbes for lignocellulosic biorefinery and the possibility of their application to consolidated bioprocessing technology. Bioresource Technol 135: 513-522. doi: 10.1016/j.biortech.2012.10.047
|
| [61] | Garvey M, Klose H, Fischer R, et al. (2014) Cellulases for biomass degradation: comparing recombinant cellulase expression platforms. Trends Biotechnol 31: 581-593. |
| [62] |
Juturu V, Wu JC (2014) Microbial cellulases: Engineering, production and applications. Renew Sust Energ Rev 33: 188-203. doi: 10.1016/j.rser.2014.01.077
|
| [63] |
Lambertz C, Garvey M, Klinger J, et al. (2014) Challenges and advances in the heterologous expression of cellulolytic enzymes: a review. Biotechnol Biofuels 7: 135. doi: 10.1186/s13068-014-0135-5
|
| [64] |
Munjal N, Jaewd K, Wajid S, et al. (2015) A Constitutive Expression System for Cellulase Secretion in Escherichia coli and Its Use in Bioethanol Production. PLoS ONE 10: e0119917. doi: 10.1371/journal.pone.0119917
|
| [65] | Chung D, Cha M, Guss AM, et al. (2015) Direct conversion of plant biomass to ethanol by engineered Caldicellulosiruptor bescii. P Natl Acad Sci U S A 111: 8931-8936. |
| [66] |
Robledo M, Jimenez-Zurdo JI, Soto MJ, et al. (2011) Development of functional symbiotic white clover root hairs and nodules requires tightly regulated production of rhizobial cellulase CelC2. Mol Plant Microbe Interact 24: 798-807. doi: 10.1094/MPMI-10-10-0249
|
| [67] | Batthacharya AS, Batthacharya A, Pletschke BI (2015) Synergism of fungal and bacterial cellulases and hemicellulases: a novel perspective for enhanced bio-ethanol production. Biotechnol Lett [in press]. |
| [68] |
Demain AL, Vaishnav P (2009) Production of recombinant proteins by microbes and higher organisms. Biotechnol Adv 27: 297-306. doi: 10.1016/j.biotechadv.2009.01.008
|
| [69] |
Streatfield AJ (2007) Approaches to achieve high-level heterologous protein production in plants. Plant Biotechnol J 5: 2-15. doi: 10.1111/j.1467-7652.2006.00216.x
|
| [70] | Ziegelhoffer T, Will J, Austin-Phillips S (1998) Expression of bacterial cellulase genes in transgenic alfalfa (Medicago sativa L.), potato (Solanum tuberosum L.) and tobacco (Nicotiana tabacum L.) Mol Breeding 5: 309-318. |
| [71] | Brunecki R, Selig MJ, Vinzant TB, et al. (2011) In planta expression of A. cellulolyticus Cel5A endocellulase reduces cell wall recalcitrance in tobacco and maize. Biotechnol Biofuels 4: 1. |
| [72] |
Petersen K, Bock R (2011) High-level expression of a suite of thermostable cell wall-degrading enzymes from the chloroplast genome. Plant Mol Biol 76: 311-321. doi: 10.1007/s11103-011-9742-8
|
| [73] |
Lilly M, Fierobe HP, van Zyl WH, et al. (2009) Heterologous expression of a Clostridium minicellulosome in Saccharomyces cerevisiae. FEMS Yeast Res 9: 1236-1249. doi: 10.1111/j.1567-1364.2009.00564.x
|
| [74] |
Bayer EA, Shimon LJW, Lamed R (1998) Cellulosomes: structure and ultrastructure. J Struct Biol 124: 221-234. doi: 10.1006/jsbi.1998.4065
|
| [75] |
Bayer EA, Belaich JP, Sholam Y, et al. (2004) The cellulosomes: multienzyme machines for degradation of plant cell wall polysaccharides. Ann Rev Microbiol 58: 521-554. doi: 10.1146/annurev.micro.57.030502.091022
|
| [76] |
Fontes CMGA, Gilbert HJ (2010) Cellulosomes: highly efficient nanomachines designed to deconstruct plant cell wall complex carbohydrates. Ann Rev Biochem 79: 655-681. doi: 10.1146/annurev-biochem-091208-085603
|
| [77] |
Shinoda S, Kanamasa S, Arai M (2012) Cloning of an endoglycanase gene from Paenibacillus cookii and characterization of the recombinant enzyme. Biotechnol Lett 34: 281-286. doi: 10.1007/s10529-011-0759-5
|
| [78] |
Shi R, Li Z, Ye Q, et al. (2013) Heterologous expression and characterization of a novel thermo-halotolerant endoglucanase Cel5H from Dictyoglomus thermophilum. Bioresource Technol 142: 338-344. doi: 10.1016/j.biortech.2013.05.037
|
| [79] |
Wei KSC, Teoh TC, Koshy P, et al. (2015) Cloning, expression and characterization of the endoglucanase gene from Bacillus subtilis UMC7 isolated from the gut of the indigenous termite Macrotermes malaccensis in Escherichia coli. Electronic J Biotechnol 18: 103-109. doi: 10.1016/j.ejbt.2014.12.007
|
| [80] |
Bhat MK (2000) Cellulases and related enzymes in biotechnology. Biotechnology Adv 18: 355-383. doi: 10.1016/S0734-9750(00)00041-0
|
| [81] | Wilson DB (2009) Cellulases and biofuels. Curr Opin Biotechnol20: 295-299. |
| [82] | Kuhad RC, Gupta R, Singh A. (2011) Microbial cellulases and their industrial applications. Enzyme Res 2011: 280696. |
| [83] |
McMullan G, Meehan C, Connely M, et al. (2001) Microbial decolourisation and degradation of textile dyes. Appl Microbiol Biotechnol 56: 81-87. doi: 10.1007/s002530000587
|
| [84] |
Anish R, Rahman MS, Rao M (2007) Application of cellulases from an alkalothermophilic Thermomonospora sp. in biopolishing of denims. Biotechnol Bioeng 96: 48-56. doi: 10.1002/bit.21175
|
| [85] | Ladeira SA, Cruz E, Delatorre AB, et al. (2015) Cellulase production by thermophilic Bacillus sp. SMIA-2 and its detergent compatibility. Electronic J Biotechnol 18: 110-115. |
| [86] | Yu M, Qiu Y, Chen W, et al., (2015) Action modes of recombinant endocellulase, EGA, and its domains on cotton fabrics. Biotechnol Lett [in press]. |
| [87] | Singh K (2015) Role of Enzymes in Fruit juices Clarification during Processing: A review. Int J Biol Technology 6: 114-124. |
| [88] | Cinar I (2004) Effects of cellulase and pectinase concentrations on the colour yield of enzyme extracted plant carotenoids. Process Biochem 40: 945-949. |
| [89] |
Wilkins MR, Widmer WW, Grohmann K, et al. (2007) Hydrolysis of grapefruit peel waste with cellulase and pectinase enzymes. Bioresource Technol 98: 1596-1601. doi: 10.1016/j.biortech.2006.06.022
|
| [90] | Meyer AS, Jepsen SM, Sorensen NS (1998) Enzymatic Release of Antioxidants for Human Low-Density Lipoprotein from Grape Pomace. J Agr Food Chem 46: 2399-2446. |
| [91] |
Bamforth CW (2009) Current perspectives on the role of enzymes in brewing. J Cereal Sci 50: 353-357 doi: 10.1016/j.jcs.2009.03.001
|
| [92] |
Himmel ME, Ruth MF, Wyman CE (1999) Cellulase for commodity products from cellulosic biomass. Curr Opin Biotechnol 10: 358-364. doi: 10.1016/S0958-1669(99)80065-2
|
| [93] |
Dhiman TR, Zaman MS, MacQueen IS, et al. (2002) Influence of corn processing and frequency of feeding on cow performance. J Dairy Sci 85: 217-226. doi: 10.3168/jds.S0022-0302(02)74070-8
|
| [94] | Beauchemin KA, Colombatto D, Morgavi DP, et al. (2003) Use of exogenous fibrolytic enzymes to improve feed utilization by ruminants. J Anim Sci 81: E37-E47. |
| [95] |
Wongputtissin P, Khanongnuch C, Kongbuntad W, et al. (2014) Use of Bacillus subtilis isolates from Tua-nao towards nutritional improvement of soya bean hull for monogastric feed application. Lett App Microbiol 59: 328-333. doi: 10.1111/lam.12279
|
| [96] | Shatya TA, Khan M (2014) Diversity of glycosyl hydrolase enzymes from metagenome and their application in food industry. J Food Sci 79: R2149-R 2156. |
| [97] |
Pottkamper J, Barthen P, Ilmberger N, et al. (2009) Applying metagenomics for the identification of bacterial cellulases that are stable in ionic liquids. Green Chem 11: 957-965. doi: 10.1039/b820157a
|
| [98] | Eriksson KEL (1990) Biotechnology in the pulp and paper industry.Wood Sci Technol 24: 79-101. |
| [99] |
Viesturs U, Leite M, Eisimonte M, et al. (1999) Biological deinking, technology for the recycling of office waste papers. Bioresource Technol 67: 255-265. doi: 10.1016/S0960-8524(98)00119-9
|
| [100] |
Bajpai P (1999) Application of Enzymes in the Pulp and Paper Industry. Biotechnol Prog 15: 147-157. doi: 10.1021/bp990013k
|
| [101] |
Garcia O, Torres AL, Colom JF, et al. (2002) Effect of cellulase-assisted refining on the properties of dried and never-dried eucalyptus pulp. Cellulose 9: 115-125. doi: 10.1023/A:1020191622764
|
| [102] |
Lynd LR, Laser MS, Bransby D, et al. (2008) How biotech can transform biofuels. Nat Biotechnol 26: 169-172. doi: 10.1038/nbt0208-169
|
| [103] |
Yang SJ, Kataeva I, Wiegel J, et al. (2010) Classification of ‘Anaerocellum thermophilum’ strain DSM 6725 as Caldicellulosiruptor bescii sp. nov. Int J Syst Evol Microbiol 60: 2011-2015 doi: 10.1099/ijs.0.017731-0
|
| [104] |
Kanafusa-Shinkai S, Wakayama J, Tsukamoto K, et al. (2013) Degradation of microcrystalline cellulose and non-pretreated plant biomass by a cell-free extracellular cellulase/hemicellulase system from the extreme thermophilic bacterium Caldicellulosiruptor bescii. J Biosci Bioeng 115: 64-70. doi: 10.1016/j.jbiosc.2012.07.019
|
| [105] | Scully SM, Orlygsson J (2015) Recent advances in second generation ethanol production by thermophilic bacteria. Energies 8: 1-30. |
| [106] | Assareh R, Zahiri HS, Noghabi KA, et al. (2012) Characterization of the newly isolated Geobacillus sp. T1, the efficient cellulose-producer on untreated barley and wheat straws. Bioresource Technol 120: 99-105. |
| [107] |
Gaur R, Tiwati S (2015) Isolation, production, purification and characterization of an organic-solvent-thermostable alkalophilic cellulase from Bacillus vallismortis RG-07. BMC Biotechnol 15: 19. doi: 10.1186/s12896-015-0129-9
|
| [108] | Li Y, Horsman M, Wu N, et al. (2008) Biofuels from microalgae. Biotechnol Prog 24: 815-820. |
| [109] |
Muñoz C, Hidalgo C, Zapata M, et al. (2014) Use of cellulolytic marine bacteria for enzymatic pretreatment in microalgal biogas production. App Environ Microbiol 80: 4199-4206. doi: 10.1128/AEM.00827-14
|
| [110] |
Rinaudo M (2006) Chitin and chitosan: Properties and applications. Prog Polym Sci 31: 603-632. doi: 10.1016/j.progpolymsci.2006.06.001
|
| [111] |
Pillai CKS, Paul W, Sharma CP (2009) Chitin and chitosan polymers: Chemistry, solubility and fiber formation. Prog Polym Sci 34: 641-678. doi: 10.1016/j.progpolymsci.2009.04.001
|
| [112] |
Zhang J, Xia W, Liu P, et al. (2010) Chitosan Modification and Pharmaceutical/Biomedical Applications. Mar Drugs 8: 1962-1987. doi: 10.3390/md8071962
|
| [113] |
Qin C, Zhou B, Zeng L, et al. (2004) The physicochemical properties and antitumor activity of cellulase-treated chitosan. Food Chem 84: 107-115. doi: 10.1016/S0308-8146(03)00181-X
|
| [114] |
Lin SB, Lin YC, Chen HH (2009) Low molecular weight chitosan prepared with the aid of cellulase, lysozyme and chitinase: Characterisation and antibacterial activity. Food Chem 116: 47-53. doi: 10.1016/j.foodchem.2009.02.002
|
| [115] |
Liu J, Xia W (2006) Purification and characterization of a bifunctional enzyme with chitosanase and cellulase activity from commercial cellulase. Biochem Eng J 30: 82-87. doi: 10.1016/j.bej.2006.02.005
|
| [116] |
Xie Y, Wei Y, Hu J (2010) Depolymerization of Chitosan with a Crude Cellulase Preparation from Aspergillus niger. Appl Biochem Biotechnol 160: 1074-1083. doi: 10.1007/s12010-009-8559-2
|
| [117] | Pedraza-Reyes M, Gutierrez-Corona F (1997) The bifunctional enzyme chitosanase-cellulase produced by the gram-negative microorganism Myxobacter sp. AL-1 is highly similar to Bacillus subtilis endoglucanases. Arch Microbiol 168: 321-327. |
| [118] |
Tanabe T, Morinaga K, Fukamizo T, et al. (2003) Novel Chitosanase from Streptomyces griseus HUT 6037 with Transglycosylation Activity. Biosci Biotechnol Biochem 67: 354-364. doi: 10.1271/bbb.67.354
|
| [119] | Sinha S, Tripathi P, Chand S (2012) A New Bifunctional Chitosanase Enzyme from Streptomyces sp. and Its Application in Production of Antioxidant Chitooligosaccharides. Appl Biochem Biotechnol 167: 1029-1039. |
| [120] |
Zhang Z, Wang Y, Ruan J (1998) Reclassification of Thermomonospora and Microtetraspora. J Syst Bacteriol 48: 411-422. doi: 10.1099/00207713-48-2-411
|
| [121] | Lao G, Changas GS, Jung ED, et al. (1991) DNA Sequences of Three 3-1,4-Endoglucanase Genes from Thermomonospora fusca. J Bacteriol 173: 3397-3407. |
| [122] | Ali WAA, Gondal ZI, Yammahi AAAK, et al. (2013) A case of small bowel obstruction due to phytobezoars. J Surg Case Report 7: 1. |
| [123] | Kramer SJ, Pochapin MB (2012) Gastric Phytobezoar Dissolution with Ingestion of Diet Coke and Cellulase. Gastroen Hepatol 8: 770-772. |
| [124] |
Ladas SD, Kamberoglou D, Karamanolis G, et al. (2013) Systematic review: Coca-Cola can effectively dissolve gastric phytobezoars as a first-line treatment. Aliment Pharmacol Ther 37: 169-173. doi: 10.1111/apt.12141
|
| [125] |
Pinos N, Moreno-Merino S, Congregado M (2015) Phytobezoar by aloe vera as long term complication after oesophagectomy resolved using cellulase. Int J Surg Case Report 13: 37-39. doi: 10.1016/j.ijscr.2015.05.008
|
| [126] |
Robert C, Bernalier-Donadille A (2003) The cellulolytic micro£ora of the human colon: evidence of microcrystalline cellulose-degrading bacteria in methane-excreting subjects. FEMS Microbiol Ecol 46: 81-89. doi: 10.1016/S0168-6496(03)00207-1
|
| [127] | Wedekind KJ, Mansfield HR, Montgomery L (1988) Enumeration and isolation of cellulolytic and hemicellulolytic bacteria from human feces. App Environ Microbiol 54: 1530-1535 |
| [128] | Robert C, Chassard C, Lawson PA, et al. (2007) Bacteroides cellulosilyticus sp. nov., a cellulolytic bacterium from the human gut microbial community. Int J Syst Evol Microbiol 57: 1516-1520 |
| [129] | Chassard C, Delmas E, Robert C, et al. (2012) Ruminococcus champanellensis sp. nov., a cellulose-degrading bacterium from human gut microbiota. Int J Syst Evol Microbiol 62: 138-143. |
| [130] | Martinez AJ, Visvesvara GS (1997) Free-living, amphizoic and opportunistic amebas. Brain Pathol 97: 583-98. |
| [131] |
Lakhundi S, Siddiqui R, Khan NA (2015) Cellulose degradation: a therapeutic strategy in the improved treatment of Acanthamoeba infections. Parasite Vector 8: 23. doi: 10.1186/s13071-015-0642-7
|
| [132] |
Loiselle M, Anderson KW (2003) The use of cellulase in inhibiting biofilm formation from organisms commonly found on medical implants. Biofouling 19: 77-85. doi: 10.1080/0892701021000030142
|
| [133] |
Ma L, Conover M, Lu H, et al. (2009) Assembly and development of the Pseudomonas aeruginosa biofilm matrix. PLoS Pathog 5: e1000354. doi: 10.1371/journal.ppat.1000354
|
| [134] | Rajasekharan SK, Ramesh S (2013) Cellulase inhibits Burkholderia cepacia biofilms on diverse prosthetic materials. Polish J Microbiol 62: 327-330. |
| [135] |
Huertas MG, Zarate L, Acosta IC, et al. (2014) Klebsiella pneumoniae yfiRNB operon affects biofilm formation, polysaccharide production and drug susceptibility. Microbiology 160: 2595-2606. doi: 10.1099/mic.0.081992-0
|
| [136] | Glick BR (2012) Plant growth-promoting bacteria: mechanisms and applications. Scientifica, 2012. |
| [137] | Haldar S, Sengupta S (2015) Plant-microbe cross-talk in the rhizosphere: insight and biotechnological potential. Open Microbiol J 9: 1-7. |
| [138] |
El-Tarabily KA, SykesML, Kurtböke ID, et al. (1996) Synergistic effects of a cellulose-producing Micromonospora carbonacea and an antibiotic-producing Streptomyces violascens on the suppression of Phytophthora cinnamomi root rot of Banksia grandis. Can J Botany 74: 618-624. doi: 10.1139/b96-078
|
| [139] |
Naing KW, Anees M, Nguyen XH, et al. (2014) Biocontrol of Late Blight Disease (Phytophthora capsici) of Pepper and the Plant Growth Promotion by Paenibacillus ehimensis KWN38. J Phytopathol 162: 367-376. doi: 10.1111/jph.12198
|
| [140] | Han W, He M (2010) The application of exogenous cellulase to improve soil fertility and plant growth due to acceleration of straw decomposition. Bioresource Technol 101: 3724-3731. |
| [141] | Compant S, Reiter B, Sessitsch A, et al. (2005) Endophytic colonization of Vitis vinifera l. by plant growth-promoting bacterium Burkholderia sp. strain PsJN App Environ Microbiol 71: 1685-1693. |
| 1. | Yuchao Zhang, Xiangjie Kong, Zhehui Shen, Jianxin Li, Qiuhua Yi, Guojiang Shen, Bo Dong, A survey on temporal knowledge graph embedding: Models and applications, 2024, 304, 09507051, 112454, 10.1016/j.knosys.2024.112454 | |
| 2. | Gege Li, Yong Liu, Wei Deng, Yafei Jia, Aoqi Zhang, Difan Qi, 2024, Multi Feature Deep Fusion Based on Relationship Path Enhancement for Multi-Modal Knowledge Graph Completion, 979-8-3315-0658-2, 548, 10.1109/CBASE64041.2024.10824600 |
Esther Menendez, Paula Garcia-Fraile, Raul Rivas. Biotechnological applications of bacterial cellulases[J]. AIMS Bioengineering, 2015, 2(3): 163-182. doi: 10.3934/bioeng.2015.3.163
| Algorithm 1 the training of MMCF |
| Input: triples (h, r, t) of a graph G; Output: $ {{\boldsymbol{z }}_h} $, $ {{\boldsymbol{z }}_r} $, $ {{\boldsymbol{z }}_t} $ 1: For each entity in G 2: Feature Extraction 2: Extract the visual representation $ {o'_i} $ of each object by Eq (1); 6: Extract the word representation wi; 3: Transform $ {o'_i} $ to vi by fully-connect layer; 4: Intra-modal Correlation Learning 4: Learn the intra-modal correlation between vi by Eq (2); 5: All the vi s are aggregated to obtain the image representation $ {{\boldsymbol{v }}^{_0}} \in {R^d} $; 6: Extract the word representation wi; 7: Use 1-dim CNN to encode the intra-modal correlation of wi by Eq (5); 8: Obtain text description embedding $ {{\boldsymbol{e }}^0} \in {R^d} $ by Eq (6). 9: Inter-modal Correlation Learning 9: Learn visual representation $ {v^1} \in {R^d} $ based on inter-modal correlation by Eq (7); 10: Learn textual representation $ {{\boldsymbol{e }}^1} \in {R^d} $ based on inter-modal correlation by Eq (8); 12: Cross-modal Feature Fusion 11: Obtain text representation $ {{\boldsymbol{e }}^2} $ by fusing $ {{\boldsymbol{e }}^0} $ and $ {{\boldsymbol{e }}^1} $ using Eq (9); 12: Obtain visual representation $ {{\boldsymbol{v }}^2} $ by fusing $ {{\boldsymbol{v }}^0} $ and $ {{\boldsymbol{v }}^1} $ using Eq (9); 13: Combine $ {{\boldsymbol{v }}^2} $, $ {{\boldsymbol{e }}^2} $, $ {{\boldsymbol{s }}^0} $ to obtain entity representation $ {{\boldsymbol{z }}_h} $ or $ {{\boldsymbol{z }}_t} $ by Eq (11); 14: end for 15: For each relation in G 16: Learn relation representation $ {{\boldsymbol{z }}_r} $ by Eq (12) 17: end for 18: Minimize $ \phi {{\text{(}}}h, r, t{{\text{)}}} $ by Eq (13). |
DownLoad:
CSV
| Dataset | #entities | #relation | #train | #valid | #test |
| FB-IMG | 11,757 | 1,231 | 285,850 | 29,580 | 34,863 |
| WN18RR | 40,943 | 11 | 86,835 | 3,034 | 3,134 |
| FB15k-237 | 14,541 | 237 | 272,115 | 17,535 | 20,466 |
DownLoad:
CSV
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MuRE | 0.475 | 0.436 | 0.487 | 0.554 | 0.336 | 0.245 | 0.370 | 0.521 |
| MuRP | 0.481 | 0.440 | 0.495 | 0.566 | 0.335 | 0.243 | 0.367 | 0.518 |
| InteractE | 0.463 | 0.430 | − | 0.528 | 0.354 | 0.263 | − | 0.535 |
| ConvKB | 0.249 | 0.057 | 0.417 | 0.524 | 0.243 | 0.155 | 0.371 | 0.421 |
| HypER | 0.465 | 0.436 | 0.477 | 0.522 | 0.341 | 0.252 | 0.376 | 0.520 |
| DistMult | 0.430 | 0.390 | 0.440 | 0.490 | 0.241 | 0.155 | 0.263 | 0.419 |
| M2GNN | 0.485 | 0.444 | 0.498 | 0.572 | 0.362 | 0.275 | 0.398 | 0.565 |
| ComplEx | 0.440 | 0.410 | 0.460 | 0.510 | 0.247 | 0.158 | 0.275 | 0.428 |
| ConvE | 0.430 | 0.400 | 0.440 | 0.520 | 0.325 | 0.237 | 0.356 | 0.501 |
| KG-BERT | − | − | − | 0.524 | − | − | − | 0.420 |
| MMCFMuRE | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
| MMCFInteractE | 0.483 | 0.448 | 0.495 | 0.570 | 0.367 | 0.273 | 0.402 | 0.563 |
DownLoad:
CSV
| Models | MRR | HITS@1 | HITS@3 | HITS@10 |
| TransE | − | − | − | 0.494 |
| MKRL | − | − | − | 0.645 |
| MuRE | 0.765 | 0.703 | 0.807 | 0.874 |
| InteractE | 0.813 | 0.762 | 0.849 | 0.895 |
| ConvKB | 0.449 | 0.337 | 0.513 | 0.621 |
| ComplEx | 0.525 | 0.392 | 0.618 | 0.754 |
| ConvE | 0.747 | 0.667 | 0.804 | 0.882 |
| MMCFTransE | 0.453 | 0.342 | 0.525 | 0.702 |
| MMCFMuRE | 0.820 | 0.768 | 0.852 | 0.899 |
| MMCFInteractE | 0.819 | 0.771 | 0.849 | 0.897 |
DownLoad:
CSV
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MMCF-intra | 0.471 | 0.426 | 0.479 | 0.554 | 0.342 | 0.256 | 0.384 | 0.544 |
| MMCF-inter | 0.475 | 0.429 | 0.482 | 0.556 | 0.348 | 0.258 | 0.389 | 0.546 |
| MMCF-gating | 0.480 | 0.435 | 0.491 | 0.566 | 0.353 | 0.265 | 0.396 | 0.552 |
| MMCF-text | 0.458 | 0.421 | 0.471 | 0.549 | 0.335 | 0.252 | 0.380 | 0.538 |
| MMCF-image | 0.482 | 0.441 | 0.494 | 0.562 | 0.356 | 0.267 | 0.394 | 0.549 |
| MMCF-entity | 0.467 | 0.438 | 0.485 | 0.559 | 0.339 | 0.254 | 0.387 | 0.542 |
| MMCF | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
DownLoad:
CSV
| Embedding Size | MRR | HITS@1 | HITS@3 | HITS@10 |
| 20 | 0.368 | 0.395 | 0.421 | 0.523 |
| 40 | 0.476 | 0.428 | 0.468 | 0.551 |
| 60 | 0.483 | 0.435 | 0.490 | 0.562 |
| 80 | 0.487 | 0.440 | 0.495 | 0.567 |
| 100 | 0.489 | 0.443 | 0.497 | 0.571 |
| 120 | 0.488 | 0.441 | 0.495 | 0.572 |
| 140 | 0.487 | 0.438 | 0.493 | 0.570 |
| 160 | 0.485 | 0.435 | 0.490 | 0.565 |
DownLoad:
CSV
| Top-N | MMCF | MuRE | TransE |
| 1 | stone fruit | structure | fruit tree |
| 2 | fruit | fruit tree | stone fruit |
| 3 | citrus fruit | stone fruit | veggie |
| 4 | fruit tree | seasoning | tree |
| 5 | structure | computer memory unit | nut |
| 6 | monocot genus | veggie | citrus fruit |
| 7 | veggie | citrus fruit | monocot genus |
| 8 | tree | monocot genus | structure |
| 9 | produce | tree | root |
| 10 | root | root | animal |
DownLoad:
CSV
| Algorithm 1 the training of MMCF |
| Input: triples (h, r, t) of a graph G; Output: $ {{\boldsymbol{z }}_h} $, $ {{\boldsymbol{z }}_r} $, $ {{\boldsymbol{z }}_t} $ 1: For each entity in G 2: Feature Extraction 2: Extract the visual representation $ {o'_i} $ of each object by Eq (1); 6: Extract the word representation wi; 3: Transform $ {o'_i} $ to vi by fully-connect layer; 4: Intra-modal Correlation Learning 4: Learn the intra-modal correlation between vi by Eq (2); 5: All the vi s are aggregated to obtain the image representation $ {{\boldsymbol{v }}^{_0}} \in {R^d} $; 6: Extract the word representation wi; 7: Use 1-dim CNN to encode the intra-modal correlation of wi by Eq (5); 8: Obtain text description embedding $ {{\boldsymbol{e }}^0} \in {R^d} $ by Eq (6). 9: Inter-modal Correlation Learning 9: Learn visual representation $ {v^1} \in {R^d} $ based on inter-modal correlation by Eq (7); 10: Learn textual representation $ {{\boldsymbol{e }}^1} \in {R^d} $ based on inter-modal correlation by Eq (8); 12: Cross-modal Feature Fusion 11: Obtain text representation $ {{\boldsymbol{e }}^2} $ by fusing $ {{\boldsymbol{e }}^0} $ and $ {{\boldsymbol{e }}^1} $ using Eq (9); 12: Obtain visual representation $ {{\boldsymbol{v }}^2} $ by fusing $ {{\boldsymbol{v }}^0} $ and $ {{\boldsymbol{v }}^1} $ using Eq (9); 13: Combine $ {{\boldsymbol{v }}^2} $, $ {{\boldsymbol{e }}^2} $, $ {{\boldsymbol{s }}^0} $ to obtain entity representation $ {{\boldsymbol{z }}_h} $ or $ {{\boldsymbol{z }}_t} $ by Eq (11); 14: end for 15: For each relation in G 16: Learn relation representation $ {{\boldsymbol{z }}_r} $ by Eq (12) 17: end for 18: Minimize $ \phi {{\text{(}}}h, r, t{{\text{)}}} $ by Eq (13). |
| Dataset | #entities | #relation | #train | #valid | #test |
| FB-IMG | 11,757 | 1,231 | 285,850 | 29,580 | 34,863 |
| WN18RR | 40,943 | 11 | 86,835 | 3,034 | 3,134 |
| FB15k-237 | 14,541 | 237 | 272,115 | 17,535 | 20,466 |
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MuRE | 0.475 | 0.436 | 0.487 | 0.554 | 0.336 | 0.245 | 0.370 | 0.521 |
| MuRP | 0.481 | 0.440 | 0.495 | 0.566 | 0.335 | 0.243 | 0.367 | 0.518 |
| InteractE | 0.463 | 0.430 | − | 0.528 | 0.354 | 0.263 | − | 0.535 |
| ConvKB | 0.249 | 0.057 | 0.417 | 0.524 | 0.243 | 0.155 | 0.371 | 0.421 |
| HypER | 0.465 | 0.436 | 0.477 | 0.522 | 0.341 | 0.252 | 0.376 | 0.520 |
| DistMult | 0.430 | 0.390 | 0.440 | 0.490 | 0.241 | 0.155 | 0.263 | 0.419 |
| M2GNN | 0.485 | 0.444 | 0.498 | 0.572 | 0.362 | 0.275 | 0.398 | 0.565 |
| ComplEx | 0.440 | 0.410 | 0.460 | 0.510 | 0.247 | 0.158 | 0.275 | 0.428 |
| ConvE | 0.430 | 0.400 | 0.440 | 0.520 | 0.325 | 0.237 | 0.356 | 0.501 |
| KG-BERT | − | − | − | 0.524 | − | − | − | 0.420 |
| MMCFMuRE | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
| MMCFInteractE | 0.483 | 0.448 | 0.495 | 0.570 | 0.367 | 0.273 | 0.402 | 0.563 |
| Models | MRR | HITS@1 | HITS@3 | HITS@10 |
| TransE | − | − | − | 0.494 |
| MKRL | − | − | − | 0.645 |
| MuRE | 0.765 | 0.703 | 0.807 | 0.874 |
| InteractE | 0.813 | 0.762 | 0.849 | 0.895 |
| ConvKB | 0.449 | 0.337 | 0.513 | 0.621 |
| ComplEx | 0.525 | 0.392 | 0.618 | 0.754 |
| ConvE | 0.747 | 0.667 | 0.804 | 0.882 |
| MMCFTransE | 0.453 | 0.342 | 0.525 | 0.702 |
| MMCFMuRE | 0.820 | 0.768 | 0.852 | 0.899 |
| MMCFInteractE | 0.819 | 0.771 | 0.849 | 0.897 |
| Models | WN18RR | FB15k-237 | ||||||
| MRR | HITS@1 | HITS@3 | HITS@10 | MRR | HITS@1 | HITS@3 | HITS@10 | |
| MMCF-intra | 0.471 | 0.426 | 0.479 | 0.554 | 0.342 | 0.256 | 0.384 | 0.544 |
| MMCF-inter | 0.475 | 0.429 | 0.482 | 0.556 | 0.348 | 0.258 | 0.389 | 0.546 |
| MMCF-gating | 0.480 | 0.435 | 0.491 | 0.566 | 0.353 | 0.265 | 0.396 | 0.552 |
| MMCF-text | 0.458 | 0.421 | 0.471 | 0.549 | 0.335 | 0.252 | 0.380 | 0.538 |
| MMCF-image | 0.482 | 0.441 | 0.494 | 0.562 | 0.356 | 0.267 | 0.394 | 0.549 |
| MMCF-entity | 0.467 | 0.438 | 0.485 | 0.559 | 0.339 | 0.254 | 0.387 | 0.542 |
| MMCF | 0.489 | 0.443 | 0.497 | 0.571 | 0.359 | 0.269 | 0.397 | 0.554 |
| Embedding Size | MRR | HITS@1 | HITS@3 | HITS@10 |
| 20 | 0.368 | 0.395 | 0.421 | 0.523 |
| 40 | 0.476 | 0.428 | 0.468 | 0.551 |
| 60 | 0.483 | 0.435 | 0.490 | 0.562 |
| 80 | 0.487 | 0.440 | 0.495 | 0.567 |
| 100 | 0.489 | 0.443 | 0.497 | 0.571 |
| 120 | 0.488 | 0.441 | 0.495 | 0.572 |
| 140 | 0.487 | 0.438 | 0.493 | 0.570 |
| 160 | 0.485 | 0.435 | 0.490 | 0.565 |
| Top-N | MMCF | MuRE | TransE |
| 1 | stone fruit | structure | fruit tree |
| 2 | fruit | fruit tree | stone fruit |
| 3 | citrus fruit | stone fruit | veggie |
| 4 | fruit tree | seasoning | tree |
| 5 | structure | computer memory unit | nut |
| 6 | monocot genus | veggie | citrus fruit |
| 7 | veggie | citrus fruit | monocot genus |
| 8 | tree | monocot genus | structure |
| 9 | produce | tree | root |
| 10 | root | root | animal |
