Research article Special Issues

Medical visual question answering via corresponding feature fusion combined with semantic attention

  • Medical visual question answering (Med-VQA) aims to leverage a pre-trained artificial intelligence model to answer clinical questions raised by doctors or patients regarding radiology images. However, owing to the high professional requirements in the medical field and the difficulty of annotating medical data, Med-VQA lacks sufficient large-scale, well-annotated radiology images for training. Researchers have mainly focused on improving the ability of the model's visual feature extractor to address this problem. However, there are few researches focused on the textual feature extraction, and most of them underestimated the interactions between corresponding visual and textual features. In this study, we propose a corresponding feature fusion (CFF) method to strengthen the interactions of specific features from corresponding radiology images and questions. In addition, we designed a semantic attention (SA) module for textual feature extraction. This helps the model consciously focus on the meaningful words in various questions while reducing the attention spent on insignificant information. Extensive experiments demonstrate that the proposed method can achieve competitive results in two benchmark datasets and outperform existing state-of-the-art methods on answer prediction accuracy. Experimental results also prove that our model is capable of semantic understanding during answer prediction, which has certain advantages in Med-VQA.

    Citation: Han Zhu, Xiaohai He, Meiling Wang, Mozhi Zhang, Linbo Qing. Medical visual question answering via corresponding feature fusion combined with semantic attention[J]. Mathematical Biosciences and Engineering, 2022, 19(10): 10192-10212. doi: 10.3934/mbe.2022478

    Related Papers:

    [1] Saboura Haghighi, Hamid Reza Goli . High prevalence of blaVEB, blaGES and blaPER genes in beta-lactam resistant clinical isolates of Pseudomonas aeruginosa. AIMS Microbiology, 2022, 8(2): 153-166. doi: 10.3934/microbiol.2022013
    [2] Chioma Lilian Ozoaduche, Katalin Posta, Balázs Libisch, Ferenc Olasz . Acquired antibiotic resistance of Pseudomonas spp., Escherichia coli and Acinetobacter spp. in the Western Balkans and Hungary with a One Health outlook. AIMS Microbiology, 2025, 11(2): 436-461. doi: 10.3934/microbiol.2025020
    [3] Ogueri Nwaiwu, Chiugo Claret Aduba . An in silico analysis of acquired antimicrobial resistance genes in Aeromonas plasmids. AIMS Microbiology, 2020, 6(1): 75-91. doi: 10.3934/microbiol.2020005
    [4] Alaa Fathalla, Amal Abd el-mageed . Salt tolerance enhancement Of wheat (Triticum Asativium L) genotypes by selected plant growth promoting bacteria. AIMS Microbiology, 2020, 6(3): 250-271. doi: 10.3934/microbiol.2020016
    [5] Mohammad Abu-Sini, Mohammad A. Al-Kafaween, Rania M. Al-Groom, Abu Bakar Mohd Hilmi . Comparative in vitro activity of various antibiotic against planktonic and biofilm and the gene expression profile in Pseudomonas aeruginosa. AIMS Microbiology, 2023, 9(2): 313-331. doi: 10.3934/microbiol.2023017
    [6] Kholoud Baraka, Rania Abozahra, Eman Khalaf, Mahmoud Elsayed Bennaya, Sarah M. Abdelhamid . Repurposing of paroxetine and fluoxetine for their antibacterial effects against clinical Pseudomonas aeruginosa isolates in Egypt. AIMS Microbiology, 2025, 11(1): 126-149. doi: 10.3934/microbiol.2025007
    [7] Rana Abdel Fattah Abdel Fattah, Fatma El zaharaa Youssef Fathy, Tahany Abdel Hamed Mohamed, Marwa Shabban Elsayed . Effect of chitosan nanoparticles on quorum sensing-controlled virulence factors and expression of LasI and RhlI genes among Pseudomonas aeruginosa clinical isolates. AIMS Microbiology, 2021, 7(4): 415-430. doi: 10.3934/microbiol.2021025
    [8] Taish Ramkisson, Diane Rip . Carbapenem resistance in Enterobacterales from agricultural, environmental and clinical origins: South Africa in a global context. AIMS Microbiology, 2023, 9(4): 668-691. doi: 10.3934/microbiol.2023034
    [9] Arsenio M. Fialho, Nuno Bernardes, Ananda M Chakrabarty . Exploring the anticancer potential of the bacterial protein azurin. AIMS Microbiology, 2016, 2(3): 292-303. doi: 10.3934/microbiol.2016.3.292
    [10] Amira ElBaradei, Dalia Ali Maharem, Ola Kader, Mustafa Kareem Ghareeb, Iman S. Naga . Fecal carriage of ESBL-producing Escherichia coli in Egyptian patients admitted to the Medical Research Institute hospital, Alexandria University. AIMS Microbiology, 2020, 6(4): 422-433. doi: 10.3934/microbiol.2020025
  • Medical visual question answering (Med-VQA) aims to leverage a pre-trained artificial intelligence model to answer clinical questions raised by doctors or patients regarding radiology images. However, owing to the high professional requirements in the medical field and the difficulty of annotating medical data, Med-VQA lacks sufficient large-scale, well-annotated radiology images for training. Researchers have mainly focused on improving the ability of the model's visual feature extractor to address this problem. However, there are few researches focused on the textual feature extraction, and most of them underestimated the interactions between corresponding visual and textual features. In this study, we propose a corresponding feature fusion (CFF) method to strengthen the interactions of specific features from corresponding radiology images and questions. In addition, we designed a semantic attention (SA) module for textual feature extraction. This helps the model consciously focus on the meaningful words in various questions while reducing the attention spent on insignificant information. Extensive experiments demonstrate that the proposed method can achieve competitive results in two benchmark datasets and outperform existing state-of-the-art methods on answer prediction accuracy. Experimental results also prove that our model is capable of semantic understanding during answer prediction, which has certain advantages in Med-VQA.



    Pseudomonas aeruginosa is a significant microorganism involved in urinary, bloodstream, pulmonary, soft tissue and surgical site infections [1],[2]. Antibiotic resistance is one of the most pressing problems in public health and will remain threatening to modern medicine in the coming decades. Because of the increasing abuse of antibiotics in hospitals and the community, some widely used antibiotics are losing their function, and scientists and doctors must develop new antibiotics to overcome this problem. [3],[4]. Metallo-β-lactamaseproducing (MBL) strains to hydrolyze the β-lactam ring of the drug compound, thereby inactivating them. In contrast to serine β-lactamases, MBLs use at least one but more commonly two Zn2+ ions in their active site to catalyze the hydrolysis of β-lactam rings [5],[6]. There are various methods for identifying MBL and NDM-producing strains, which fall into two phenotypic and genotypic groups. Usually, phenotypic methods have low specificity and low speed, and error results [7],[8]. Therefore, it is necessary to use molecular methods along with phenotypic methods. High-resolution melting (HRMA) analysis is one of the most sensitive and precise molecular methods based on real-time PCR [9],[10].

    HRMA is used to characterize bacterial DNA samples according to their dissociation behavior as they transition from double-strand DNA to single-strand DNA with increasing temperature and fluorescence detection [8],[11]. The HRMA is entirely precise warming of the amplicon DNA from around 50 °C up to around 95 °C. During this process, the amplicon's melting temperature is reached, and the two strands of DNA separate or “melt” apart. The concept of HRMA is to monitor this process happening with real-time PCR [12]. This rationale is achieved by using intercalating dyes. The intercalating dye binds explicitly to double-strand DNA and forms the stable fluorescent, and thus one can monitor the relative quantity of the product during DNA amplification in real-time PCR [13]. However, HRMA takes it one step further in its ability to capture much more detail. It has an increased resolving power, as melting curves from different amplicons may be differentiated based on the melt curve's shape even when the melt temperature values are the same [14],[15].

    HRMA has the potential to be a powerful tool in the clinical microbiology laboratory, providing rapid detection of genetic determinants conferring antibiotic resistance to complement current phenotypic antimicrobial susceptibility testing methods [16],[17]. These methods are labor-intensive, expensive and require unique expertise, and the results are difficult to interpret [18]. An accurate, rapid and cost-effective typing scheme is urgently needed for active surveillance and epidemiological investigations [19]. HRMA typing is a compassionate, rapid and cost-effective option for detection purposes [20],[21]. It can perform highly accurate genotyping to a hefty quantity of samples in a short amount of time [8],[22]. HRMA also is a straightforward technology that can perform both the PCR analysis and HRMA in one instrument [20]. This reduces extra expenses, saves time and creates a simpler workflow [16]. It is only necessary to create the PCR reaction volume for each sample to be analyzed, and it eliminates the need for solvents and electrophoresis gels [22].

    Therefore, we try to extend the application of the HRMA assay to this area to help detect the antibiotic resistance in different strains of NDM producing P. aeruginosa. We modified the multiplex HRMA to amplify both the bacteria NDM producing gene and MBL producing gene and analyze its melting curve.

    Subcultures of P. aeruginosa ATCC 15442, P. aeruginosa PAO-1, and P. aeruginosa ATCC 27853 were used from Hamadan Medical University, Microbiology Department microbial bank. Reference strains were cultivated at 37 °C for 24 h in cetrimide agar (Merck, Germany). All strains were kept at −20 °C in TSB containing 20% glycerol. The Ethics Committee approved this study of Hamadan University of Medical Sciences (Code No: IR.UMSHA.REC.1398.573).

    P. aeruginosa DNA extraction was performed using a DNA extraction kit (Qiagen, Germany); the steps were followed according to the kit protocol. DNA concentration was determined using a spectrophotometer (Nanodrop-200, Hangzhou Allsheng Instruments Co., Ltd., China). In this study, the sequencing method used was the dideoxy chain-terminating Sanger method [23].

    Primer sequences were initially set up as outlined in previous studies [24][27] (Table 1). For sensitivity and specificity of primers, nine-fold serial dilutions of 0.5 McFarland DNA (1.5 × 108 CFU/mL) were made (1:1–1, 1:1–2, 1:1–3, 1:1–4, 1:1–5, 1:1–6, 1:1–7, 1:1–8). Serial dilution real-time PCR tested primers' efficiencies for both target genes and reference genes. Standard curves were constructed by the Ct (y-axis) versus log DNA dilution (x-axis). The primer efficiency (E) of one cycle in the exponential phase was calculated according to the equation: E = 10−(1/slope)−1 × 100 [28].

    Briefly, 2 µL (0.5 µM) of each primer, 2 µL of DNA template, 4 µL of EvaGreen, and made up to a final volume of 20 µL using ddH2O. Real-time PCR reactions were performed on the ABI real-time machine (ABI StepOnePlus, USA). The thermal cycles were set for reverse transcription steps (55 °C for 5 min, 95 °C for 10 min, 95 °C for 20 s) followed by PCR steps: 95 °C for 15 s, 59 °C for 30 s, repeated for 40 cycles. The slope value was calculated from serial dilutions for each gene, which was then used to determine the reaction's efficiency [12].

    Table 1.  Oligonucleotide sequences used in this study.
    Target Primer Name Sequence of Primers Melting Tm (±0.5 °C) Accession Number Primer Location Product Size (bp) Ref
    NDM-1 N-1 F: GACCGCCCAGATCCTCAA
    R: CGCGACCGGCAGGTT
    89.57 MN193055.1 71–122 55 [24]
    N-2 F: TTGGCCTTGCTGTCCTTG
    R: ACACCAGTGACAATATCACCG
    76.92 MH168506.2 630–586 85 [25]
    N-3 F: GCGCAACACAGCCTGACTTT
    R: CAGCCACCAAAAGCGATGTC
    82.97 MK371546.1 298–452 155 [23]

    MBL blaSIM F: TACAAGGGATTCGGCATCG
    R: TAATGGCCTGTTCCCATGTG
    85.35 KX452682.1 1–570 570 [27]
    blaVIM F: TCTCCACGCACTTTCATGAC
    R: GTGGGAATCTCGTTCCCCTC
    84.56 NG_068039.1 332–455 124 [26]
    blaSPM F: AAAATCTGGGTACGCAAACG
    R: ACATTATCCGCTGGAACAGG
    86.62 KX452683.1 1–271 271 [27]

     | Show Table
    DownLoad: CSV

    The efficiency and the analytical sensitivity of the HRMA-PCR were evaluated by triplicate testing of 9-fold serial dilution series of each of the three reference strains. The Applied Biosystems StepOnePlus real-time PCR system was used to amplify and detect products. The reaction mix was prepared using the following components for each of the samples: 4 µL of Master Mix HRMA (HOT FIREPol EvaGreen HRMA Mix), 1 µM of each respective primer and 12 µL of DMSO (Sigma-Aldrich, USA). The following cycle parameters were used: 2 min at 50 °C, 10 min at 95 °C. Moreover, 40 cycles with denaturing for 15 s at 95 °C and by annealing/elongation for 1 min at 60 °C. Melting curves were generated after each run to confirm a single PCR product (from 60 °C to 95 °C, increasing 1 °C/3 s).

    Data analysis was performed using ABI Thermo Fisher software (release 2018, version 3.0.2) and BioEdit 7.4 software (Caredata, Inc., USA). Normalized and difference plots were generated.

    After 9-fold dilutions, a high CT was observed in the 100 CFU/mL and low CT in the 108 CFU/mL. Moreover, increasing CT values were identified as the inhibitor binding to the DNA, as such binding will reduce the amount of template available for amplification. More comprehensive CT value ranges indicated more binding; likewise, smaller ranges corresponded to less interaction between DNA and inhibitor. The CT values for these cell densities were within the 9 to 40 cycle range in the amplification process, while higher DNA concentrations appeared within 9 to 31 cycles. As seen in Figures 1 and 2, relative to a linear range of each standard curve, melting peaks can be seen for many lower DNA concentrations; however, the concentrations could not be quantified. Therefore, the actual quantitative, linear portions of the calibration curves did not extend as low as the detection limit. Melting curves displayed a single melting Tm: 89.57 °C for the N-1 gene, 76.92 °C for the N-2 gene, 82.97 °C for the N-3 gene, 84.56 °C for the blaVIM gene, 86.62 °C for the blaSIM gene, and 85.35 °C for the blaSPM gene (Figures 1 and 2). Samples containing DNA exhibited positive real-time PCR amplification, and negative controls failed to show amplification. Also, serial dilutions of positive-control DNA amplification curves showed Ct values inversely related to template DNA concentration (Figure 3).

    Figure 1.  Analytical sensitivity of real-time PCR and examples of optimization of primer pairs based on melting curve analysis for NDM primers to detect NDM producing P. aeruginosa strains. The melting curves for each primer pair were investigated: (left) N-1 gene with a melting point of 89.53 ± 0.5 °C, (middle) N-2 gene with a melting point of 76.92 ± 0.5 °C, (right) N-3 gene with a melting point 82.97 ± 0.5 °C. The mean of a: 108; b: 107; c: 106; d: 105; e: 104; f: 103; g: 102; h: 101 and i: 100 CFU/mL of DNA dilutions. Bold black horizontal lines represent the cycle threshold of real-time PCR. One peak with a shoulder corresponds to genomic DNA amplification; no peak corresponds to no amplification. EvaGreen color and single-tube reactions were used in this test. Also, real-time PCR was performed as a single step.
    Figure 2.  Analytical sensitivity of real-time PCR and examples of optimization of primer pairs based on melting curve analysis for MBL primers to detect NDM producing P. aeruginosa strains. The melting curves for each primer pair were investigated: (left) blaVIM gene with a melting point of 84.56 ± 0.5 °C, (middle) blaSPM gene with a melting point of 85.35 ± 0.5 °C, (right) blaSIM gene a melting point of 86.62 ± 0.5 °C. The mean of a: 108; b: 107; c: 106; d: 105; e: 104; f: 103; g: 102; h: 101 and i: 100 CFU/mL of DNA dilutions. Bold black horizontal lines represent the cycle threshold of real-time PCR. One peak with a shoulder corresponds to genomic DNA amplification; no peak corresponds to no amplification. EvaGreen color and single-tube reactions were used in this test. Also, real-time PCR was performed as a single step.
    Figure 3.  Melting curve analysis and analytical specificity of real-time PCR for NDM primers (A) and MBL primers (B) used to detect NDM producing P. aeruginosa strains: (a) blank tube, (b) P. aeruginosa PAO-1 and (c) P. aeruginosa ATCC 27853. One peak with a shoulder corresponds to genomic DNA amplification; no peak corresponds to no amplification. EvaGreen color and single-tube reactions were used in this test. Also, real-time PCR was performed as a single step. A 0.5-McFarland concentration (1.5 x 108 CFU/mL of DNA) was used to determine primer specificity.

    Reaction efficiencies were found to be within the range of 3 to 3.5 when calculated from the standard curves using the ABI Thermo Fisher analysis software (Version 2.3.2) with a formula of E = 10(−1/slope)−1. For the N-1, N-2 and blaVIM primer set, the reaction efficiency reached a value slightly greater than 3, at 3.2, which would suggest the efficiency of 101%. Efficiencies greater than 100% can be obtained. All the investigated dilutions showed low efficiencies: N-3, E = 98.8%; blaSMP, E = 95.588%; blaSIM, E = 96.493 (Figures 1 and 2).

    For the N-1 and blaVIM primer set, the linear range was determined to extend as low as 100 CFU/mL, N-2 was 103 CFU/mL, N-3 was 104 CFU/mL, blaSMP was 102 CFU/mL, and blaSIM was 105 CFU/mL, as indicated by the lowest DNA concentration value on each of the standard curves. Points that caused the curves to deviate from linearity (mostly those with lower concentrations) were excluded (Figures 1 and 2).

    Fluorescence data were analyzed using the tools for HRMA incorporated in the ABI Thermo Fisher analysis software. HRMA PCR amplification curves of samples analyzed for the presence of NDM producing P. aeruginosa are shown in Figures 3, 4, and 5. Difference plots of normalized data show the differences in fluorescence between each sample of DNA. Derivative plots display the rate of fluorescence change; the peak indicates the melting temperature of a sample. All plots displayed a single melting domain, typically between 87.07 °C–87.57 °C for the N-1 gene, 76.42 °C–76.92 °C for the N-2 gene, 82.47 °C–82.97 °C for the N-3 gene, 84.06 °C–84.56 °C for the blaVIM gene, 86.12 °C–86.62 °C for the blaSIM gene and 85.30 °C–85.80 °C for the blaSPM gene, following different product sizes.

    The results of this representative experiment show that all samples containing P. aeruginosa DNA had measurable amplification, as detected by exponential fluorescence (Figure 3), and all the DNA dilutions of NDM producing P. aeruginosa were identified (dilution of 108 to 100 CFU/mL). The N-1 and blaVIM genes in NDM producing P. aeruginosa were detected in all dilutions of DNA. Moreover, N-2, N-3, blaSPM and blaSIM primers can be able to detect bacterial DNA in dilutions of 103 CFU/mL, 104 CFU/mL, 102 CFU/mL and 105 CFU/mL, respectively (Figures 4 and 5).

    Figure 4.  HRMA graphs corresponding to one high-resolution melting analysis of a subset of NDM producing P. aeruginosa strains by N-1 (A), N-2 (B) and N-3 (C) genes. DNA samples from all the dilutions involved in this study were prepared and amplified successfully using the EvaGreen dye-based method in the ABI instrument. Primers' specific melting peaks (Tm) were obtained via HRMA, allowing the differentiation of all investigated β-lactamase enzymes. Due to the positively saturating EvaGreen dye and the HRMA, the resolution accuracy was ±0.1–0.5 °C.
    Figure 5.  HRMA graphs corresponding to one high-resolution melting analysis of a subset of NDM producing P. aeruginosa strain by blaVIM (A), blaSIM (B) and blaSPM (C) genes. DNA samples from all the dilutions involved in this study were prepared and amplified successfully using the EvaGreen dye-based method in the ABI instrument. Primers' specific melting peaks (Tm) were obtained via HRMA, allowing the differentiation of all investigated β-lactamase enzymes. Due to the positively saturating EvaGreen dye and the HRMA, the resolution accuracy was ±0.1 °C–0.5 °C.

    The software automatically analyzed the raw melting curve data and set the starting (pre-melt) and ending (post-melt) fluorescence signals of all data to actual values to aid interpretation and analysis (Figures 4 and 5). The cursors for these two points have defaulted to the ends of the curve. However, these regions were manually adjusted to encompass a representative baseline for the pre-melt and post-melt phases. Widening the normalization regions into the melt phase was avoided to ensure that curves normalize effectively. Moreover, we performed a melt curve analysis of HRMA PCR samples to assess the amplicon's specificity. The results of the HRMA showed a very similar melt peak for all serial dilutions of P. aeruginosa.

    Based on the current study, the efficiency of genes was at least 99.99%, the r2 was >0.99.99, and melt curves yielded single peaks. Interestingly, the Ct vs. DNA relationship's slope varied little across the nine-fold dilutions tested, ranging from −3.589 to −3.955. According to previous studies, this can be justified because short fragments bind less fluorescent and are compensated by a higher primer concentration [29],[30]. However, sometimes the peak height of the short amplicon increases in a different replicate. This problem gets worse in the MCA, such that sometimes we lost the long amplicon even at the primer ratio of 1:1. Furthermore, these results agree with Mentasti et al. [5].

    Moreover, three blaNDM-1 primers with different amplicon length and MBL primers were used to detect NDM producing P. aeruginosa strains. These results indicated that the primers' specificity, with a 0.5 °C error range, could detect NDM producing P. aeruginosa. Andini et al. showed that an accurate analysis of the melting curve could play a significant role in the diagnosis [31]. Ashrafi et al. found that, to obtain the best performance in sophisticated methods such as HRMA, the DNA's melting temperature must be monitored in various dilutions to obtain accurate sensitivity and specificity [21]. Tahmasebi et al. also confirmed that efficiency is probably due to the shorter length of primers' products, which enabled better amplification in PCR [9].

    Based on the current study, Figures 1 and 2 showed that the highest analytical sensitivity and specificity were reported for the detection of antibiotic-resistant strains, as indicated by the lowest DNA concentration value on each of the standard curves. Smiljanic et al. also illustrated that identifying Gram-negative NDM and MBL producing strains is difficult because the resistance to carbapenems in these bacteria is encoded by similar sequences [32]. Thus, using a sensitive and precise method such as HRMA and specific primers could identify strains such as MBL and NDM producing strains. In a study, Ding et al. proposed the PASGNDM699 strain resistance to a wide range of antibiotics. They also confirmed the clinical importance of PASGNDM strains in causing resistant infections [33].

    Based on Figure 4 and Figure 5, DNA dilutions of NDM producing P. aeruginosa strains were identified (dilution of 108 to 100 CFU/mL). The results were different from those obtained by Naas et al. and Smiljanic et al. [32],[34]. Identification of MBL and NDM producing strains has been performed in various studies in Sweden [17], USA [35], Australia [25] and Italy [36] in Gram-negative bacteria by the HRMA method.

    Nevertheless, one of the most important benefits of the multiplex HRMA method is the simultaneous identification of different NDM varieties. Identification of these variants using phenotypic methods has low accuracy and speed. Those methods also require spending much time optimizing. Makena et al. found that the identification of NDM variants by phenotypic methods requires protein stability and is not practical due to the evolution of NDM-producing strains [37].

    According to our results, primers with a short length had the best sensitivity and specificity in the HRMA assay. Słomka et al. demonstrated that when the DNA quality is low, DNA degradation or long DNA breaks during extraction makes the long template harder to amplify [20]. Though, the capacity to monitor PCRs in real-time has revolutionized how PCR is used in the clinical microbiology field. HRMA assay is used to amplify and concurrently quantify a targeted DNA molecule and enables both detection and quantification of DNA. HRMA PCR needs a fluorescent reporter that binds to the finished product and reports its presence by fluorescence. The Eischeid study confirms these results [38].

    In this study, we optimized the HRMA method to identify antibiotic resistance gene variants. We did not use designed primers in this study. The design and optimization of C + G values ​​ in designed primers provide the sensitivity and specificity of HRMA in the simultaneous identification of drug resistance [39]. Thus, the length of the selected primers should be considered to identify bacterial sub-strains. Another limitation of our study was the lack of use of different fluorescent dyes in identifying subtypes. Based on previous studies, the type of dye used significantly affects the sensitivity and specificity of HRMA [40]. Therefore, in different master mixes from different suppliers, calibration is necessary to establish the new Tm data on the reference and clinical strains.

    We demonstrated that the HRMA assay is a rapid and sensitive pre-sequence screening tool that allows the detection of low DNA concentration. Further, our study results showed that NDM and MBL genes' co-existence could be detected using the HRMA method reference strains. Moreover, the HRMA method for identifying NDM and MBL producing strains has high sensitivity and specificity. The present study also confirmed that primer product length and fluorescent dye play critical roles in increasing the sensitivity and specificity of the HRMA assay. However, the selection of the melting temperature range is essential to the analysis, as there needs to be sufficient data both before and following the melting transition to allow reliable normalization of the melting curves.



    [1] Z. Chen, X. Guo, P. Y. M. Woo, Y. Yuan, Super-resolution enhanced medical image diagnosis with sample affinity interaction, IEEE Trans. Med. Imaging, 40 (2021), 1377-1389. https://doi.org/10.1016/j.media.2020.101839 doi: 10.1016/j.media.2020.101839
    [2] W. A. Al, I. D. Yun, Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images, IEEE Trans. Med. Imaging, 39 (2019), 1245-1255. https://doi.org/10.1109/TMI.2019.2946345 doi: 10.1109/TMI.2019.2946345
    [3] A. Jungo, R. Meier, E. Ermis, M. Blatti-Moreno, E. Herrmann, R. Wiest, et al., On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention, (2018), 682-690. https://doi.org/10.1007/978-3-030-00928-1_77
    [4] Y. Tang, Y. Tang, Y. Zhu, J. Xiao, R. M. Summers, A disentangled generative model for disease decomposition in chest x-rays via normal image synthesis, Med. Image Anal., 67 (2021), 101839. https://doi.org/10.1016/j.media.2020.101839 doi: 10.1016/j.media.2020.101839
    [5] H. Abdeltawab, F. Khalifa, F. Taher, N. S. Alghamdi, M. Ghazal, G. Beache, et al., A deep learning-based approach for automatic segmentation and quantification of the left ventricle from cardiac cine MR images, Comput. Med. Imaging Graphics, 81 (2020), 101717. https://doi.org/10.1016/j.compmedimag.2020.101717 doi: 10.1016/j.compmedimag.2020.101717
    [6] J. Ker, L. Wang, J. Rao, T. Lim, Deep learning applications in medical image analysis, IEEE Access, 6 (2017), 9375-9389. https://doi.org/10.1109/ACCESS.2017.2788044 doi: 10.1109/ACCESS.2017.2788044
    [7] X. Xie, J. Niu, X. Liu, Z. Chen, S. Tang, S. Yu, A survey on incorporating domain knowledge into deep learning for medical image analysis, Med. Image Anal., 69 (2021), 101985. https://doi.org/10.1016/j.media.2021.101985 doi: 10.1016/j.media.2021.101985
    [8] C. Li, G. Zhu, X. Wu, Y. Wang, False-positive reduction on lung nodules detection in chest radiographs by ensemble of convolutional neural networks, IEEE Access, 6 (2018), 16060-16067. https://doi.org/10.1109/ACCESS.2018.2817023 doi: 10.1109/ACCESS.2018.2817023
    [9] D. Bardou, K. Zhang, S. M. Ahmad, Classification of breast cancer based on histology images using convolutional neural networks, IEEE Access, 6 (2018), 24680-24693. https://doi.org/10.1109/ACCESS.2018.2831280 doi: 10.1109/ACCESS.2018.2831280
    [10] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, et al., Vqa: Visual question answering, in IEEE International Conference on Computer Vision, (2015), 2425-2433. https://doi.org/10.1109/ICCV.2015.279
    [11] P. Gao, H. You, Z. Zhang, X. Wang, H. Li, Multi-modality latent interaction network for visual question answering, in IEEE/CVF International Conference on Computer Vision, (2019), 5825-5835. https://doi.org/10.1109/ICCV.2019.00592
    [12] Z. Yu, J. Yu, Y. Cui, D. Tao, Q. Tian, Deep modular co-attention networks for visual question answering, in IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 6274-6283. https://doi.org/10.1109/CVPR.2019.00644
    [13] M. Malinowski, M. Fritz, A multi-world approach to question answering about real-world scenes based on uncertain input, Adv. Neural Inf. Proces. Syst., 2014 (2014), 1682-1690.
    [14] M. Ren, R. Kiros, R. Zemel, Exploring models and data for image question answering, Adv. Neural Inf. Proces. Syst., 2015 (2015), 2953-2961.
    [15] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, et al., Visual genome: Connecting language and vision using crowdsourced dense image annotations, Int. J. Comput. Vision, 123 (2017), 32-73. https://doi.org/10.1007/s11263-016-0981-7 doi: 10.1007/s11263-016-0981-7
    [16] Y. Zhu, O. Groth, M. Bernstein, F. Li, Visual7w: Grounded question answering in images, in IEEE Conference on Computer Vision and Pattern Recognition, (2016), 4995-5004. https://doi.org/10.1109/CVPR.2016.540
    [17] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, D. Parikh, Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering, in IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6904-6913. https://doi.org/10.1007/s11263-018-1116-0
    [18] B. Ionescu, H. Müller, R. Péteri, A. B. Abacha, M. Sarrouti, D. Demner-Fushman et al., Overview of the ImageCLEF 2021: Multimedia retrieval in medical, nature, internet and social media applications, in International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, Cham, (2021), 345-370. https://doi.org/10.1007/978-3-030-85251-1_23
    [19] J. J. Lau, S. Gayen, A. B. Abacha, D. Demner-Fushman, A dataset of clinically generated visual questions and answers about radiology images, Sci. Data, 5 (2018), 180251. https://doi.org/10.1038/sdata.2018.251 doi: 10.1038/sdata.2018.251
    [20] B. Liu, L. M. Zhan, L. Xu, L. Ma, Y. Yang, X. Wu, SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering, in IEEE International Symposium on Biomedical Imaging, (2021), 1650-1654. https://doi.org/10.1109/ISBI48211.2021.9434010
    [21] A. B. Abacha, S. Gayen, J. J. Lau, S. Rajaraman, D. Demner-Fushman, NLM at ImageCLEF 2018 visual question answering in the medical domain, in Working Notes of CLEF, (2018).
    [22] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. https://doi.org/10.1109/CVPR.2016.90
    [23] I. Allaouzi, M. B. Ahmed, B. Benamrou, An encoder-decoder model for visual question answering in the medical domain, in Working Notes of CLEF, (2019).
    [24] B. Liu, L. Zhan, X. Wu, Contrastive pre-training and representation distillation for medical visual question answering based on radiology images, in International Conference on Medical Image Computing and Computer-Assisted Intervention, (2021), 210-220. https://doi.org/10.1007/978-3-030-87196-3_20
    [25] H. Gong, G. Chen, S. Liu, Y. Yu, G. Li, Cross-modal self-attention with multi-task pre-training for medical visual question answering, in International Conference on Multimedia, (2021), 21-24. https://doi.org/10.1145/3460426.3463584
    [26] S. Liu, X. Zhang, X. Zhou, J. Yang, BPI-MVQA: a bi-branch model for medical visual question answering, BMC Med. Imaging, 22 (2022), 79. https://doi.org/10.1186/s12880-022-00800-x doi: 10.1186/s12880-022-00800-x
    [27] U. Naseem, M. Khushi, J. Kim, Vision-language transformer for interpretable pathology visual question answering, IEEE J. Biomed. Health Inf., (2022), forthcoming 2022. https://doi.org/10.1109/JBHI.2022.3163751
    [28] J. Li, S. Liu, Lijie at imageclefmed vqa-med 2021: Attention model based on efficient interaction between multimodality, in Working Notes of CLEF, (2021), 1275-1284.
    [29] Q. Xiao, X. Zhou, Y. Xiao, K. Zhao, Yunnan university at vqa-med 2021: Pretrained biobert for medical domain visual question answering, in Working Notes of CLEF, (2021), 1405-1411.
    [30] N. M. S. Sitara, K. Srinivasan, SSN MLRG at VQA-MED 2021: An approach for VQA to solve abnormality related queries using improved datasets, in Working Notes of CLEF, (2021), 1329-1335.
    [31] H. Gong, R. Huang, G. Chen, G. Li, et al., Sysu-hcp at vqa-med 2021: A data-centric model with efficient training methodology for medical visual question answering, in CEUR Workshop Proceedings, (2021), 1613.
    [32] Y. Li, Z. Yang, T. Hao, Tam at vqa-med 2021: A hybrid model with feature extraction and fusion for medical visual question answering, in Working Notes of CLEF, (2021), 1295-1304.
    [33] A. Al-Sadi, H. A. Al-Theiabat, M. Al-Ayyoub, The inception team at VQA-Med 2020: Pretrained VGG with data augmentation for medical VQA and VQG, in Working Notes of CLEF, (2020).
    [34] K. Gasmi, Hybrid deep learning model for answering visual medical questions, Supercomput., 2022 (2022), 1-18. https://doi.org/10.1007/s11227-022-04474-8 doi: 10.1007/s11227-022-04474-8
    [35] Z. Liao, Q. Wu, C. Shen, A. Van Den Hengel, J. Verjans, AIML at VQA-Med 2020: Knowledge inference via a skeleton-based sentence mapping approach for medical domain visual question answering, in Working Notes of CLEF, (2020).
    [36] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput., 9 (1997), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735 doi: 10.1162/neco.1997.9.8.1735
    [37] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, preprint, arXiv: 1406.1078.
    [38] J. Devlin, M. V. Chang, K. Lee, K. B. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2019), 4171-4186. https://doi.org/10.18653/v1/N19-1423
    [39] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. So, et al., BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36 (2020), 1234-1240. https://doi.org/10.1093/bioinformatics/btz682 doi: 10.1093/bioinformatics/btz682
    [40] Z. Yang, X. He, J. Gao, L. Deng, A. Smola, Stacked attention networks for image question answering, in IEEE conference on computer vision and pattern recognition, (2016), 21-29. https://doi.org/10.1109/CVPR.2016.10
    [41] J. H. Kim, J. Jun, B. T. Zhang, Bilinear attention networks, Adv. Neural Inf. Process. Syst., 31 (2018), 1571-1581.
    [42] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, M. Rohrbach, Multimodal compact bilinear pooling for visual question answering and visual grounding, preprint, arXiv: 1606.01847.
    [43] B. D. Nguyen, T. T. Do, B. X. Nguyen, T. Do, E. Tjiputra, Q. D. Tran, Overcoming data limitation in medical visual question answering, in Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, (2019), 522-530. https://doi.org/10.1007/978-3-030-32251-9_57
    [44] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in Proceedings of the 34th International Conference on Machine Learning, (2017), 1126-1135.
    [45] J. Masci, U. Meier, D. Cireşan, J. Schmidhuber, Stacked convolutional auto-encoders for hierarchical feature extraction, in International conference on artificial neural networks, (2011), 52-59. https://doi.org/10.1007/978-3-642-21735-7_7
    [46] L. Zhan, B. Liu, L. Fan, J. Chen, X. Wu, Medical visual question answering via conditional reasoning, in The 28th ACM International Conference on Multimedia, (2020), 2345-2354. https://doi.org/10.1145/3394171.3413761
    [47] Y. Khare, V. Bagal, M. Mathew, A. Devi, U. D. Priyakumar, C. V. Jawahar, MMBERT: Multimodal BERT pretraining for improved medical VQA, in IEEE 18th International Symposium on Biomedical Imaging, (2021), 1033-1036. https://doi.org/10.1109/ISBI48211.2021.9434063
    [48] T. Do, B. X. Nguyen, E. Tjiputra, M. Tran, Q. D. Tran, A. Nguyen, Multiple meta-model quantifying for medical visual question answering, in Medical Image Computing and Computer Assisted Intervention, (2021), 64-74. https://doi.org/10.1007/978-3-030-87240-3_7
    [49] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, et al., Don't stop pretraining: Adapt language models to domains and tasks, preprint, arXiv: 2004.10964.
    [50] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, et al., Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, in Proceedings of the AAAI Conference on Artificial Intelligence, (2019), 590-597. https://doi.org/10.1609/aaai.v33i01.3301590
    [51] J. Cheng, Brain tumor dataset, Figshare Datasets, (2017). https://doi.org/10.6084/m9.figshare.1512427.v5
    [52] Y. Zhang, Q. Chen, Z. Yang, H. Lin, Z. Lu, BioWordVec, improving biomedical word embeddings with subword information and MeSH, Sci. Data, 6 (2019), 52. https://doi.org/10.1038/s41597-019-0055-0
    [53] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in IEEE Conference on Computer Vision and Pattern Recognition, (2018), 7132-7141. https://doi.org/10.1109/CVPR.2018.00745
    [54] X. Wang, S. Zhao, B. Cheng, Y. Yin, H. Yang, Explore modeling relation information and direction information in KBQA, Neurocomputing, 471 (2022), 139-148. https://doi.org/10.1016/j.neucom.2021.10.094 doi: 10.1016/j.neucom.2021.10.094
    [55] M. Gao, J. Lu, F. Chen, Medical knowledge graph completion based on word embeddings, Information, 13 (2022), 205. https://doi.org/10.3390/info13040205 doi: 10.3390/info13040205
    [56] L. Liu, M. Wang, X. He, L. Qing, H. Chen, Fact-based visual question answering via dual-process system, Knowl. Based Syst., 237 (2022), 107650. https://doi.org/10.1016/j.knosys.2021.107650 doi: 10.1016/j.knosys.2021.107650
  • This article has been cited by:

    1. Xiaoyu Xu, Nan Wang, Liguo Feng, Jianwen Wang, Simple Sequence Repeat Fingerprint Identification of Essential-Oil-Bearing Rosa rugosa via High-Resolution Melting (HRM) Analysis, 2023, 13, 2218-273X, 1468, 10.3390/biom13101468
  • Reader Comments
  • © 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Metrics

Article views(3316) PDF downloads(146) Cited by(6)

Figures and Tables

Figures(9)  /  Tables(4)

Other Articles By Authors

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog