Export file:


  • RIS(for EndNote,Reference Manager,ProCite)
  • BibTex
  • Text


  • Citation Only
  • Citation and Abstract

First steps in the investigation of automated text annotation with pictures

York University Dept. of Electrical Engineering and Computer Science 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada

We describe the investigation of automatic annotation of text with pictures, where knowledge extraction uses dependency parsing. Annotation of text with pictures, a form of knowledge visualization, can assist understanding. The problem statement is, given a corpus of images and a short passage of text, extract knowledge (or concepts), and then display that knowledge in pictures along with the text to help with understanding. A proposed solution framework includes a component to extract document concepts, a component to match document concepts with picture metadata, and a component to produce an amalgamated output of text and pictures. A proof-of-concept application based on the proposed framework provides encouraging results
  Article Metrics

Keywords Natural language processing; natural language understanding; information extraction; information visualization; artificial intelligence

Citation: J. Kent Poots, Nick Cercone. First steps in the investigation of automated text annotation with pictures. Big Data and Information Analytics, 2017, 2(2): 97-106. doi: 10.3934/bdia.2017001


  • [1] B. Coyne and R. Sproat, WordsEye:An automatic text-to-scene conversion system, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 3(2003), 487-496.
  • [2] D. Genzel, K. Macherey and J. Uszkoreit, Creating a high-quality machine translation system for a low-resource language:Yiddish, (2009), Available from:www.mt-archive.info/MTS-2009-Genzel.pdf
  • [3] A. Handler, An empirical study of semantic similarity in WordNet and Word2Vec, Columbia University, (2014).
  • [4] D. Joshi, J. Z. Wang and J. Li, The Story Picturing Engine|a system for automatic text illustration, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), 2(2006), 68-89.
  • [5] J. McCarty, Programs with common sense, Defense Technical Information Center, (1963).
  • [6] J. K. Poots and E. Bagheri, Automatic annotation of text with pictures, (in-press), IEEE IT Professional, (2016).
  • [7] S. Rose, D. Engel, N. Cramer and W. Cowley, Automatic keyword extraction from individual documents, Text Mining, (2010), 1-20.
  • [8] V. Uren, P. Cimiano, J. Iria, S. Handschuh, M. Vargas-Vera, E. Motta and F. Ciravegna, Semantic annotation for knowledge management:Requirements and a survey of the state of the art, Web Semantics:science, services and agents on the World Wide Web, 4(2006), 14-28.
  • [9] N. UzZaman, J. P. Bigham and J. F. Allen, Multimodal summarization of complex sentences, Proceedings of the 16th international conference on Intelligent user interfaces, 2(2004), 43-52.
  • [10] T. Veale, A. Conway and B. Collins, The challenges of cross-modal translation:English-toSign-Language translation in the Zardoz system, Machine Translation, 13(1998), 81-106.
  • [11] L. Zhao, K. Kipper, W. Schuler, C. Vogler, N. Badle and M. Palmer, A machine translation system from English to American Sign Language, Envisioning Machine Translation in the Information Future, (2000), 54-67.


This article has been cited by

  • 1. Jezia Zakraoui, Moutaz Saleh, Jihad Al Ja’am, Text-to-picture tools, systems, and approaches: a survey, Multimedia Tools and Applications, 2019, 10.1007/s11042-019-7541-4

Reader Comments

your name: *   your email: *  

Copyright Info: 2017, Kent Poots, et al., licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution Licese (http://creativecommons.org/licenses/by/4.0)

Download full text in PDF

Export Citation

Copyright © AIMS Press All Rights Reserved