Artificial intelligence for video game visualization, advancements, benefits and challenges

1.   Introduction

Artificial intelligence (AI) has emerged as a pivotal driving force propelling the evolution of the gaming industry, empowering developers to craft increasingly realistic and immersive gaming experiences ^[1,2,3]. Over the years, AI has proven to be instrumental in revolutionizing game visualization ^[4], facilitating the generation of intricate and lifelike characters, environments and interactions ^[5]. Consequently, this has led to an enhanced visual experience and more engaging gameplay captivating a wider audience and fueling the expansion of the gaming market.

AI has made remarkable strides in various aspects of gaming, notably in the domain of real-time strategy games ^[6,7,8,9]. In this context, AI is utilized to govern NPC (non-player character) units and make strategic decisions thereby enhancing the complexity and challenge of gameplay. Furthermore, by simulating elements such as weather patterns and wildlife behavior AI techniques generate more immersive and realistic game environments^[10,11,12]. Another crucial application of AI in gaming is the adaptation to player behavior which enables a more personalized and engaging experience ^[13,14]. For example, an AI-controlled opponent in a fighting game could analyze the player's tactics and modify its strategies accordingly resulting in more challenging and dynamic gameplay. By embracing the diverse capabilities of AI, game developers can create more sophisticated, interactive and captivating experiences for players further advancing the gaming industry.

Game visualization refers to the use of artificial intelligence techniques and technologies to enhance the visual aspects of games including graphics, animations and virtual environments ^[15,16]. In general, it encompasses the employment of computer graphics and AI techniques to render and exhibit the visual components of a video game including environments, characters, objects and effects such as lighting and shading. In recent years, significant advancements have been made in leveraging AI for game visualization ^{[17,18,19,20,21,22]} resulting in considerable enhancements in realism, immersion and overall quality of video games ^[23,24]. The integration of AI in game visualization holds immense potential for augmenting the overall player experience and empowering developers to generate more realistic and dynamic game worlds ^[25,26]. Nonetheless, the implementation of AI also presents a range of challenges and ethical concerns such as ensuring fairness and transparency in AI-driven systems. As the gaming industry continues to evolve, addressing these issues becomes increasingly crucial to foster responsible and inclusive AI adoption, ultimately maximizing the benefits of AI-enhanced game visualization while minimizing potential drawbacks.

In this review paper, we delve into the diverse applications of AI in game visualization as well as the benefits and challenges associated with its implementation. The structure of this review is organized as follows: We begin by presenting a concise overview of the historical development and evolution of AI in the gaming industry. Subsequently, we introduce prevalent game AI techniques with a particular focus on those employed in game visualization (Figure 1). Moving forward, we discuss the benefits and challenges of incorporating AI in game visualization emphasizing its potential impact on player experience and the ethical considerations at play. Subsequently, an extensive analysis of existing literature on AI in game visualization is conducted encompassing a thorough assessment of research methodologies employed and the corresponding findings. Ultimately, we synthesize the key points of the review and delineate areas for future research providing a comprehensive understanding of the current state of AI in game visualization and offering insights into potential future developments.

Overall, this review paper aims to offer an extensive overview of the current state of AI in game visualization as well as to pinpoint trends and opportunities for future research within this domain. In our investigation, we address the following critical questions: What are the principal advancements in AI for game visualization? How does AI contribute to enriching the player experience in games? What challenges and ethical considerations arise from the implementation of AI in game visualization? By exploring these questions, we seek to provide valuable insights and guidance for researchers and industry professionals working at the intersection of AI and game visualization paving the way for further innovation in this rapidly evolving field.

2.   Related work

In recent years, the gaming industry has witnessed significant advancements ^[27], especially video games which is largely attributable to the extensive adoption of AI techniques ^[2,23,28]. These techniques encompass a range of approaches such as machine learning (ML) ^[29,30,31], tree search algorithms ^[32,33] and others. To offer valuable insights for researchers and practitioners working on similar projects this section will survey the literature on AI and video game visualization from the past decades highlighting key developments and trends in the field.

In the 1990s, early AI techniques such as rule-based systems and simple decision trees were utilized for basic game behaviors, character movement and basic graphics rendering. Notably, the advent of finite state machines achieves basic character behaviors and decision-making logic. These mechanisms allow game characters to respond to player actions, enhancing the visual interactivity and dynamism of the game. Moreover, the application of pathfinding algorithms enables game characters to intelligently navigate around obstacles, traverse complex terrains and move more naturally, enhancing the visual fluidity and realism of the game ^[1].

The 2000s–2010s witnessed the emergence of more sophisticated AI algorithms including neural networks and genetic algorithms. Noteworthy contributions include the application of neural networks for character animation and the use of genetic algorithms for procedural content generation ^[34]. In ^[35], Smith and Mateas introduced an approach to content generation in games that focused on explicitly defining the design space. By utilizing domain-independent procedures and representing the design space as an answer set program, designers could efficiently create generators for different game content domains. Using genetic algorithms as a search-based approach, researchers investigated the automated generation of platform videogame levels to achieve enhanced expressivity and diversity without the need for human creation ^[36]. By employing simple heuristics, the system successfully generated playable levels within a short timeframe while maintaining considerable variation as evidenced by our findings. In order to eliminate the need for manual parameterization, Frade et al. ^[37] proposed a genetic programming-based procedural content generation technique for creating parametrization-free procedural terrains in video games. Moreover, the generated terrains exhibit aesthetic appeal and incorporate a novel evaluation process based on terrain accessibility and obstacle edge length metrics.

The advancements in AI technology during the 2010s–2020s, particularly in deep learning with CNNs and RNNs, have had a profound impact on game visualization ^[38,39]. These breakthroughs have revolutionized image-based rendering, natural language generation and texture/scene synthesis through the introduction of GANs significantly enhancing the visual aspects of games. Beheiry et al. ^[40] demonstrated that VR combines total immersion, stereoscopic vision and motion capture to create artificial environments. It enhances perception of volumetric details and allows users to actively engage with 3D data creating a spatially realistic experience. While widely used in entertainment, VR's potential in scientific visualization is still being explored with emerging platforms expected to drive wider adoption. Another study ^[41] proposed deep reinforcement learning (DRL) which has made significant progress enabling agents to learn from high-dimensional inputs and achieve super-human performance in various 3D video games. In ^[42], researchers provide an overview of popular algorithms and techniques in shadow maps generation. They classified and systemized these techniques in order to enhance the realism of virtuality.

3.   Methodology

By employing preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines, we conducted a systematic review of recently published literature on virtual reality applications for intelligent manufacturing. The inclusion criteria are: (i) publications indexed in the Web of Science, Scopus, and ProQuest databases; (ii) publication dates between 1987 and 2022; (iii) written in English; (iv) being a review paper or an innovative empirical study; and (v) coverage of specific search terms. We only excluded editorial materials. We utilized the systematic review data repository (SRDR), a software program for the collection, processing and inspection of data for our systematic review. The quality of the identified scholarly sources was assessed using the mixed method appraisal tool. After extracting and analyzing publicly accessible papers as evidence, no institutional ethics approval was required prior to commencing our research (Figure 2).

Between April 1987 and October 2022 (primarily in 2021 and 2022), we conducted a systematic literature review of the Web of Science, ProQuest, and Scopus databases using search terms such as "Artificial intelligence for game", "AI techniques for game visualization", "AI for 2D and 3D game visualization", "AI techniques for game visualization", "Applications of AI techniques to game visualization", "Machine learning" and "Computer vision". We determined these search keywords based on their frequency in the relevant literature. After reading article titles and abstracts, only 235 publications met the qualifying requirements. We selected 100 primarily empirical sources by excluding those with ambiguous or controversial findings (insufficient/irrelevant data), outcomes not substantiated by replication, overly broad content or nearly identical titles (Figures 3 and 4).

4.   Overview of artificial intelligence for video game

Artificial intelligence (AI) is an expansive field encompassing numerous techniques and approaches aimed at constructing systems capable of exhibiting intelligent behavior ^[43,44,45]. The implementation of AI in video games can enhance the immersiveness and dynamism of gameplay experiences by offering players intelligent adversaries and allies that can react to their actions in significant ways ^[46,47]. This section will present an overview of artificial intelligence in the context of video games.

4.1.   Current state-of-the-art of artificial intelligence for video games

The visualization process of a video game is a multi-step procedure aimed at creating and presenting the visual elements of the game. The main steps involved in the visualization process of a video game, in general, include concept design, art design, 3D modeling and animation, rendering and lighting, special effects and post-processing and user interface design ^[48]. Hence, we will explore some examples of the latest research trends from the above aspects.

Artificial intelligence for concept design: At the beginning of the visualization process, the game development team collaborates with designers to establish the overall visual concept and style of the video game. This includes determining the art style, theme, color scheme and overall visual experience of the game. AI techniques such as generative algorithms can assist game developers and designers in generating visual concepts and styles based on predefined parameters or existing artwork ^[49]. This can include generating art styles, color schemes and visual themes that align with the desired aesthetic of the game.

Artificial intelligence for art design: During this stage, artists and designers are responsible for creating the visual elements of the game such as character models, environments, props, special effects and more. They use professional design tools and software to sketch concepts, perform detailed design work and create texture assets ^[50]. AI can aid artists and designers in creating visual elements by automating certain tasks. For example, AI-based tools can assist in generating or enhancing character models, environments, props, special effects and textures thereby accelerating the art creation process.

Artificial intelligence for 3D modeling and animation: Using 3D modeling software, artists transform the game's characters, environments and props into 3D models. These models can undergo further detailing, texture mapping and animation based on the design requirements ^[51]. Correspondingly, AI-driven algorithms can enhance the 3D modeling and animation process. For instance, machine learning techniques can be employed to automate or optimize aspects of character rigging, motion capture or physics-based simulations leading to more realistic and efficient character and object animations ^[52,53,54].

Artificial intelligence for rendering and lighting: In this stage, programmers and artists collaborate in rendering the 3D models and environments into the final images using rendering engines and lighting techniques. This involves determining light sources, adding materials, setting up shadows and lighting effects ^[55]. By introducing AI techniques, rendering and lighting processes can be improved by enhancing the quality and realism of visual effects. This includes real-time rendering optimization ^[56], global illumination algorithms ^[57] and AI-powered denoising techniques ^[58] which contribute to achieving more immersive and visually appealing scenes.

Artificial intelligence for special effects and post-processing: By incorporating visual special effects and post-processing techniques, the visual effects and atmosphere of the game are enhanced. This includes particle effects, smoke, explosions, blur effects, color grading and more ^[59]. AI can assist in creating and optimizing visual effects and machine learning algorithms can learn from existing effects or artistic styles to generate novel effects or enhance the quality of existing ones ^[60].

Artificial intelligence for user interface design: In the final stage of the visualization process, designers are responsible for creating the user interface (UI) of the game ^[61]. This includes designing menus, control panels, titles, buttons and other graphical elements that interact with the user ^[62,63]. AI can contribute to user interface design by automating certain aspects such as layout generation or adaptive UI systems that personalize the interface based on user preferences or gameplay context ^[64]. Additionally, AI techniques can be employed to optimize user interaction and responsiveness leading to a more intuitive and engaging user experience.

4.2.   Common video game AI techniques

AI techniques are employed to craft more realistic and interactive gaming experiences by facilitating the creation of NPCs that exhibit and respond in a more human-like manner ^[65]. This section will encompass some of the pivotal techniques and technologies utilized in the application of AI in video games.

4.2.1.   Ad hoc behavior authoring

Ad hoc behavior authoring is a technique employed to create or modify the behavior of artificial intelligence (AI) characters in a game, frequently in real-time or on-the-fly ^[66,67]. This approach is beneficial for developing dynamic and responsive AI capable of adapting to changing game conditions ^[68,69].

One method to implement ad hoc behavior authoring is through behavior trees, graphical representations of hierarchical behaviors that can be combined and modified to create complex AI behavior ^[70]. A behavior tree is composed of nodes, each representing a specific action or decision with branches connecting the nodes. The AI follows the tree's branches based on the conditions at each node and can execute different behaviors by modifying the nodes and branches.

Ad hoc behavior authoring can also be implemented using rule-based systems or decision-making algorithms such as finite state machines or decision trees. These systems enable the AI to choose from a set of predefined behaviors based on the current game state.

In addition to allowing for dynamic and reactive AI behavior, ad hoc behavior authoring can also be used to improve the efficiency and performance of the AI by allowing it to adapt to the specific needs of the video game and avoid unnecessary calculations.

4.2.2.   Tree search algorithms

Tree search algorithms are a class of algorithms that are used to explore the possible states of a system and find the optimal solution to a problem ^[71,72]. In the context of video game AI, tree search algorithms can be employed to determine the best course of action for an AI character within a video game.

In video game AI, various tree search algorithms are utilized to enhance both gameplay logic and visualization. Among them, Minimax focuses on determining optimal moves in two-player, zero-sum games like chess or tic-tac-toe ^[73]. By evaluating the game tree, it identifies moves that maximize the player's score while minimizing the opponent's score. An extension of Minimax, Alpha-beta pruning optimizes the search process by eliminating branches that do not impact the final outcome ^[74].

Another influential algorithm Monte Carlo tree search ^[75], explores game outcomes through random simulations making it particularly advantageous for games with a high number of possible moves. On the other hand, depth-first search exhaustively explores each branch before moving to the next, providing simplicity and effectiveness in certain cases but it may become inefficient for extensive or branching game trees ^[76,77]. In contrast, breadth-first search systematically explores nodes at each level ensuring comprehensive coverage but potentially suffering from slower execution in large or complex game trees ^[78].

Choosing the appropriate tree search algorithm is crucial to meet the specific requirements of the game and the intended AI implementation. These algorithms play a significant role in enhancing the visual experience of video games by enabling intelligent decision-making and optimizing gameplay dynamics.

4.2.3.   Evolutionary computation

Evolutionary computation is a set of techniques that employ principles of natural evolution, such as reproduction, mutation and selection to solve problems and optimize systems ^[79]. In the context of video game AI, evolutionary computation can be utilized to enhance the performance and capabilities of AI characters within a game.

One method of applying evolutionary computation in video game AI involves using genetic algorithms ^[80]. This approach entails creating a population of potential solutions to a problem and employing reproduction, mutation and selection to evolve the population over time. The solutions can be assessed based on their performance in the video game and the most successful solutions can be combined to generate new, more effective solutions.

Another strategy is the use of evolutionary neural networks ^[81]. These networks are neural networks trained using evolutionary algorithms rather than traditional backpropagation. Such networks can be employed to learn complex behaviors and adapt to changing conditions within the video game.

Evolutionary computation proves valuable for video game AI because it allows the AI to adapt and improve over time without requiring explicit programming. Additionally, it can be used to optimize the behavior of the AI for specific tasks or goals such as maximizing the score in a video game or minimizing the number of moves needed to complete a level.

4.2.4.   Machine learning

Machine learning is a subfield of artificial intelligence that involves training algorithms to automatically improve their performance on a specific task through experience. In the context of video game AI, machine learning can be employed to create more intelligent and realistic AI characters that can adapt to changing game conditions.

Incorporating machine learning into game AI allows for the adaptation and improvement of AI over time while also enabling it to learn and respond to the specific needs and preferences of the player. Machine learning techniques including supervised learning, unsupervised learning, reinforcement learning, and deep learning play a crucial role in achieving these goals.

Supervised learning involves training an AI model on a labeled dataset where the correct output is provided for each input ^[82]. For example, an AI model can be trained to predict the next move of an opponent in a game of chess by analyzing a large number of examples from past chess games. On the other hand, unsupervised learning entails training an AI model on a dataset without providing the correct output ^[83]. The model must independently learn to identify patterns and relationships in the data. For instance, an AI model can be trained to recognize different types of terrain in a game by analyzing numerous examples of various terrains.

Reinforcement learning is another approach that involves training an AI model to take actions in an environment to maximize a reward ^[84,85]. This method is often utilized in games where the AI must make decisions with long-term consequences such as resource allocation or strategic planning ^[86]. Deep learning is a subset of machine learning that involves training a neural network on a large dataset, enabling it to learn complex patterns and relationships within the data ^[87,88]. Deep learning techniques can be employed to enhance the performance of game AI in tasks such as image recognition, natural language processing and decision-making.

By leveraging these machine learning techniques, game AI can continuously evolve, adapt and deliver personalized experiences to players based on their preferences and gameplay style.

4.2.5.   Neural networks

A neural network is a type of machine learning algorithm inspired by the structure and function of the human brain ^[89,90,91]. It consists of a series of interconnected "neurons" that process and transmit information. In the context of game AI, neural networks can be employed to create intelligent and adaptive AI characters that learn from their experiences and adapt to changing game conditions (Figure 5).

Neural networks have found several applications in game AI, enabling AI characters to learn, adapt, and respond to a wide range of stimuli. One of these applications is decision-making where a neural network can be trained to analyze the current game state and available actions ^[92]. This empowers AI characters to adapt and respond appropriately in various situations enhancing their ability to make strategic decisions within the game. Another important application is predictive modeling where a neural network can be trained to predict the outcomes of different actions based on past experiences ^[93]. This capability allows AI characters to anticipate the actions of players or other AI characters and respond accordingly.

Pattern recognition is a valuable application of neural networks in game AI. By training a neural network to recognize patterns in data such as images or sounds, AI characters can react to various stimuli in the game environment ^[94]. For example, they can identify different types of terrain or specific objects enhancing their ability to interact with the game world in a realistic and immersive manner. Furthermore, natural language processing with neural networks enables AI characters to understand and respond to natural language input such as text or speech. This application enhances the interaction between AI characters and players, allowing for more natural and intuitive conversations. By training neural networks in natural language processing, AI characters can engage with players in a way that feels more human-like ^[95].

By incorporating neural networks into game AI, AI characters can continuously learn and adapt over time. This integration enables them to handle more complex tasks, respond to a broader range of stimuli and provide a more engaging and personalized gameplay experience. The use of neural networks in game AI opens up new possibilities for immersive and interactive video game visualization.

4.3.   Role of AI for game visualization in 2D and 3D games

Two-dimensional (2D) games refer to those displayed on a flat, two-dimensional plane while three-dimensional (3D) games are displayed in a three-dimensional space ^[96]. In 2D games (Figure 6), characters and objects are represented by flat, 2D sprites or images. In 3D games (Figure 7), they are represented by 3D models possessing height, width and depth. One of the primary distinctions between 2D and 3D games lies in the levels of immersion and realism they provide. 3D games tend to be more immersive and realistic, allowing players to experience a game world that more closely resembles the real world.

The primary distinction between AI for game visualization in 2D and 3D games lies in the added complexity found in 3D games due to the extra dimension ^[97,98,99]. In a 2D game, the AI system might be responsible for controlling non-player characters' (NPCs) or enemies' movements and behaviors or for generating visual effects or animations. In a 3D game the AI system might handle similar tasks. Still, it may also need to manage tasks such as pathfinding and navigation in a three-dimensional environment as well as more intricate animations and behaviors ^[100]. Additionally, AI systems employed in 3D games often work with more complex data structures and may need to handle larger and more detailed environments. This necessitates the use of advanced algorithms and techniques such as machine learning or procedural generation to accommodate the increased complexity of 3D games.

Here are some examples of AI systems employed in game visualization in 2D and 3D games:

In 2D games: NPC behavior and movement: Many 2D games utilize AI to control the movements and behaviors of NPCs or enemies ^[101]. For instance, in the game "The Legend of Zelda: A Link to the Past, " AI is employed to dictate the movements and behaviors of enemies such as their pursuit of the player or their attack patterns. Visual effects: AI can also generate visual effects in 2D games. In the game "Braid" (Figure 8), AI is used to create realistic smoke and fire effects.

In 3D games: Pathfinding and navigation: In 3D games, AI is often responsible for pathfinding and navigation for NPCs or enemies ^[97]. For example, the game "Doom 3" incorporates an AI system called the "monster AI" to manage pathfinding and navigation for enemies as well as their behaviors and attacks.

Animation: AI can also manage complex animations in 3D games ^[102]. In the game "Red Dead Redemption 2" an AI system called the "Pedestrian Navigation AI" is utilized to produce realistic and diverse NPC animations.

Here are some examples of AI systems employed in 2D games like Tetris, PAC-MAN (Figure 9) and Super Mario (Figure 10):

Tetris: In Tetris, AI is often utilized to generate the sequence of tetrominoes that are dropped into the playfield ^[103]. This can be achieved using various techniques such as predetermined patterns or random generation. Furthermore, some versions of Tetris employ AI to predict the best move for the player at any given time using techniques like search algorithms or machine learning.

PAC-MAN: In PAC-MAN, AI is responsible for controlling the movements and behaviors of the ghosts that chase the player ^[104]. The ghosts employ a set of predetermined behaviors such as chasing the player or patrolling a specific area in their attempt to catch the player. Some versions of PAC-MAN also use AI to generate the layout of the maze employing techniques like procedural generation.

Super Mario: In the Super Mario series of games, AI is utilized to control the movements and behaviors of NPCs including enemies and power-ups ^[105]. For instance, Goombas (a type of NPC in Super Mario) exhibit a set of predetermined behaviors such as walking in a straight line or turning around when they reach an edge to navigate the level. AI is also employed in some Super Mario games to generate the layout of levels, using techniques like procedural generation.

5.   Enhancement of video game visualization through AI

From the perspective of video games, AI can enhance the visualization of video games in several aspects including graphics generation and enhancement, real-time rendering and lighting, animation and motion capture, intelligent camera control and visual effects enhancement.

A variety of evidence has demonstrated that high-quality graphics and textures can be generated via AI technology making game scenes, characters and objects more realistic and detailed. For example, Fadaeddini et al. ^[106] used generative adversarial networks to autonomously generate original textures saving time and cost in game development with a specific focus on ground surface textures. The framework demonstrates visually acceptable generated textures with a mean score of 2.45 and 0.1 standard deviations after 2K iterations making it an effective tool for procedural texture synthesis in video game design. Using generative machine learning, Murphy et al. ^[107] proposed a face texture generation system with high-resolution allowing manipulation of skin, eye, lip and hair colors. Our system generates realistic face textures that strongly adhere to input appearance attributes providing 4K resolution and material property maps for raytraced rendering.

In addition, AI-driven advancements also enhance real-time rendering and lighting for more realistic and immersive game visuals. Zhong et al. ^[108] indicated that convolutional neural network technology had achieved results in various fields, and its powerful classification ability and learning ability have played a certain role and influence in enhancing the realistic visual effects of games. In ^[109], researchers proposed a machine learning approach for real-time upsampling of rendered content in high-resolution gaming and virtual reality. Leveraging temporal dynamics and specific information from modern renderers, the method achieves real-time performance, high fidelity and temporal stability. Training on a synthetic dataset yields superior results, outperforming existing techniques in challenging upsampling scenarios while maintaining enhanced video quality in real-time rendering. In addition utilizing video compositing algorithms that achieve interactive frame rates offers an alternative to photorealistic rendering in real-time ^[110]. Additionally, the system incorporates artificial intelligent agents to fulfill filmmaking roles. The user evaluation demonstrates the potential of this method for enhancing immersive interactive cinema experiences and advancing real-time rendering and lighting in video games.

In recent years, many studies have shown that the animation performance and motion capture effect of game characters have improved with the development of AI-related technologies. Nawalagatti et al. ^[111] in their article mentioned that artificial intelligence (AI) is a crucial aspect of game development, enhancing fun and realism. AI has advanced from playing tic-tac-toe to defeating humans in complex games. Motion capture and data-driven techniques combined with machine learning enable convincing character animations. Motion graphs and motion blending aid interactive character control. AI-driven animation techniques continue to push boundaries in the gaming industry. A study ^[112] presented a theoretical framework for procedural animation in video games using Laban's movement analysis (LMA) as a basis. By leveraging the space component of LMA, the proposed framework enables the generation of novel movements in virtual environments. The research aims to develop an AI system that can reason about human movements, laying the groundwork for real-time interaction in games utilizing motion capture or virtual reality and expanding gesture sets for humanoid avatars.

^[113] proposed a cloud-based VR system for coal worker training, using a browser/client architecture and eight modules to demonstrate the entire method of an underground coal mine. AI is used to enhance the emotional exchange between the system and users and two virtual miners are created to enhance learners' experience. The first virtual miner is a task-oriented NPC that provides broad mine information and directs players to subterranean workplaces. The second virtual miner is a disaster-preparation figure who prepares users for common calamities and the technology has been successfully tested in a laboratory setting. In 2022, Anantrasirichai et al. ^[114] illustrated that AI methods have revolutionized animation by automating the process and making it faster and more realistic. ML-based AI can learn motion characteristics from real sequences, enabling the animation of characters and dynamic movements. Technologies like PoseNet and FaceMesh allow real-time pose estimation and facial movement capture for cartoon animation. Adobe's character animator software offers real-time lip synchronization, eye tracking and gesture control widely used in Hollywood studios and online content creation.

Camera tracking is an integral part of the gameplay experience with the player's viewpoint switching and camera effects, etc. The use of AI enhances this feature, creating a wider and clearer view for the player. Khanian et al. ^[115] in their paper proposed several examples using photometric stereo to produce a useful 3D reconstruction. Future cameras with 3D forms can be created using the suggested methodologies for use in a variety of fields including film, video games and virtual reality. It is also a method that can provide high-quality 3D shapes from outside pictures and the internet without any prior expertise. Besides, a pose-assisted multi-camera collaboration system that is suggested in research ^[116] enables cameras to work together by exchanging camera postures for active object tracking. The system consists of two controllers and a switcher with the vision-based controller moving the camera in line with the poses of the other cameras and the pose-based controller tracking targets based on observed pictures.

Over the years, the visual effects of video games have gradually enhanced especially the application of AI has made the user's visual experience more three-dimensional. In a study ^[117], researchers examined how players perceive an identical video game when it is played in 2D, stereoscopic 3D or head-mounted-display (HMD) VR. Results indicated that presence was higher in the HMD VR than in the stereoscopic 3D than in the 2D video game. This means that the use of more advanced technology, such as AI or VR will make the user experience better. In 2020, a convolutional neural network model was presented in ^[118] to address the long-standing aliasing issue in the real-time 3D graphics sector. Max pooling layers and two-dimensional convolutional layers make up the neural network design which reduces the spatial dimension. The model is trained on a generalized and specialized version for anti-aliasing (trained for each application separately). It was discovered that a specialized version of the model performs best in terms of visual score and image quality metrics based on SSIM and PSNR scores.

Overall, these AI techniques can be leveraged to visualize gameplay in various ways such as predicting the best moves for a player to make, evolving strategies over time or making decisions based on past game situations. By incorporating these techniques, developers can create more advanced and sophisticated games that offer a more engaging and immersive experience for players.

6.   Recent applications of AI techniques to game visualization

The application of AI in game visualization is an active area of research, with the objective of creating more realistic, immersive, and interactive gaming experiences for players. AI techniques have been employed in various ways for game visualization and in this section we will discuss recent advancements in research on AI applications for game visualization.

In addition to generating synthetic images and ground truth, computer graphics enables the creation of virtual worlds (Figure 11). However, incorporating realism into virtual environments can be challenging. The video game industry has devoted substantial effort to developing 3D environments that players can interact with. Qiu and Yuille ^[119] introduced the open-source plugin UnrealCV for the renowned game engine Unreal Engine 4 to facilitate this process. They demonstrated two applications: (i) a proof-of-concept image dataset and (ii) the integration of Caffe with a virtual environment to evaluate deep learning methods.

In game visualization, AI techniques have been utilized in various ways. In this section, we discuss recent applications of AI techniques to game visualization to demonstrate the advancements in this area of research. Fink et al. proposed the first implementation of a general method for learning the behavior of NPCs (and other objects) in a game using only the game's graphical output during gameplay. The nearest-neighbor rule is combined with object tracking, situation-action pairs and object tracking to provide insight into what a human player might be able to perceive from the game ^[120].

Deep Reinforcement Learning (DRL) enables agents to learn methods for solving challenging tasks and has been applied to various problems, including gaming and natural language processing. However, due to the unpredictable nature of each action and the unknown reasons behind the outcomes, applying DRL to some real-world problems remains challenging. To address this issue, Joo and colleagues ^[121] developed a technique known as "explainable artificial intelligence" (XAI). This technology provides a representation of the AI process, making it easier for people to understand the outcomes of AI. Joo et al. suggested using Grad-CAM one of the XAI approaches for visualizing the actions of AI players who had undergone DRL training. Their experimental findings demonstrate which areas of the input state are targeted when a trained agent takes action.

In the domain of video game visualization, significant advancements have been made in the application of artificial intelligence (AI) particularly through the utilization of deep neural networks (DNNs). Despite their dominance, effectively comprehending and analyzing the underlying data within these networks continues to pose challenges. Visualizing these complex structures is crucial for comprehending how a network learns and functions and this area of research continues to be significant. Aamir et al. ^[122] address the issue of using the interactive presentation of DNNs in a virtual reality (VR) setup for conceptual understanding and architectural evaluation. They developed a static library as a plugin for the Caffe framework in the Unity gaming engine. For an image classification task, they designed and displayed a VR-based AlexNet architecture using the plugin's routines. Their layered interactive model allows users to easily navigate through the network during visual exploration. By selecting specific connections, users can understand the activity flow at a particular neuron enhancing the usability of the DNN model. This immersive tool is particularly informative for both beginners and experts in the field of DNNs providing more direct access to network architecture and results.

The challenge of automatically generating engaging and balanced units in real-time strategy (RTS) games remains an unsolved problem in procedural content generation (PCG) research. Even for human designers, creating distinctive and well-balanced units in an RTS game can be difficult. An automated unit design approach could assist developers in discovering new ideas and expediting the creative process. Sorochan et al. ^[123] propose a method for generating balanced and practical RTS units. They employ a fitness function based on Monte Carlo tree search and search-based PCG (MCTS). They present ten units generated by their algorithm specifically for the game microRTS, along with data demonstrating their originality, utility and balance.

Strategy video games present AI agents with challenges in combinatorial search spaces. State abstraction is a prevalent method for reducing the complexity of the state space. However, current state abstraction techniques for games are costly to apply to new ones as they rely on domain expertise. In the planning domain, state abstraction techniques that do not require domain knowledge have been extensively researched. Nevertheless, their suitability for coping with the inherent complexity of strategy games remains unverified. Xu and colleagues ^[124] present Elastic MCTS, an algorithm that leverages state abstraction to play strategy games. In Elastic MCTS, the nodes of the tree are dynamically clustered, initially grouped together by state abstraction and then divided when an iteration threshold is reached.

To solve the Sokoban puzzle, Crippa et al. ^[125] extend Monte Carlo tree search for single-player games (SP-MCTS) and compare the solver's performance against that of a solver combining Iterative Deepening A* (IDA*) with a number of problem-specific heuristics. The results indicate that IDA* remains the leading Sokoban solver, partly because it can easily incorporate a substantial amount of domain knowledge.

MCTS is a stochastic simulation-based best-first search method. Yoshida et al. ^[126] apply MCTS to a fighting game AI and evaluate its performance on FightingICE, a platform for game AI competitions at conferences on computational intelligence and video games. Their findings support the claim that MCTS is an effective method for influencing game AI on the aforementioned platform.

The utilization of the metaverse in education has witnessed a surge in popularity, as evidenced by recent studies focusing on enhancing learning outcomes through increased student engagement and immersive experiences. Hyun et al. ^[127] develop educational gaming materials about the decision tree algorithm in the metaverse to teach middle school students about AI. By incorporating games into the classroom, they aim to make learning about artificial intelligence more engaging and effective for students.

The choice of algorithm used in AI plays a crucial role in developing advanced and effective gaming AI. Man-versus-machine conflicts appear in nearly all games and to enhance the experience for human players it is essential to investigate how to select the best algorithm to improve computer players' efficiency and winning percentage. Ma and colleagues ^[128] use a specific game as an example to compare the effectiveness of four different algorithms. They then match the various algorithms in pairs and determine which algorithm has the highest winning percentage considering the game's first and second hands, running time, memory usage and CPU utilization rate. After conducting their research, they find that the Minimax theorem method has the best winning percentage when running time is disregarded.

Artificial intelligence has traditionally been centered on computer games. From the standpoint of simulating human brain intelligence, it builds a number of computer systems that can imitate human information acquisition, processing, decision-making and learning in the context of gaming.

Recent advancements in AI have been found to significantly impact people's quality of life, particularly by enabling unique types of human-computer interaction for individuals with disabilities. In their study, Nasri et al. ^[129] propose a 3D sEMG game that employs a deep learning-based architecture for real-time gesture recognition. They collect a novel dataset of seven movements using the Myo armband device and use it to train the suggested deep learning model. The recorded signals are fed into a Conv-GRU architecture to identify the gestures.

The evolution of AI and ML has brought the study of human-AI interaction to the forefront of HCI research. Zhu et al. ^[130] argue that the examination and experimentation of how people engage with AI are best conducted in the context of games. The primary conclusion is that AI, as a tool, can expand current notions of human-AI interaction, which are predominantly productivity-based. Their research suggests that game and UX designers should consider flow when structuring the learning curve of human-AI interaction, incorporate discovery-based learning for users to experiment with AI and observe the outcomes and extend an invitation to play to help users discover new ways to interact with AI.

7.   Discussion

Artificial intelligence (AI) has profoundly transformed the field of video game visualization, providing numerous advantages and paving the way for innovative possibilities in the future. As with any technology, it also poses certain challenges and potential issues that warrant careful consideration and exploration.

The application of artificial intelligence (AI) in video game visualization offers a multitude of benefits that have significantly enhanced various aspects of video games. On one hand, AI algorithms contribute to improved graphics and realism by generating realistic graphics, animations and special effects in video games including AI-powered procedural generation for creating authentic environments, characters and objects. On the other hand, AI promotes increased efficiency by automating tasks like graphics optimization and content generation saving time and resources for developers. AI contributes to improved accessibility for players with disabilities by offering options such as text-to-speech or visual aids making gaming experiences more inclusive and enjoyable for all players.

AI in video game visualization also presents several problems and challenges that need to be addressed. One primary concern is bias, as AI algorithms can inherit biases from the data they are trained on, potentially leading to unfair or unjust outcomes in games such as AI-powered NPCs exhibiting biased behaviors based on gender, race or other factors ^[65,101,131]. Another challenge is the lack of transparency in AI algorithms which can be difficult to understand and interpret, hindering players from comprehending the logic behind certain actions or events in the game ^[132]. Over-dependency on AI can also make games less engaging or creative by declining players' sense of control over their actions or outcomes ^[133]. Moreover, AI-powered games may be susceptible to security risks such as hacking. Addressing these challenges is crucial for ensuring the responsible and effective integration of AI in game visualization.

In the future, we can anticipate several trends in AI for video game visualization. One of these is the continuous improvement of AI models, leading to more advanced and realistic AI in video games such as AI that can learn from and adapt to player behavior or generate more varied and dynamic content. Additionally, there is likely to be increasing use of machine learning, allowing video games to adapt and evolve over time and providing more personalized and unique experiences for each player. AI is also expected to be increasingly integrated with other technologies such as virtual and augmented reality, resulting in more immersive and interactive gaming experiences. As AI becomes more prevalent in video games, it is essential to address the ethical implications of its use including issues like bias, transparency and player autonomy to ensure trustworthy and beneficial applications of AI in video game visualization. The implementation of rigorous data preprocessing techniques such as data augmentation and de-biasing has been proven helpful.

In summary, AI holds significant potential to revolutionize the field of video game visualization, offering advantages such as improved graphics and realism, enhanced gameplay and increased efficiency. Nevertheless, it is crucial to be mindful of potential problems and challenges while considering the ethical implications of AI utilization in gaming. The future of AI in video games is poised to feature more advanced and realistic AI models, an increased application of machine learning and greater integration with other cutting-edge technologies paving the way for innovative and immersive gaming experiences.

8.   Conclusions

In this review paper, we have explored a wide range of AI applications in video game visualization, encompassing machine learning algorithms for character motion, terrain generation and lighting effects. We have also discussed the advantages, challenges and potential applications of AI in this domain. One emerging trend is using AI to enhance the visual quality of video games either by employing machine learning algorithms to generate more realistic graphics or by optimizing the rendering process. Another trend involves the creation of more immersive and interactive game environments through AI. Moreover, there is a growing tendency to leverage AI for the development of more realistic and intelligent game characters by utilizing machine learning algorithms that empower them to make decisions and adapt to dynamic game environments. The future of artificial intelligence in video games is anticipated to feature increasingly sophisticated and lifelike AI models, expanded applications of machine learning and enhanced integration with other advanced technologies.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Conflict of interest

The authors declare there is no conflict of interest.