A Vision sensing-based automatic evaluation method for teaching effect based on deep residual network

1.   Introduction

University English teaching activities are a dynamic process involving many variables and influencing factors, which makes the challenging to establish an automatic evaluation scheme from the side of teachers ^[1]. Except for the consideration of teachers' classroom teaching ^[2,3], it is also necessary to take into account the factors of students who directly participate in teaching ^[4]. Students participate in the full range of classroom teaching, directly feeling the teacher's teaching quality. Also, students participate in the whole process of classroom teaching and can directly feel the quality of teachers' teaching, as well as their own learning effect ^[5].

Digital teaching mode is one of the new classroom teaching modes promoted by the Ministry of Education, not only enriches English teaching resources, but also optimizes English teaching methods. Its re-integration of English teaching and learning is even more advantageous ^[6]. An information-based teaching environment was created, and this changes the traditional teaching environment^[7]. It not only enriches English teaching resources and optimizes English teaching methods, but also re-integrates and gives full play to teaching advantages, creates an information-based teaching environment, changes the traditional teaching mode, meets the current requirements for "hardware facilities" in curriculum construction, and promotes the benign development of the university teaching ecosystem ^[8,9]. The software construction of "golden course" is about improving the quality of teaching in the classroom. Among them, the key to ensure the quality of the course is to build a scientific and effective course quality evaluation system. At present, most domestic universities' classroom teaching quality evaluation is coordinated by an academic affairs organization, including the construction of evaluation system and the publication of an evaluation results. Especially in a setting of evaluation indexes, many universities do not distinguish the nature of courses and use uniform evaluation indexes that include both teaching attitudes and contents and teaching methods and effects, which has the advantage of ensuring the evaluation of common problems and the disadvantage of ignoring the characteristics of each course ^[10,11]. With the innovation of the education evaluation concept, the drawbacks of this teaching evaluation system gradually appear, lagging behind and having limitations, which can neither improve the quality of course teaching nor restrict the creation of high-quality courses. Therefore, it has become an important task to build an evaluation system which is in line with the new teaching concept and adapts to the new mode of teaching reform in the digital era ^[12,13].

Classroom teaching quality assessment is the basic content of education quality evaluation, which should not only assess the effect of the credit hours but take note of the effect of on a time period as well. An effective teaching evaluation model needs the support of effective evaluation indexes in order to get scientific evaluation results. In recent years, academics have actively carried out research on digital classroom teaching evaluation, such as Fang Xucai^[14], who proposed four dimensions of planning and preparing classroom teaching, second classroom extension and teaching responsibility and other four dimensions of constructing a foreign language teaching evaluation model; Li Zhihe et al.^[15] constructed evaluation indexes of online teachers' teaching ability, etc. However, these evaluation model studies focus on the exploration of the validity of evaluation indicators. A deepening of the research on evaluation methods is still needed.

Classroom teaching behavior analysis technology is moving toward automation and intelligence with information technology, which makes the original teaching evaluation system and method extremely challenging. The application of new technologies to normalize and scale up classroom teaching evaluation provides strong support for studying the laws of classroom teaching and exploring the essence of learning. In the learning process, emotions influence human cognition and behavior, and grasping the emotional state of learners is particularly important for future research on intelligent and personalized education. Therefore, a number of scholars have introduced emotion recognition into instructional analysis models. Han Li compared cognitive behaviors with students' head posture and facial expression behavior to build a classroom teaching evaluation system based on face detection and expression analysis^[16]. Cao et al.^[17] used multimodal data to build a student learning engagement recognition model based on deep learning networks. Zhao Min et al. based on the original quality evaluation indexes, through modeling and obtaining teaching quality evaluation through deep learning network. Sun et al.^[18] analyzed and studied students' emotions based on video processing technology. There are few papers dedicated to the identification of students' classroom behaviors. Zhou et al. ^[19] used a face detection, wheel. Contour detection and subject action amplitude detection were obtained from the dataset, and a Bayesian causal net was used as an inference model to determine the characteristics of subject behavior for classroom teaching behavior recognition. Dang ^[20], on the other hand, described and judged actions by extracting Zernike moment features, optical flow features, and global motion direction features of actions and combining them with a plain Bayesian classifier. Zhang ^[12] classified and identified the action vectors by extracting features from human skeletal vectors, and then using an SVM classifier^[13]. The above methods mainly use traditional machine learning methods that require a large number of manual steps and have a low accuracy rate. Liao et al. ^[21] successfully identified three classroom behaviors: sleeping, playing with the phone, and normal, by capturing students' classroom behaviors through a camera and extracting the target regions through background differencing into a VGG network ^[22]. This study provides a new idea and method for classroom behavior recognition by applying deep learning technology to classroom teaching image recognition, but the number of students recognized is small, and the recognition of a students' actions in the classroom is simple and the accuracy rate is still low.

In recent years, deep convolutional neural networks have been rapidly developing, and models such as AlexNet ^[22], VGGNet ^[23], and GoogLeNet ^[24,25,26] have been proposed one after another. However, when the number of network layers keeps deepening, the gradient explosion or gradient disappearance problem in deep neural networks during training process becomes more and more obvious. To solve this issue, He et al. proposed ResNet ^[27]. One of the important features of this network is the inclusion of the residual module, which successfully alleviates the network degradation problem when the network layers are too deep by adding Shortcut structures between the convolutional layers. The ResNet has been extended into many application scenarios due to its resilience and proper processing performance. Qiao et al. explored the utilization of ResNet in the scene of heart disease-related medical image processing in work ^[28]. On this basis, Qiao et al. also proposed a novel feature learning detection system with the use of deep learning-based vision sensing technology in the work ^[29]. Besides, a more advanced four-chamber semantic parsing network is also proposed in a similar area by Qiao et al. in the work ^[30]. To identify more students' behaviors in the classroom with higher accuracy, we applied a deep residual network to classroom behavior recognition. The classroom behavior recognition dataset is constructed from a wide range of images of live student classroom behavior. A deep residual network for this dataset over the characteristics of the residual module, which offers a fresh approach to recognizing students' classroom behaviors.

2.   Methodology

The assessment of the teaching effect is implemented by perceiving various behaviors in the classrooms, with the assistance of a deep neural network. The ResNet model is selected here for this purpose. The main solution thought of this paper is to recognize some basic behaviors of classes using the ResNet model. These involved behaviors contain both positive behaviors and negative behaviors. The ResNet model is utilized to perceive these behaviors and extract these features as the final discriminative basis.

Then, the discriminative results of the teaching effect can be calculated.

2.1.   Residual network

Residual network belongs to a deep convolutional neural network. As far as convolutional neural networks are concerned, the fitting ability can be enhanced by increasing as many layers of the network as possible. But, as the number of layers deepens, the training of convolutional neural networks turns out to be extremely difficult. Once more than a certain level of layers are added, there is a decrease in the recognition ability of the network ^[31,32,33]. During gradient back propagation, parameters of the network near the output layer converge fast, whereas those near the input layer converge slowly due to the deeper layers of the network.

To overcome the problem of decreasing recognition accuracy caused due to too many layers, a residual unit is introduced by the residual network. That is, a Shortcut structure is added between the convolutional layers, which leads to a residual after the objective function of the network training is made to differ from the input function, as shown in Figure 1. In traditional learning goal, the objective is to learn a mapping function, as follows:

where $x$ denotes input of learning function, and $y$ denotes learning result. Differently, in residual network, the expression of learning goals is:

Its goal is to learn the residual between $y$ and $x$, instead of single $y$. In other words, the actual output is the sum of the original output and the original input, thus transforming the fit of the network to $f(x)$ into the fit of $h(x)$. The residual item is denoted as follows:

This structure does not add new parameters and extra computational effort, and also solves the problem of gradient dispersion in the back-propagation process of the network. The Figure 1 gives structure of a residual block, and a number of residual blocks are able to constitute the different residual neural networks.

2.2.   Deep residual network

The structure of the deep residual network used to identify students' classroom behavior is shown in Figure 2. The network is made up of one convolutional layer, two ReLU layers, three pooling layers, one convolutional module, two constant modules, two fully connected layers, and finally a classification layer, in which the convolutional layers are filled with same. The input image first passes through the convolutional layer, which contains 64 convolutional kernels of size $2\times 2$ with an operation step of 2, activated by the ReLU activation lattice for initial feature extraction. It is followed by a convolutional module, two constant modules for deep feature extraction, and two fully connected layers of different sizes for feature dimension reduction. The output neurons of the latter fully connected layer are 6. This corresponds to the latter fully connected layer has six neurons, which corresponds to the 6 behaviors of students in the classroom, and finally, the classification results are output through the classification layer. For convolution layers and pooling layers, the scale transformation rule is represented as the following the formula:

where ${\Delta _{transform}}$ denotes scale of feature map after convolution or pooling operation, ${{\Delta _{initial}}}$ denotes scale of feature map before convolution or pooling operation, ${{\Delta _{pad}}}$ denotes scale of padding operation, ${{\Delta _{filter}}}$ denotes scale of convolution filter or pooling filter, ${{\Delta _{stride}}}$ denotes scale of the stride operation.

The structure of the constant module in Figure 4. The network structure of the convolutional module is shown in Figure 3. The constant module combines three convolutional layers, three ReLU layers and a shortcut connection operation. The convolutional layer 1 contains 64 channels of $1\times 1$ convolutional kernels with operation step 1, the convolutional layer 2 contains 64 channels of $3\times 3$ convolutional kernels with operation step 1, and the convolutional layer 3 contains 256 channels of $1\times 1$ convolutional kernels with operation step 1. The Shortcut join operation is used to add the input of the constant module with the output after three convolutional operations, which reflects the basic idea of residual network. This operation reflects the basic idea of the residual network. The following formula can be deduced to represent the transformation process of convolution operations:

where $W_C$ denotes the weighted parameter for this transformation, $B_C$ denotes the bias parameter for this transformation, $\star$ denotes convolution operation symbol, and $\mu_1 \left(\cdot \right)$ ia another activation function named as Sigmoid, whose representation is as follows:

The convolutional module consists of four convolutional layers, three ReLU layers and a Shortcut connection operation. The convolutional layer 1 contains 64 channels of kernels with a size of $1\times 1$; the convolutional layer 2 contains 64 channels of kernels with a size of $3\times 3$; the convolutional layer 3 contains 256 channels of kernels with a size of $1\times 1$; and the convolutional layer 4 contains 256 channels of kernels with a size of $1\times 1$. Compared with the constant module, it performs a convolutional operation on the network input x before the Shortcut connection operation. After some residual blocks, the initial input is transformed into representations with the abstract format. It will be transferred into the prediction result via a full connection operation structure. This process takes the form of the following equation:

where $W_C$ denotes weight parameter in this operation, $B_C$ denotes bias parameter in this operation, and $\mu_2 \left(\cdot \right)$ denotes sigmoid activation function, which is expressed as:

It can be seen from the above formula that it tries to confine the output into the range of $\left(0, 1\right)$.

2.3.   Implementation of deep residual network

Convolutional neural networks have powerful fitting capabilities and are capable of learning complex mapping relationships from input to output. Even if the exact mathematical expressions from input to output are not known, the convolutional neural network can establish the mapping relationship between them more accurately by learning specific patterns from input to output. The training of convolutional neural networks is generally done by supervised training. The training process is divided into two main phases, namely the forward propagation phase and the backward propagation phase^[34,35]. In the forward propagation phase, to improve the accuracy of the model and enable the network to converge rapidly. In this paper, the training set is first randomly disrupted, and then a fixed number of small batches of images are selected as the network input in each-iteration with the machine situation. The input is propagated forward through the constructed network architecture layer by layer, and finally, the probability of recognition of each behavior is output through the softmax classification layer.

In the back-propagation stage, the error value is first calculated using cross-entropy as the loss function, and then the error is back-propagated by the Adam optimizer to update the network weights and gradually make the loss function close to the optimal value in order to optimize the whole network. In addition, this paper adopts drop-out coding method when coding label categories, and the network the learning rate is 0.001, and uses Drop-out technique back in the fully connected layer, i.e., the neurons are deactivated randomly at each training, so as to alleviate the network overfitting and achieve the regularization effect. And the loss function is shown as follows:

Then, the stochastic gradient descent can be utilized to solve the above objective function. Let $\Theta$ denotes the set of all the parameters to be learned in the proposed model. The learning process of $\Theta$ with use of stochastic gradient descent can be represented as the following formula:

where $l$ denotes the index number of iterative rounds, and $r$ denotes the learning rate.

3.   Experiment and evaluation

3.1.   Experimental setup

Because classroom behavior data is not publicly available on the web, we collected the data ourselves to create a classroom behavior recognition dataset. The video is collected by a camera installed in the classroom with a resolution of 2560 × 1536. The video collection includes six actions that students frequently perform in the classroom, such as taking notes, looking around, and reading books. Then the videos are collected, videos are first sampled with uniform frames and converted into images. The images were then cropped into images containing individual students and reshaped to a resolution of 128 × 128, and the classroom behaviors of students in each image were labeled to obtain a total of 1020 images with labels. The original dataset was expanded by mirror-symmetric data augmentation to obtain a classroom behavior recognition dataset containing 2040 images. The number of images for each behavior is the same. The data was randomly selected for training and testing. The training set and test set had 1560 and 480 images respectively.

Based on the data from both sides of teaching obtained by video analysis method, this paper initially designs an index system for online teaching quality evaluation, and the techniques used for each index are different. The most difficult one is the emotional changes of both teaching parties, where the emotional identification includes the emotional interaction between teaching and learning process in addition to the original single emotional identification^[36,37,38]. This interactive emotional analysis can better reflect students' engagement in learning and teachers' adjustment of teaching according to students' emotional changes.

To verify the effectiveness of the index system of the teaching quality assessment model constructed in this study, this study organized a total of 347 undergraduate students in the 2019 class of a university to evaluate 12 English teachers of the university by means of a questionnaire.

3.2.   Evaluation and analysis

The ResNet is first implemented on the experimental image dataset^[39]. It serves as the backbone network for visual feature extraction. Two demos for recognition results from the ResNet method are demonstrated in Figure 5. It has two subfigures, in which (a) corresponds to the recognition under scenes of individual objects, and (b) corresponds to the recognition under scenes of groups of objects. It can be observed from Figure 5 that the ResNet can well deal with scenes of both individual objects and groups of objects, and that different behaviors in classrooms can be well recognized.

To better reveal the performance superiority of the ResNet method, his paper compares the recognition performance of ResNet with DCNN and YOLO models on classroom behavior data sets. The average recognition accuracy of the three experimental methods is illustrated in Figure 6. It can be seen from this image that the utilization of ResNet as a backbone can achieve better results than the other two. And the final accuracy of the ResNet and DCNN is shown in Figure 7. From this figure, we can see that the generalization accuracy of DCNN is 89.46%, while that of ResNet is 91.91%. That is, the generalization accuracy of ResNet is higher than CNN, indicating that the addition of residual structure can enhance network performance. One of the comparison plots for the iterative update of the model recognition accuracy is shown in Figures 8 and 9. The former focuses on behaviors of sleeping, reading, and looking around, while the latter focuses on behaviors of playing with the phone, taking notes, and attending classes. Besides, the relationship between data amount size and average accuracy is shown in Figure 10. The X-axis denotes the data size amount, and the Y-axis denotes accuracy values. The graph shows that as the amount of data size increases, the average accuracy shows an upward trend. This may be attributed as the fact that more video amount is able to help train better recognition models.

The accuracy of each behavior recognized by the deep residual network is shown in Table 1, in which sleeping and reading achieved higher recognition accuracy of 97.06% and 94.12%, respectively, while the recognition accuracy of playing with the phone, taking notes, attending class, and looking around were 92.65%, 89.71%, 91.18%, and 86.76%, respectively. The recognition accuracy of looking around and class is relatively low, which is probably due to the fact that students sit in various positions in the classroom and their heads have certain biases during class, resulting in a certain similarity between the behavior of students in class and looking around in the dataset, which leads to network misidentification and reduces their recognition accuracy.

In addition, we compared the results of this paper's proposed method with those of traditional methods for assessing satisfaction with teaching quality, and a comparison graph of satisfaction is given in Figure 11. From this figure, it shows that the satisfaction of teaching quality assessment using the method proposed in this paper is more than 32% higher than that of the traditional method due to the full consideration of students' classroom interaction performance.

4.   Conclusions

It takes complex operations to extract features from classroom behavioral images with traditional machine learning methods. And it has a low classification accuracy. The image features can be automatically extracted by CNN compared to traditional methods. Its end-to-end approach to training the network from input to output allows the network to recognize classroom behavior, which improves the accuracy and reduces computational complexity at the same time. However, network training becomes more complicated as the level of the network increases. It even brings the problem of network performance degradation. In this paper, we propose a deep residual network model for classroom behavior recognition by introducing the residual structure into the CNN. The results of the experiments demonstrate that this network has a better performance compared with the deep convolutional neural network.

As has been noted by Amelio et al. ^[40], although the ResNet has many advantages in some specific image processing tasks, it still suffers from some rough issues such as heavy network structure. This requires high computational performance and proper hardware conditions. In addition, it is a typical peer-to-peer model that acts as a black box during processing tasks. This limits its explainability when dealing with various tasks. The two major points are our future direction after this work. It is expected to explore the better applications of the ResNet methods to make it have a lighter network structure and better explainability.

Conflict of interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work was supported by First-class Course of Zhengzhou Sias University.