Applied convolutional neural network framework for tagging healthcare systems in crowd protest environment

1.   Introduction

Internet of things (IoT) is an interconnection of networks where multiple devices are connected through Internet ^[1]. The devices can range from camera, computers, sensors etc through which information is ingested and processed in the ubiquitous IoT framework. The information can be accessed anywhere, anytime. Wireless Sensor Networks (WSN) ^[2] propels IoT system framework which has revolutionized the manner of data transmission. Within a WSN the job of the sensors is to sense the respective environment and send it to the appropriate devices such as local host computer or any central cloud server. Modern IoT systems have various smart systems such as smart dustbin ^[3], smart irrigation system ^[4], smart healthcare system ^[5], and smart traffic control ^[6] to name a few. The focus of this proposed research is on the smart healthcare systems. The focus in the smart healthcare system is usually on the reactive state of healthcare systems. This reactive state deals after an event have happened and the patient has been admitted in the hospital where his vital life signs are monitored, stored and analysed. Smart healthcare is one of the serious challenges that needs to be addressed especially for elderly patients. Smart wearable devices for elderly patients ^[7] can be included in intelligent framework for smart cities. The devices record the sensor data and transmit it to a secured cloud-based platform. Smart healthcare has multiple attributes such as remote patient monitoring, wearable devices based smart computing and preparing response for emergency services.

Intelligent transport systems in smart cities works assists emergency situations ^[6].Intelligent transport improves the chances of saving lives in case of accidents, natural disasters and man-made situations such as violent protests. Smart cities must have provisions for dealing emergency situations. Proactive emergency response preparation is the key to face crowd protests and violence. Smart healthcare systems tag heterogeneous fields and different departments to ensure safety of patients, especially elderly patient as well as health care institutions. Smart heterogeneous applications like smart homes ^[8] have proven that the convergence of AI and IoT technologies in the best optimized ways during video streaming through wireless micro medical devices (WMMDs) in smart healthcare homes. Constant research in improving surveillance framework for managing crowd protests must be developed. This framework will assist in the deduction of severity of protests and can alert the healthcare systems for the availability of healthcare professionals and specialist.

1.1.   Psychology of protest

The psychology of protest has been studied and the researchers have approached the idea of protests and demonstrations using social psychological approach. Van Stekelenburg et al. ^[9] presented motives for protests and demonstration by humans. The paper presented five unique domains which accelerates the initiation and processing of crowd protest behavior. These five things are as follows:

a) Grievances: Public anger about situations drives the initiations of protests. Joining of any popular political personality can drive agitations and protests ^[10].

b) Efficacy: It is an individual's person belief that can change their present conditions, policies through protest and demonstrations.

c) Identity: Identification with group, belief or any organization can give the impetus to join the protests.

d) Emotions: Anger is the main emotion that gives the energy to humans to go for protests.

e) Social Embeddedness: Sharing grievances of society as a whole with other fellow members can initiate the drive for protests.

As per Figure 1, an overview of the envisioned IoT environment-based framework is presented. The framework is designed for a crowd protest environment. It can be clearly observed that crowd protests activities must be monitored through CCTV and drone cameras in a more sophisticated environment. This is a centrally distributed approach. The feeds are transferred to the central cloud device where the crowd protests feeds are deciphered and classified into multiple stages based on the severity of the protests. Deep learning is used for training the crowd protests models and then further classification is done on the live feeds of the protesting crowd. Based on the severity of crowd protests appropriate tagging is initiated for available healthcare systems and professionals. IoT devices such as CCTV sensors and cloud computing servers would be needed to integrate the seamless transmission of information across the city. The aim of the IoT service is to make sure that primary healthcare infrastructure is secured and upgraded and people dissenting, protesting and indulging in violent activities are attended to in any emergency situation. Thus, an effective response is computed and the specialist healthcare infrastructures and professionals are tagged in such unfortunate event.

1.2.   Psychology of violence

The psychology of violence believes that there is a difference that exists in the domain of biology, psychology, sociology, culture, and mass communication which leads to the idea of violence. A general theory breaking down violence is by describing their respective attributes. External as well as internal environment of individual is responsible for motivation to carry out violence. DeKeseredy et al. ^[11] came up with strong theory on psychology of violence in twelve partitions which include exchange theory, subcultural theory, resource theory, patriarchal theory, ecological theory, social learning theory, evolutionary theory, sociobiological theory, pathological conflict theory, psychopathological theory, general systems theory, and inequality theory. The same book cites reasons such as feelings of alienation, shame, humiliation by individual or group of people, mortification, rejection by individual or society as a whole, abandonment, denial, mental health ailments leading to depression, anger due to societal injustice, hostility within communities, projection, and displacement of individual or a group of people can cause extreme protests and violence.

1.3.   Subsystems of healthcare

Healthcare systems consists of internal as well as external healthcare subsystems. Internal framework systems focus on the individual patient level interactions admitted to the hospital. External subsystem analyzes the external environment to decipher the need of healthcare systems, professionals, specialist doctors, emergency services in a real time situation. These subsystems facilitate the tagging of healthcare systems to the needy people. Figure 2 highlights the subsystems of healthcare framework.

The focus of this research is to study external crowd subsystems where crowded places and their events are observed in a camera based IoT environment. The present research focuses on external subsystem where concentration is on protests sites and violent crowds' sites. Scenarios concerning these two events are included in the current study for tagging healthcare systems.

Crowd protests consists of demonstration, remonstration or remonstrance and it is an expression of disagreement or objection towards an idea or action, typically a political one ^[12,13]. Protests have been related to cause stir in the mentality of the nation ^[14,15]. Social movements ^[16] can trigger protests in countries ^[17]. Protests can spiral in to violence if the demands of protesters are not met. Aggravated people can resort to violent methods to show their disapproval. Violence classification assumes prime significance in securing human life and public places and properties ^[18]. Figure 3 gives us the fair idea about how protests is organized using group of people, banners, children, banners, shouting and so on. Protesting crowds have high energy ^[19] which can translate itself in to violence laced situation. Interesting cues regarding images and sounds have been proposed in last decade with aim to accelerate the research ^[20,21]. This research paved the path for more research in computer vision domain.

Convolutional neural networks (CNNs) are a new framework for deep learning solutions. The paper shall refer Convolutional neural networks (CNNs) as ConvNets. ConvNets have been applied for different computer vision applications ^{[22,23,24,25]}. A sample ConvNets is shown in the figure below.

The ConvNets have the ability to retain significant image features which it uses to learn the respective classification. The paper presents a novel application of using pre-trained ConvNets to classify stages protests and violence. Popular pretrained models are VGG16 ^[26], Inception ResNet V2 ^[27]. Each of these architectures have performed exceptionally well in ImageNet Large Scale Visual Recognition Challenge (ILSVRC) ^[28]. In the event of moderate protests to extreme protests, it is the responsibility of police and law enforcement agencies to control the protests. The response from the police team for countering protests and violence is use of tear gas, pepper spray, batons, shields, and rubber bullets. The consequences of using these measures presents grave dangers to the protestors and rioters. In the midst of the protests and counter protest processes injuries are observed to the crowds. Most common injuries in case of moderate to extreme protests are ocular trauma and subsequent vision loss ^[29]. Thus, the paper introduces the framework of crowd behavior situations concerning protests and violence, use of deep learning to classify the situations and popular injuries that happens to the participating crowd in such situation. The contribution of the paper is as follows:

● Utilization of an existing protests datasets ^[30].

● Curation and utilization of violence dataset from YouTube.

● Explain the rule base to categorize the protests.

● Fuzzification of rules for degree of protests seen in the images/videos.

● Based on the degree of protests tagging of hospitals and healthcare professionals is generated.

● The idea can be spread to various other crowd domains where individual or mass of people gets injured. The whole scenario is envisaged for deployment in an IoT based smart city framework.

Our paper follows the following structure of the paper. Section 2 discusses the brief related work of the latest protest and violence detection methods. Section 3 discusses the proposed methodology describing the pre-processing philosophy of the input and current proposed ConvNets frameworks. Section 4 describes the experimental setups along with the datasets, Section 5 describes the protests detection performance, Section 6 comes up with the discussion of the paper and Section 7 deals with the conclusion of the paper.

2.   Related work

A novel multi-task convolutional neural network was developed for recognizing protesters. This model deciphered the protestors activities using their visual attributes and estimate the perceived violence in the image ^[30]. Our literature survey specifically focusses on deep learning-based methods for classifying protests or protests like events. Computer vision can be used for deciphering political events related questions using large-scale visual content. A study has been performed for classification of facial features for deciphering their biases towards political movements and leaders using twitter users' photographs ^[31]. Another study has been performed on the images of political leaders that has been shared on social and multimedia platforms ^[32] for their perceived emotions and personality traits. Image classification domain depends upon visual features in the input media content. These features can help classify the content in to creative content, interesting content and adult provocative themes ^[33]. Social event detection has been a constant source of research using computer vision and a deep learning approach has been applied using Graph Convolutional Network ^[34]. Computer vision has also been used for detecting extreme stages of protests such as arson and stone pelting ^[35]. A ConvNets and recurrent neural network approach known as collective action from social media (CASM) image data and recurrent neural networks with long short-term memory on text data was used in a two-stage classifier to identify offline collective action from social media data ^[36]. A novel research study was dedicated for extracting protests news by using Long Short-Term Memory and combined with a deep learning model developed on the inter country datasets ^[37].Our study also includes related work on violence detection methods which can be a direct result from the protest event. Novel method of Bidirectional Convolutional LSTM has been given by applied by Hanson et.al ^[38]. A multistream ConvNets architecture using 3D CNN and ConvNets was proposed in ^[23] and conducted on the CCTV-Fights dataset. A flow mechanism was deployed to detect violent event in crowds using deep learning ^[39] and is entirely based on the local level conditions of Bangladesh. Song et al. ^[40] used application of 3D CNN for violence detection approach. 3D CNN was especially used for convolutional operations on extracting the spatio-temporal features of the frames. Xu et al. proposed P3D-LSTM violent video classification ^[41]. Li et al. ^[42] synthesized a novel method by using 3D CNN utilizing multiplayer violence detection procedure. Another excellent method using 3D CNN method was proposed by Tran et al. using 3D CNN. The 3D ConvNets were designed to learn the procedure of learning the spatiotemporal features of the input videos and then classifying the videos using machine learning procedures ^[43]. Zhou et al. applies 3D-CNN for getting deep image features for violence classification ^[44]. Mandal et al. experimented with discriminative features using fine-tuned deep convolutional neural residual network for classifying crowd behavior ^[45]. Ammar et al. worked extracted video frames and applied CNN and LSTM layers for classification ^[46]. Meng et al. experimented with trajectory-based motion features and deep CNN for violence detection ^[47].

Solving protest environment using ConvNets is a challenging task. A simpler and yet a novel approach is to use pretrained ConvNets subjected to fine tuning which can classify the protests and can then the situation can be classified in to their respective fuzzy categories. The authors believe that the crowd formation itself presents an opportunity to tag healthcare system although then the threat level is very less and tagging is not necessary in the normal crowd situation. The approach presented in the current paper is novel, simple and easy to tag the healthcare systems.

3.   Proposed methodology

We approach the problem of deciphering two domains and the convergence of one domain in to one another. Crowd protests can transform themselves in to crowd violence and in some case crowd violence reverts back to crowd protests. As long as the protests are under control, peaceful, local law and order authorities have no problems with the demonstrators. This research article tries to decipher the various stages of convergence of protests in to violence using pretrained ConvNets. The idea of this research is to assess the protesting site for any need of the specialist healthcare workers/doctors. Escalation of protest stage to violent stage can have serious ramifications. To prove a point pretrained ConvNets are utilized in the current research. We utilize the cluster of pretrained ConvNets architecture and combined them for deducing the protests-violence classification using VGG16 ^[26] and Inception ResNet V2 ^[27]. It is a deeper network and pre-trained on more than a million images on the famous ImageNet database ^[48]. These ConvNets have been chosen as they have shown better performance in tackling the problem of vanishing gradient ^[48].

The previous section has highlighted the methods that is being applied for the protests and violence detection methods. Latest literature has few methods that directly deals with crowd protests. Also, violence detection using ConvNets are also fewer. The methods that are being touched here utilizes the pretrained Convnets. The main objective of choosing this method is to make sure that this proof of concept transforms itself in to real time program which can be used for Android and web-based devices equally. Table 2 presents a simpler difference approach between existing methodologies and proposed methodology.

Let us assume that X to be the of the protest dataset $\left|X\right|$ The error $\in$is calculated as:

where ${X}_{i}$ is the ${i}^{th}$ image of training dataset X, $\mathcal{F}({X}_{i}, w)$ is the ConvNet function that predicts the class ${c}_{i}$ of ${X}_{i}$ given the respective weights $w$, and $\widehat{{c}_{i}}$ is the ground-truth class of the ${i}^{th}$ image, and $\mathcal{L}({c}_{i}, \widehat{{c}_{i}})$ is a penalty function for predicting ${c}_{i}$ instead of $\widehat{{c}_{i}}$. We can set $\mathcal{L}$ to the logistic loss function ^[49]. In the proposed method we Adaptive Gradient Descent (AGD/AdaGrad) ^[50].The cost function to compute the optimization error is as follows:

$p\left({c}_{i}\right|{X}_{i})$ denotes the probability by which ${X}_{i}$ is correctly classified in the protest dataset. If ${W}_{h}^{t}$ denotes the weights in ${h}^{th}$ hidden convolutional layer at iteration $t$, and $\widehat{\mathcal{L}}$ denotes the cost over a mini-batch of size M, then the updated weights in the next iteration are computed as below:

where ${\alpha }_{h}$ is the learning rate for the ${h}^{th}$ layer, $ϼ$ is the momentum that signifies the contribution of the last weight update in current iteration and $\eta$ is the scheduling rate that decreases $\alpha$ at the end of each epoch. The Adaptive Gradient Descent was computed for optimizations vide the following equations with respect to the mini batches formed similar to that used in SGD.

where ${K}_{h, h}^{t}\in {R}^{d\times d}$ represents a matrix such that the diagonal elements $(h, h)$ equals the sum of gradients. Following the above calculation, the weights in the fine-tuning architecture were updated for optimization as indicated by Eq (7). We can clearly infer from Eqs (3)–(7), the back-propagation activity was carried out iteratively to fine tune the pretrained model. Following two pretrained ConvNets are used in deciphering the multiclass protests and then used for computing severity of the protests.

3.1.   VGG16 ConvNets

VGG16 is one of the popular pretrained ConvNets and had a good classification accuracy in ImageNet large scale visual recognition challenge (ILSVRC) ^[28]. This is simple to use and is popular in the transfer learning domain. In the process of fine tuning VGG16, the fully connected top layers are removed and new fully-connected layers are added for our classification of tasks. The model utilizes a set of weights pre-trained on ImageNet. VGG16 uses gradient descent combined with a loss function over a sufficiently large dataset propels the ConvNets to learn the patterns in the complex visual space data that is fed to the network. VGG16 consists of 16 convolutional layers. These layers help the architecture in feature extraction. VGG16 has three main constituents: convolution, pooling, and fully connected layers. VGG16 proceeds with two convolution layers. Then a pooling layer is added. The pattern is repeated again. Then three convolutions are repeated with pooling attached. Finally, three FC (Fully Connected) layers are added.

Figure 5 presents the VGG16 architecture which is used in the current research to highlights the working of ConvNets and is generated using ^[51].The above architecture has single global max pooling layers, single fully connected layers with 512 units, single drop out layer and single softmax activation layers of violence enabled inputs. The layers in VGG16 are frozen and feature extraction is performed for the additional FC layer.

3.2.   Inception ResNet ConvNets

With the introduction of residual connections with normal inception blocks has yielded good performance in 2015 ILSVRC challenge and the new architecture is known as Inception ResNet V2 ^[27]. These residual connections facilitate the shortcuts in the model to train deeper neural networks, Hence the performance has increased for inception blocks.

Figure 6 shows the steps of the input images to the final classification of the protests. This model reduces the representational bottleneck by introducing factorization of large block in to two smaller blocks. Also, it expanded the filter banks horizontally rather than vertically. This ensured less loss of information. The above steps were incorporated to build three different types of inception modules known as Figure 6. Instead of training the model from scratch the paper relies upon the transfer learning concept and does fine tuning of the Inception-ResNet V2 on the protest dataset.

4.   Experimental approach

We have evaluated our approach on standard imbalanced protests datasets ^[30] and have tried to classify various classes of protests dataset using transfer learning. To further prove the research analysis, we have curated a violence related dataset mostly from YouTube and other related videos. Protest dataset ^[30] have been used extensively for classification using three pretrained ConvNets. The number of images in this dataset is 40,000 images. We have added an additional category of loudspeakers for protest dataset and removed violence category from this dataset. Instead, we have curated our own images and added to the violence category. The approach is twofold. One is to analyse protests from the angle of Table 2 and find out the severity of the protest events. The second is to classify protests and violence. We are focussing on protests dataset and their combinations among them for the fuzzification of the severity of protest. We design a rule base for identifying the stages of protests and its severity. Table 3 presents the rule base in a tabular form:

The above table gives a fair idea about how to interpret the table. There are twelve attributes which decide the situation. Based on the presence of few or more attributes beyond a threshold, the protest situation is classified in to different normal, medium and extreme level protests. For example, if there is a presence of night, children, fire, shooting and police presence the protests situation is in severe category and appropriate solution is suggested. One thing we noted that combinations of fire, night, police, children, shouting and loudspeakers and group of more than 100 denoted severe protests in to combining in to violence. These combinations have inherently more chances of transformations of protests in to violence. As soon as these combinations are detected an alert notification is pushed to the all the available health care centres of the city. Based on the severity categories an alert for specialist healthcare personnel is generated. Based on the presence of these twelve attributes of any crowd protest event, healthcare systems are tagged and a ready alert is generated. A report by Amnesty international ^[52] discussed about the police actions on the protestors. The report discussed about the actions taken by police in tackling unarmed protestors and use of weapons to neutralize them. Serious injuries amounting to loss of lives have been reported. As stated in the literature, the standard police response to control protests is to use multitude of weapons, such as tear gas, pepper spray, rubber bullets, batons, riot shields, and TASERS ^[53]. Similarly violent protestors display their potential by using rocks, sticks, knives, guns, and Molotov cocktails ^[54]. Taser devices are used by law enforcement agencies to control violent protests and riots. These devices are known to reduce the risk of injury and death by minimizing the use of lethal force. Spectrum of injuries reported by usages of taser ^[55] is shown in the Table 4:

Hence locating these surgeons nearby to the protests and violence can help save many lives and prevent major injury losses. Below table is a research article that focusses on the types of injuries that is faced by the crowd protestors.

Table 5 describes the types of serious injuries that happens in severe protests and violence. Table 6 describes the types of doctors needed for the kind of injuries reported in Table 4.

The above tables clearly describe the types of police response used, types of injuries inflicted and types of specialists needed in case of severe protests. Thus, a proper relationship between protests, response, injuries and specialist healthcare professionals has been established in this research. Now based on the severity of protests, general physicians as well as specialists' doctors would always be available in the hospitals during protests. For the implementation of proposed protests classification and their respective typical injuries detection in crowded scene algorithm, we used standard Keras application with Python on Ubuntu operating system and experiments were conducted on a standard machine (Intel Core i5, 8 GB RAM) equipped with NVIDIA TITAN X GP.

5.   Protest detection performance

We deployed the above-defined architecture. The results are divided in to two approaches:

● The first approach uses pretrained ConvNets for multiclass classification of protests datasets. We use VGG16 and Inception Resnet V2 for the protest's classification.

● A rule based is defined in the previous section. Based on this rule base an alert is generated which can be sent to the nearest healthcare centers demanding the readiness of specialist's doctors as described in the previous sections. Sending the alerts can be automatic or through a human in the loop ^[62]. This research believes that the human is necessary to broadcast the protest level to all the healthcare centers in the vicinity of the protest's sites. The whole research is dedicated to the protest's behavior and its respective severity.

Total dataset is revised by adding loudspeakers to the protests datasets and violence category is separated from the original dataset. Then the utility is invoked for displaying of dataset images. The image is as follows:

Then the experiment proceeded with VGG16 and Inception Resent V2 pretrained ConvNets. The experiment utilizes k-fold cross Validation technique. Cross-validation is a unique procedure used for evaluation of deep learning models on the limited data sample. As specified in the literature ^[63], the choice of k is normally taken to be 5 or 10. Arbitrary increase in the value of k cannot guarantee better results as resampling subsets gets smaller. The experimental procedure is as follows:

1. Protest dataset is shuffled randomly.

2. Split the dataset into k = 5 groups

3. For each such group:

√ Take the group as a hold out or test data set

√ Take the remaining groups as a training data set

√ Fit a model on the training set and evaluate it on the test set

√ Retain the evaluation score and discard the model

4. Model evaluation scores and respective confusion matrix is described thereafter.

Evaluation metrics: The performance of the proposed research concept was tested and on the parameters of accuracy, specificity, sensitivity, precision, and f1-score. The parameters used in the following equations are true-positive ${N}_{TP}$, number of true negative ${N}_{TN}$, number of false positive ${N}_{FP}$, and number of false negatives ${N}_{FN}$ to measure predicted classes which are based on the evaluation metric mentioned in Eqs (8–12):

VGG16 confusion matrix and classification results are as follows. The respective confusion matrix is as follows:

The training accuracy is a follow:

The results of Inception Resnet V2 in the form of confusion matrix and five-fold accuracy graphs is as follows:

The 5-fold accuracy graph for all the three approaches is as follows:

From the above Table 7, it becomes clear that our proposed framework achieves substantial performance on the protest dataset proving that our association strategy of pretrained ConvNets for protest detection have given practical results. The results are promising and they can be further improved using more balanced dataset by improving protest dataset. The results in the multiclass classification are returned in the probability. The returned probabilities of the classes beyond a threshold are taken as the entity present in the input images. Multiclass classification helps in classification of image datasets and helps in classifying the images. Based on this multi class classification various rule set can be framed and a decision support system can be easily made for assisting humans. Further a simple binary classification for classification of protests and violence category is also complied in this research. Here a binary classification between protest and violence dataset is undertaken. The aim for such a classification is that the detection of violence in the input images/videos can make the alert system more perfect for alerting healthcare systems personnel.

The protest and violence dataset were subsampled to total of 9000 images each. A classification scheme using pretrained ConvNets was utilized. We use VGG16 for the classification purpose. Finetuning of VGG16 is undertaken. The top fully connected layers are removed newly fully connected layer is added to the existing pretrained ConvNets. This new layer consisted of global max pooling layers, one fully connected layer, one drops out layer with 0.49 rate and one softmax activation function for 2 types of categories. Feature extraction is performed for the newly added FC layer. This ensure that the weights in the system is not random and gradient would be under control. Post feature extraction the final ConvNets layer is unfreeze and fine tuning of the model starts for father iteration. The learning is rate is 0.001 and the Ada Grad optimizer is applied on the classification algorithm.

The results of the system are as follows:

Fine tuning the whole model using transfer learning gives better result and the model converge faster.

Figure 15 and Table 8 shows the accuracy and loss diagrams of the VGG16 approach for classifying protest and violence. To the best of knowledge, this is a unique classification scheme of classifying image categories of protests and violence. Violent activities automatically qualify for tagging of specialist healthcare system and surgeons. Figures 14 and 15 describes the result and shows overfitting results. The sample is size is limited which may not generalize well on unseen data. As per Figure 12 a simplistic framework for classification is shown where convolution and pooling layers combinations extend in to the deep learning architecture. The approach to tackle overfitting is to optimize and reduce the deep learning capacity by removing convolutional layers as shown in Figure 12 and reducing the number of elements in the hidden layers. Application of regularization and usage of dropout layers can help in reducing the overfitting in the model performance. A recent research article in the medical field has proposed a novel deep learning model for more efficient and precise recognition of Tetralogy of Fallot (TOF) ^[64] has discussed a novel method of deploying stochastic pooling methods to counter overfitting models. A rank-based stochastic pooling (RSP) for breast cancer classification has been recently published ^[65]. A future version of this paper can be extended by combining two advanced neural networks namely Graph convolutional network (GCN) and ConvNets (CNN).

6.   Discussion

Protest detection for crowd management is a challenging job. This research has presented a way to link crowd behaviour to deep learning and tagging healthcare system to the safety of people. There are various reasons which affects individuals and group of people which can trigger the cause of protests. Protest can manifest itself due to various reasons which includes anger, emotions, identity to social embeddedness. The idea percolated in this research is that protests are collective crowd behaviour. If not handled properly, protests can spiral in to violence. The paper explores post protests experiences and safety of the people who are in protesting, police who are controlling the protestors and normal crowd who are just bystanders, onlookers who are witnessing the protesting events. Thus, a lot of people gets involved, voluntarily or involuntarily. The paper explores this crowd behaviour using protest dataset. Using multiple attributes of the protest dataset this research classifies the severity of crowd protests using rule set belonging to the respective attributes of protest dataset. Converging of protests in to violence can cause stern retaliatory action from the law enforcement agencies. The significance of this research is to come out with a safety plan of people who may get injured in the protests. This research highlighted the common injuries that happens during severe protests. These injuries are sustained by the protestors as well as the police forces. Even the bystanders can get fatally injured in these protests. The highlight of the research is to make a protest storyboard. People protesting would definitely face police controllable actions. Protesting people are united either by anger or identity issues, they usually turn violent to make their point of view heard by the government system and authorities. Amidst all this chaos, if the healthcare systems, professionals, ambulances, medicine stores etc are ready for protest and violence situation, the objective of this research is achieved. At the end of the day no human lives must be lost due to violent protests. Crowd behaviour like protests and riots is a very hard problem to solve using any images datasets. Further the current research links the type of injuries that can happen during protests and severe protests. Each such injury is mapped to a specialist health care personnel to specific surgeons. Also, it is imperative to view the tagging of healthcare systems from two point of view. The first view is dependent solely on the multiclass protest classification. Based on their respective threshold probability score, the rule sets are fired and an alert is generated for tagging healthcare systems. The second view is to generate the classification of each frame of the real time of the input video in to protests and violent categories. This research believes that if the violent state is detected then it automatically is qualified for tagging specialist healthcare systems and doctors to be at alert for saving human lives. This research can be a game changer in the IoT enabled smart cities and devices. Future versions of this can be Android complied and an app can be made functional for directly pushing notifications to the healthcare professionals. All the medicines, injections, anti-biotics, oxygen cylinders needed for such scenarios could be readied beforehand and the human lives can be saved even in the midst of sever protests and violence.

6.1.   Contexts, scenarios and solution

People believe in the power of protests. The time frame of acceptance of the protesters' demands may vary. In the initial stages of protests, the event looks like a failure but the power of protests becomes visible in the long term. A system is envisaged which interconnects the subsystems of healthcare tagging. This system is connected to the videos / live stream of the protesting sites. These frames from these feeds are extracted and pumped into the trained model which classifies the protest attributes. Each combination of protest attributes forms a rule which is triggered respectively. There are usually two types of healthcare centers. Primary healthcare centers are controlled by the government and the private hospitals. All the primary health care centers are geo mapped. As soon as the protests' severity begins, their respective classification and notifications regarding the state of the protests begin to generate. These notifications are pushed through the pipeline and concerned authorities are informed about the ongoing situations. Following prototype is envisaged to prove our points of tagging healthcare amidst protests and violence. There are eight camera sub screen which can receive the video feeds from the CCTV cameras via RTSP protocols. For testing purpose, the videos have been taken from Internet and a proof-of-concept procedure is developed. The images shown are the camera feeds of the protesting and violent crowds and the respective police presences. Protest severity is determined and push notifications to the ten primary health care centers are sent to be ready for receiving patients with external injuries. This research can be used in different protests scenarios. For example, in a developing country like India, two recent protests have created a widespread headline all over the world. These protests are farmers protests ^[66] which is still in continuing and protests against the Citizenship Amendment Act ^[67] have given the impetus to authors to analyze the protests scenarios and minimize injuries to people, police and innocent bystanders. Below there are samples of images that highlights the proof of system images where feeds can be received on line from a connected CCTV / camera. As it has already been mentioned in Figure 1 of introduction section, the concept is to make a full fledge system where a command-and-control center deals with all the feeds. It is accompanied by the preprocessing and decision support system. All these operations converge onto crowd controlling deployable system using the deep learning framework. The video streams undergo deep learning based analysis and severity of protests is ascertained. The final results goes to the decision support system which helps the human operator to take a final call on the prevailing situation.

The system shall take feeds from various CCTV feeds. For testing purpose, the system is shown to have eight screens. It can be extended to multiple screens. The system can see all feeds in one screen or if a user wants to explore the feeds, individual camera screen can be accessed.

The feeds that are received is ingested by the trained model. The model classifies it presence of various protests attributes. As soon as the protests attributes are received in the single frame, the rules are fired. Based on the rules, the ten nearest primary healthcare centers are identified and notifications to these centers are pushed via messaging application. The researchers assume that there are two levels of healthcare centers involved in this strategy. First the Primary Health Care centers of the Government, and top government hospitals and secondly all private hospitals which are well equipped to handle severe traumatic cases. The assumption is that in the actual implementation details all the health care centers would be marked on a map and a central authority would be responsible for sending the grievances messages to the nearest healthcare centers. Based on their responses, actual actions would be initiated to inform the driver of the ambulance which in turn can take the injured persons to the nearest healthcare centers.

7.   Conclusions

This research is a unique attempt to link the protest storyboard to healthcare systems. This research uses deep learning for multiclass classification of the protest datasets. Based on the respective thresholds of the protest's attributes, a rule set has been presented which classifies the severity of protests in to normal, medium and severe category. Also, it shows an exclusive differentiation between protests and violence. In case of violence a direct tagging of healthcare infrastructures and doctors can be raised by the human in the loop. This is a novel study to alert the specialist healthcare doctors to be on standby in case the severity of protests breaches the thresholds. This framework can be integrated further with an IoT environment and Android application so that respective messaging push notification can be send to directly to all the concerned specialist healthcare personnel. The researchers have assumed that it is full fledge Internet based system with linkage to CCTV cameras, the video streaming, the geo-location of all the healthcare centres situated in the city on a map, and a human in the loop operator which can guide the messaging of injured person to the nearest health care centres. As soon as the healthcare institution accepts the injured person, the system moves on the next phase of taking care of the injured persons with specialist doctors, surgeons, equipment and so on. The main objective is to save human lives of the protestors as well as the police personnel. Automatic surveillance becomes a lot easier and any sort of protests in public places can be tactfully handled. A real time framework of assessing the protests site can help prepare for any eventuality. The current research can be a big boon for tagging healthcare systems and can ensure availability of specialist doctors as per the severity of doctors, medicines, ambulances and other healthcare entities.

Conflict of interest

All authors declare no conflicts of interest in this paper.