Figure 1.
The structure diagram of the proposed gesture recognition system.
Citation: Daniele Agostinelli, Roberto Cerbino, Juan C. Del Alamo, Antonio DeSimone, Stephanie Höhn, Cristian Micheletti, Giovanni Noselli, Eran Sharon, Julia Yeomans. MicroMotility: State of the art, recent accomplishments and perspectives on the mathematical modeling of bio-motility at microscopic scales[J]. Mathematics in Engineering, 2020, 2(2): 230-252. doi: 10.3934/mine.2020011
| [1] | Yan Dong . Local Hölder continuity of nonnegative weak solutions of inverse variation-inequality problems of non-divergence type. Electronic Research Archive, 2024, 32(1): 473-485. doi: 10.3934/era.2024023 |
| [2] | Yiyuan Qian, Haiming Song, Xiaoshen Wang, Kai Zhang . Primal-dual active-set method for solving the unilateral pricing problem of American better-of options on two assets. Electronic Research Archive, 2022, 30(1): 90-115. doi: 10.3934/era.2022005 |
| [3] | Yiyuan Qian, Kai Zhang, Jingzhi Li, Xiaoshen Wang . Adaptive neural network surrogate model for solving the implied volatility of time-dependent American option via Bayesian inference. Electronic Research Archive, 2022, 30(6): 2335-2355. doi: 10.3934/era.2022119 |
| [4] | Qing Zhao, Shuhong Chen . Regularity for very weak solutions to elliptic equations of p-Laplacian type. Electronic Research Archive, 2025, 33(6): 3482-3495. doi: 10.3934/era.2025154 |
| [5] | Yaojun Ye, Qianqian Zhu . Existence and nonexistence of global solutions for logarithmic hyperbolic equation. Electronic Research Archive, 2022, 30(3): 1035-1051. doi: 10.3934/era.2022054 |
| [6] | Xincai Zhu, Yajie Zhu . Existence and limit behavior of constraint minimizers for elliptic equations with two nonlocal terms. Electronic Research Archive, 2024, 32(8): 4991-5009. doi: 10.3934/era.2024230 |
| [7] | Yong Zhou, Jia Wei He, Ahmed Alsaedi, Bashir Ahmad . The well-posedness for semilinear time fractional wave equations on RN. Electronic Research Archive, 2022, 30(8): 2981-3003. doi: 10.3934/era.2022151 |
| [8] | Tao Zhang, Tingzhi Cheng . A priori estimates of solutions to nonlinear fractional Laplacian equation. Electronic Research Archive, 2023, 31(2): 1119-1133. doi: 10.3934/era.2023056 |
| [9] | Erlin Guo, Patrick Ling . Estimation of the quadratic variation of log prices based on the Itô semi-martingale. Electronic Research Archive, 2024, 32(2): 799-811. doi: 10.3934/era.2024038 |
| [10] | Hui Jian, Min Gong, Meixia Cai . Global existence, blow-up and mass concentration for the inhomogeneous nonlinear Schrödinger equation with inverse-square potential. Electronic Research Archive, 2023, 31(12): 7427-7451. doi: 10.3934/era.2023375 |
Gesture recognition has attracted much attention across different fields due to its various applications in healthcare, rehabilitation, caring for the deaf-mutes, etc. [1]. Through our field trip to a Chinese provincial medical institute, we got to know that it is normally hard for doctors to measure the effectiveness of rehabilitation follow-up treatment after patients leave the hospital. Therefore, we are commissioned by this institute to design an offline wearable device for real-time gesture recognition to help with remote diagnosis. In the field of hand gesture recognition, we laid the groundwork for the design of an edge device that is able to undergo on-device AI computing. In other words, with our efficient hardware, robust software, and improved AI algorithms, we are able to process data and complete the recognition process locally, rather than in the cloud or on PC.
In the field of gesture recognition, image processing has been a popular established protocol [2,3], and the use of artificial intelligence (AI) has dramatically improved the accuracy and robustness of image-based gesture recognition [4,5]. However, it usually requires a powerful processor for completing various functions, which severely limits the portability of the device. Moreover, image-processing technology may be interfered with by multiple factors, including ray and skin color, which significantly raises training costs and may potentially bring ethical issues. Furthermore, image-processing technology cannot efficiently ascertain the speed during gesture recognition, making it hard for its application in similar medical rehabilitation scenarios.
In terms of sensors for gesture recognition, surface electromyography (SEMG) sensor [6], highly flexible fabric strain sensor [7], and many other sensors are limited by their higher costs and power consumption. That is why we decide to use the inertial measurement unit (IMU) sensors. With its characteristics of low cost, low power consumption, and high performance, it is an excellent solution to record the acceleration and angular velocity of gestures.
To date, in existing gesture recognition systems to our knowledge, gesture data are collected and then sent to a remote processing system with higher computing power, which significantly increases the costs.
In this paper, a prototype of a low-cost wearable computing system for gesture recognition is proposed, which uses IMU sensors to collect data and locally implements gesture recognition functions, without relaying anything to other devices with higher computing power. As far as we know, it is probably the most effective system that simultaneously achieves low-power, low-latency, and high-accuracy in the field of gesture recognition.
The contributions of this study are as follows: (1) A highly-integrated wearable edge device that locally recognizes hand gestures, rather than uploading data to the cloud as in previous studies; (2) A low-power field-programmable gate array (FPGA) Accelerator deployment via a serial-parallel hybrid method to reduce the resource consumption and improve computational efficiency; (3) A software-hardware co-design based on the Cortex-M0 IP core that improves the system’s generalization ability, which is also referable for other scenarios; (4) A new activation function for the NN+CNN structure, which promotes the recognition accuracy; (5) Open-source data for experiments that will be continuously updated.
The rest of the paper is organized as follows. Section 2 introduces the existing gesture recognition methods and AI implementations in wearable designs. The architecture for this software-hardware co-design system, as well as the proposed algorithms for the accelerator, are explained in detail in section 3. Section 4 displays the main part of our work, including the de-noising process, serial-parallel hybrid method, and resource-efficient scheduling of calculations in the FPGA accelerator. Section 5 demonstrates the datasets, experimental results, and the comparisons of resource consumption and performance between different systems on-chip, including MCUs, SBCs, and desktop processors. Finally, we conclude our work in section 6.
Some common gesture recognition methods and AI implementations in wearable devices are reviewed in the subsequent two paragraphs.
The main methods for gesture recognition are image (video) processing and sensor data processing. Visual interest point [8,9,10], a classic and effective means of gesture recognition, is based on the color, shape, or motion. Neural network algorithms, including three-dimensional (3-D) CNNs, hybrid deep learning, and long short-term memory (LSTM), are widely used in image processing for gesture recognition [11,12,13]. These approaches effectively reduce the effects of various illuminations, views, and self-structural characteristics. Depth maps have been used to enhance the effect of a large number of small articulations in the hand [14,15]. The IMU or similar sensors are widely used, for instance, in studies where gloves have been used to measure acceleration, finger knuckle angle, and spatial position of hands to recognize gestures [6,16]. In such applications, sensors, including IMU, resistance strain gauge, image sensors, and acceleration sensors are distributed on the back of the hand, forefinger, and the middle finger. Typically, IMU or acceleration sensors are integrated into the peripherals of many wearable devices [17,18,19]. Previous studies have reported that the values collected by the acceleration sensor are combined with the gestures dictionary, and then feature extraction is used to convert the problem to the classification problem, thus realizing the gesture recognition [20,21]. However, all the proposed methods above used personal computers (PCs) or more powerful devices to process data, which seems redundant and inconvenient for mobile uses such as follow-up treatment for rehabilitation. In contrast, our device can achieve data collection, data processing, and gesture recognition totally offline. Meanwhile, it’s worth noting that although RNN and LSTM generally have better performance on time series data, and have been implemented on FPGA in several studies [22,23,24], they do not contribute to our research objective. They are usually achieved by High Level Synthesis (HLS) instead of hardware description language, which brings uncontrollability to resource and power consumption, reduces the system’s portability, and adversely affects future Application-Specific Integrated Circuit (ASIC) deployment. Compared with its drawbacks, the mere improvements in accuracy are not worth mentioning in our case.
Researchers in [25] designed a CNN accelerator on heterogeneous Cortex-M3/FPGA architecture. The accelerator consists of a convolution circuit and a pooling circuit. This CNN accelerator uses 4901 LUTs without hardware multipliers. A throughput of 6.54 GPOS was achieved. A different study used a very low-power NN accelerator on Micro-semi IGLOO FPGA [26], with a multilayer perceptron NN architecture for classification. The shortest accelerator latency was 1.99 μs and the lowest energy was 34.91 MW under different coupling. Similar findings were drawn in [27], which proposed a recurrent neural network (RNN) model for IoT (internet of things) devices with an end-to-end authentication system based on breathing acoustics. The advancements of these systems in achieving lower power and lower resource consumption is fundamental to our work. And by combining CNN + multilayer perceptron NN, as well as improving the activation function, we lowered the resource consumption and improved recognition accuracy.
The structure of the IMU sensor gesture recognition system designed in this paper is shown in Figure 1. This system consists of an FPGA for the Cortex-M0 IP core implementation, and a glove with IMU sensors driven by Cortex-M0. The data is first sent to the acceleration module by different coupling ways, and then the data processing is accelerated by Cortex-M0.
We made two sets of devices, one for the left hand and one for the right hand. The actual right-handed glove prototype with six IMU sensors is shown in Figure 2. There are five sensors distributed on the second joint of each finger, including thumb, fore, middle, ring, and little, as well as one sensor placed on the back of the palm. The sensors are driven, and simultaneously the data is obtained by the Cortex-M0 via the I2C communication protocol. The data collected undergoes Kalman filtering in the inner module. Two-dimensional (2-D) data in the six sensors are collected and then extracted in the return system for gesture classification. The gestures we sought to recognize are gestures for numbers from one to ten, as demonstrated in Figure 3.
A schematic diagram of the accelerator is shown in Figure 4, including the preprocessing module (PM), feature extraction module (FEM), and classification module (CM).
After the Cortex-M0 drives the IMU, each of the six sensors collects the 2-D angle data with 12 dimensions. The data are then sent to the PM by the Kalman filtering algorithm in the IMU module. The interference caused by external factors such as a slight movement in gesture changes is reduced by smoothing de-noising based on the wavelet transform (WT). The sliding-window and bottom-up (SWAB) algorithm is subsequently used to segment and extract effective data. The extracted effective data are input into the FEM to extract the eigenvalue with CNN. Then the extracted eigenvalues undergo the feature rescaling operation by the Cortex-M0, and are then input into the CM for classification by the multilayer perceptron neural network. Finally, the recognized gesture results are sent to M0 as the output.
Compared to the traditional gesture recognition methods or neural network classifications in the MCU platform, the PM introduced in our approach effectively reduces noise in the downstream processing and the amount of data processed. Additionally, the system does not need expert supervision once trained to properly classify the input data, as CNN is introduced to extract hidden features from the data collected by the IMU sensors [20].
As previously mentioned, in order to achieve effective gesture classification, it is necessary to extract various gesture features from the original data. And instead of extracting features by hand, which is highly dependent on expert supervision in the training process [28], we adopted CNN in the system to automatically complete the process. The original signal is preprocessed for feature extraction enhancement. Moreover, the effect of classification is directly affected by the extracted features.
The PM may fall into two classes i.e., the wavelet de-noising and SWAB (the hardware implementation is explained later in section 4.1). During data collection, fatigue or unconscious trembles of the user may introduce noise in the collected gesture data. While noise has large impacts at the stage of feature extraction and lowers the learning rate, de-noising can effectively improve the accuracy of the final model [29,30,31]. Hence, to overcome this challenge, WT is used for de-noising and filtering [32]. The square-integrable signal x(t) is expressed as
| WTx(a,b)=1√a∫x(t)ψ∗(t−ba)dt=∫x(t)ψ∗a,b(t)dt=<x(t),ψa,b(t)> | (1) |
where b is the time shift, a is the scale factor, and ψ(t) is the basic wavelet. The wavelet basic function is obtained via the shift and stretching of the parent wave as shown in Eq 2.
| ψa,b(t)=1√aψ(t−ba) | (2) |
After passing through the WT, the signals distributed in each layer are uncorrelated to achieve de-noising. As the thresholds of signals at various scales are determined, the thresholds can be quantitatively processed. The corresponding threshold formula is as shown in Eq 3.
| thj=γ×√median(|d_j(k)|)0.6754×√2ln(length(dj(k)))ln(j+1) | (3) |
where γ is the noise power spectrum parameter, median( ) is the median corresponding to the input, dj(k) is the coefficient of the wavelet in the jth layer, j is the decomposition scale, and length(dj(k)) is the length of the input data. The threshold in the wavelet is the standard for deciding whether to keep or zero the wavelet coefficients. Under high resolution, if the signal amplitude is lower than the threshold value, the wavelet coefficient is set to zero; otherwise, the original wavelet coefficient is retained.
To facilitate quantification, the calculation of FPGA is executed using the fixed threshold formulas shown in Eqs 4 and 5. After de-noising, signals are obtained by reconstruction.
| th=γ×σ×√2ln(length(dj(k))) | (4) |
| σ=median(|dj(k)|)0.6754 | (5) |
The process of data collection by the IMU glove includes the preparation stage, movement stage, and completion stage. Regarding signals, values are stable in the preparation stage, but they change in the movement stage before stabilizing again in the completion stage. Some of the data collected in the preparation and completion stage are not essential for sample identification. Such data may severely influence the performance of the subsequent module. Therefore, the filtered data is extracted twice to minimize unnecessary data. Here, this is achieved using the improved SWAB algorithm, which combines the sliding-window (SW) and bottom-up (B-up) algorithms [33]. First, the sum of squares of the 2-D signal values of the collected by the six IMU sensors is taken as the changed measurement as expressed in Eq 6.
| signal=6∑i=1(a2xi+a2yi) | (6) |
Figure 5a demonstrates the segmentation of valid data in the traditional SWAB algorithm. This algorithm makes the length of feature data varied in different signal channels. However, as it is required to have the same data length in the next stage, other irrelevant and non-contextual data will have to be filled into those short data, which greatly influences the recognition effects.
Figure 5b presents the segmentation in the improved SWAB algorithm. After up-sampling different signal channels, a unified length of valid feature signal is determined and the starting and ending points are passed to the next module. There are three major advantages (a) the feature data lengths are the same after segmentation, (b) the computation required by the filling operation is reduced, (c) the original context of the feature data is preserved.
Signals that can be regarded as the basis for gesture recognition are segmented by setting appropriate thresholds and then passed through the PW. The corresponding waveform is shown in Figure 6.
Then, in FEM, a CNN model was constructed to extract gesture features (the hardware implementation is explained in section 4.2). After passing signals through PW, these data were used as the bottom input on CNN. Training features were adopted to extract the model. After the training, the model parameters were saved. Finally, the collected gesture data were put into the trained model to extract gesture features for output. A schematic diagram of the CNN structure is shown in Figure 7. This design is composed of three convolutional layers with the number of convolutional kernels similar to those reported in previous studies (4, 6, 6). Moreover, the size of the convolutional kernel was fixed at 3 × 3. The step size was 1 and the max-pooling was used. The window size was fixed at 2 × 2. The output feature dimension of the full connection layer was 4 and the ReLU (rectified linear activation unit) was selected as the activation function. CNN parameters in each layer are shown in Table 1. The 2-D gesture angle data collected by six sensors, with a size of 50 × 12, were input into the network. To prevent the loss of the boundary data in the process of convolution, the boundary is filled with a numerical value of 180.
| Layer | The number ofconvolution kernels | Window size | Input | Output |
| Convolution layer C1 | 4 | 3×3 | (50, 12, 1) | (50, 12, 4) |
| Pooling layer P1 | / | 2×2 | (50, 12, 4) | (25, 6, 4) |
| Convolution layer C2 | 6 | 3×3 | (25, 6, 4) | (25, 6, 6) |
| Pooling layer P2 | / | 2×2 | (25, 6, 6) | (12, 3, 6) |
| Convolution layer C3 | 6 | 3×3 | (12, 3, 6) | (12, 3, 6) |
| Pooling layer P3 | / | 2×2 | (12, 3, 6) | (6, 2, 6) |
DownLoad:
CSV
1200 sets of original data and 1200 sets of enhanced data from 10 participants comprised of different genders and ages are used for training. The data is described in a more detailed way in section 5 and the original data is publicly available on GitHub and will be updated continuously (the link is provided in SUPPLEMENT). Based on the signal waveforms, the differences in signal results of the different participants corresponded to the differences in the frequency or intensity. Numerically, the differences are in the data continuity and the size of the data. Hence, some methods, such as the frequently used scale, crop, translation, and Gaussian noise [34], increase with image dataset expansion and are effective for the data in this paper.
For FEM and post-stage CM, the input data were normalized to improve the convergence speed of training [35]. The mean normalization formula [36] was used for the FEM and CM inputs with [−1, 1] interval as shown in Eq 7.
| x'=x−mean(x)max(x)−min(x)×2 | (7) |
The multilayer perceptron NN framework is suited for classification in CM (the hardware implementation is explained in section 4.3). The input and output of NN were set according to the CNN output requirements as shown in Figure 8. Four inputs connected with the FEM output and the final output results corresponded to the recognized gesture number. Next, some parameters of the NN were optimized and different frameworks were designed. They were tested 20 times using different datasets. This analysis revealed that an increase in the number of individual layers or changes in the number of neurons in each NN framework layer had minimal influence on the accuracy [26]. Hence, two hidden layers and eight neurons in each hidden layer were selected.
A new activation function was introduced based on the different layers with different activation functions. Although the traditional ReLU activation function has sparse activation characteristics, the effect of nonlinear fitting data on sparse activation is reduced in the case of a few network layers. Moreover, the response boundary of the ReLU activation function is linear. The ReLU activation function also has a lower fitting ability compared to the Tanh activation function. Thus, ReTanh, a novel activation function, was designed based on the characteristics of the Tanh and ReLU activation functions as shown in Eq 8. The function image is shown in Figure 9.
| ReTanh=max(0,Tanh(x)) | (8) |
The results of the test accuracy corresponding to the different activation functions used in different layers are shown in Table 2. Note that the last output layer must select ReLU as the activation function. The result shows that the effect of the first layer with the ReLU activation function is poor. This is because, first, the unilateral activation is a linear function, making it hard to fit the complex features under fewer layers. Second, the output range of data cannot be specified twice, resulting in uncontrollable data size and non-convergence in training. Hence, we chose Tanh, ReTanh, and ReLU as the activation functions.
| Activation function | The average accuracy | |
| Hidden Layer 1 | Hidden Layer 2 | |
| ReTanh | ReTanh | 92.23% |
| ReTanh | ReLU | 93.40% |
| ReTanh | Tanh | 92.59% |
| Tanh | ReTanh | 97.15% |
| Tanh | Tanh | 95.56% |
| Tanh | ReLU | 96.89% |
| ReLU | ReTanh | 89.24% |
| ReLU | Tanh | 91.39% |
| ReLU | ReLU | 87.32% |
DownLoad:
CSV
The analysis of the various NN operation modes in the FEM and CM found that the data between network layers are interrelated and that the calculation in the layer only depends on the input data of the previous layer. Thus, each operation unit in the layer is independent. Currently, most neural network operations are based on the central processing unit (CPU). However, the serial computing mode of the CPU cannot give full play to the advantages of a parallel computing network. The development of deep learning has gradually expanded the network structure. For the CPU-based neural network, accelerating the operation by increasing the clock frequency alone would inevitably increase power consumption, but the acceleration effect would not be ideal. Thus, the recently proposed accelerators based on the FPGA, GPU (graphics processing unit), and an ASIC [37,38,39] should be used to improve the NN performance. The FPGA-based accelerator is more attractive due to its good performance, high-energy efficiency, fast development period, and reconfigurability. Accelerator design can be based on the FPGA hardware and the Cortex-M0 software algorithm introduced in section 3 to minimize resource consumption.
In the following subsections, details about how we simultaneously achieve low-power, low-latency, and high-accuracy on this edge device are explained: (a) how we de-noised data to improve recognition accuracy, (b) how we effectively combine the serial and parallel methods to save resource consumption and improve calculation efficiency, (c) how we schedule the order of calculations to make the best use of the limited resource.
The introduction of PM in our approach effectively reduces noise in the downstream processing and the amount of data processed, and improves the recognition efficiency of subsequent models. The core of the PM lies in the implementation of the wavelet de-noising, while the SWAB algorithm can be easily implemented as previously described in section 3. The wavelet de-noising is based on the quadrature mirror filter banks. The formulas for calculating the filter coefficient are shown in Eqs 9-12 [40,41].
| G(z)=(1+a[0]z−1−a[0][1]z−2+a[1]z−3)s | (9) |
| (z)=(−a[1]−a[0]a[1]z−1−a[0]z−2+z−3)s | (10) |
| ˆG(z)=H(−z) | (11) |
| ˆH(z)=−G(−z) | (12) |
To facilitate the FPGA calculation, s is 0.483, a[0] is 1.732, and a[1] is −0.268 in the equations. Thus, the above structure can be directly realized through the FPGA as shown in Figure 10. The signals are divided into two parts by the filter banks in Figure 10.a. One part with a high frequency-coefficient and the other with a low-frequency coefficient. The noise is reduced by changing the weight of the high-frequency coefficient.
The main entities in the FEM include convolutions and max-pooling that constitutes the CNN. This part introduces the FPGA hardware implementation of CNN. In the convolution operation, the whole idea is serial input and serial-parallel hybrid calculations. For max-pooling, serial input and parallel calculations are adopted. In this way, resource consumption is saved and calculation efficiency is greatly improved.
Since line-buffer is a common way to image processing [42,43], we proposed register-based line-buffers to implement the convolution operation of 3 × 3 window as shown in Figure 11.a. The weight value corresponding to the current convolution is obtained through the corresponding preset address in the read-only memory (ROM), which allows up to six convolutions to be executed simultaneously. The realization of pooled sampling is similar to convolution implementation. The sample values are obtained systematically through three comparators using the serial data for the local caching as shown in Figure 11.b. These two processes ensure the matching of data transmission, matching of window movement, and performance of each module. This design saves both the data cache capacity and minimizes the consumption of on-chip resources. The implementation of the fully connected layer is similar to the implementation of the NN in the subsequent CM. This is further discussed in the next section.
The CM is a typical multilayer perceptron with multiple interconnected neurons. While the resource is quite limited, we make the best use of it by dividing and combining the order of calculations of the eight multipliers, in short, we adopted a pipeline-multiplexing structure. This structure is consistent with the fully connected module in the FEM. Regardless of the Tanh or ReTanh function used in the first stage, the corresponding function of the core Tanh is not directly implementable by the FPGA. Here, we used the smoothing interpolation combined with the lookup-table method to fit the activation function [44]. Because the input for the negative Tanh function interval is negative, its absolute value is substituted to the positive interval for calculation. Hence, the negative interval results are the opposite value of the positive interval results. This operation saves the FPGA resources. The function relationship can be fitted by the (n-1) polynomial.
Let xj=xi(j=1,2,…n). The corresponding linear form of a polynomial of ny=∑ni=0aixi is shown in Eq 13.
| y=a0+n∑j=1ajxj | (13) |
The I = 1, 2, 3…, m experimental points satisfy xij=xji，They are substituted into the Eq 13 and then
| {ma0+n∑j=1(m∑i=1xij)aj=m∑i=1yim∑i=1xijxika0+n∑j=1(m∑i=1xijxik)aj=m∑i=1xikyi | (14) |
Thus, the polynomial is fitted by the least square method as follows.
| n∑i=1(m∑i=1xj+ki)aj=m∑i=1xikyik=0,1…n | (15) |
The equation is solved thus obtaining a0,a1…an.
Here, the positive interval is divided into [0, 1], (1, 2], (2, 3], (3, 4] and (4, +∞). The corresponding final fitting functions are listed in Table 3. The corresponding absolute error was calculated using MATLAB. The relationship between the fitting function and the original function is drawn by sampling as shown in Figure 12. The data best fit in the interval [0, 3]. Moreover, the error is larger when there are more data points. Nevertheless, this does not affect the results since the FPGA hardware calculation is implemented by shifting with 2 decimal places.
| Numerical Interval | Formula | Absolute Error |
| [0, 1] | y=−0.3275x2+1.0977x−0.0038 | 0.0038 |
| (1, 2] | y=−0.1690x2+0.7021x+0.2324 | 0.0039 |
| (2, 3] | y=−0.0282x2+0.1703x+0.7370 | 0.0055 |
| (3, 4] | y=−0.0039x2+0.0313x+0.9363 | 0.0101 |
| (4, +∞) | 1 | / |
DownLoad:
CSV
In this paper, eight multipliers are used in the hidden layer and the processing schedule is demonstrated in Figure 13. The weight and bias in the neurons were input into the corresponding neurons by the external storage for calculation. The data from each module of the previous level were serially input, while the entire neural network was still in a pipeline structure. For the eight multipliers in the hidden layer, operation with the four inputs in each neuron can be conducted simultaneously, except for the three multipliers needed by the activation functions in the first two layers. To complete the calculations in the whole neuron, multiple operation periods are needed in the hidden and output layer. To solve this, a cache should be used to store data between layers. It is also worth noting that, the whole operation was completed in the first hidden layer after eight cycles. For the second layer, it takes eight cycles to complete the operation in each neuron. Thereafter, all the multipliers are used to calculate the activation functions. The operation was completed in three cycles. The whole process takes 21 cycles.
In this section, experiments and discussions are conducted as evidence to support the superiority of the suggested system. In the following sub-sections, we first describe the datasets for experimentation, and then compare our algorithm with other past algorithms that can also be implemented on MCU-level platforms [45]. Finally, we focus on evaluating resource consumption, time performance, and power consumption on different hardware development platforms.
The algorithm proposed for the accelerator was first implemented and tested using python language to validate the concept with our database [46]. This dataset contains: (a) 1200 sets of original anonymous data of 10 different gestures, which are proudly offered by 10 volunteers, and (b) 1200 sets of enhanced data gained by translation, increasing noise, scaling the relative position and value of the original data.
For the original data, each data group consisted of 12 dimensions collected by 6 sensors (2 dimensions for each sensor). We labeled the data by strictly following the file-naming rule of a-b-c-d.txt. a represents the position of the sensor node, b represents the name of the gesture, c represents the dimension of the sensor data, d represents the collection times. The age and gender distribution of the volunteers and their finger-moving conditions (including moving stability and whether one is left- or right-handed) are shown in Tables 4 and 5.
| Age | |||||
| 11~20 | 21~40 | 41~60 | 61+ | ||
| Gender | Male | 1 | 2 | 2 | 1 |
| Female | 0 | 1 | 2 | 1 | |
DownLoad:
CSV
| Normal | With some sort of problems or instability | Left handedness |
| 8/10 | 2/10 (male, 41~60, chronic alcoholic, slightly shaking hands when relaxed; female, 21~40, has wound to the ring finger) | 3/10 (Male, 21~40, 41~60; Female, 41~60) |
DownLoad:
CSV
In this paper, KNN and SVM are selected as comparable classification algorithms with our CNN+NN model, as they conform to our objectives of deploying offline identification algorithm on an MCU-level development platform and then ASIC. These two supervised multiple classifiers are usually designed with hardware description language and then implemented on ASIC in past studies [47,48,49].
We divided the raw data into two groups - training data and testing data - at the ratio of 2:1 for the three algorithms. We then pinched, cropped, and added stochastic noise to the raw data to obtain 1200 sets of enhanced data. They are split to training and testing data still at the ratio of 2:1. Both raw data and enhanced data are tested for recognition accuracy.
The final results are shown in Table 6. Ideally, the accuracy of the three algorithms tested on raw data is close. However, when data are enhanced in various ways, CNN+NN has a much distinct accuracy at 95.12%, which proves its strong robustness, generalization ability, and suitability for real application scenarios.
| Classification algorithm | Accuracy of the raw data test | Accuracy of the enhanced data test |
| KNN | 99.09% | 89.1% |
| SVM | 98.88% | 85.12% |
| CNN+NN | 97.15% | 95.12% |
DownLoad:
CSV
The performance of an HW accelerator for the MCU can be divided into 3 categories: overall hardware utilization, time performance, and power consumption.
Here, we chose Intel’s DE10-Lite development board as the hardware development platform with 10M50DAF484C7G as the main chip. For low power consumption, this design uses a 55 nm CMOS technological process, 49760 LUTs, 1638 Kbit M9K memory, 144 18 × 18 multipliers, and 4 PLLs. Table 7 shows the re-source consumption of this design. It should be noted that the 144 18 × 18 multipliers in the MAX10 are recognized as 288 9 × 9 multipliers in the Quartus software.
| Module | LUTs | Registers | Memory bits | Multiplier elements |
| Cortex-M0 Kernel | 5427 | 1324 | 0 | 6 |
| AHB-Lite BUS | 148 | 46 | 0 | 0 |
| APB BUS | 125 | 24 | 0 | 0 |
| Peripherals | 576 | 495 | 16884 | 4 |
| Preprocessing Module | 1580 | 717 | 3360 | 6 |
| Feature Extraction Module | 3665 | 1014 | 6726 | 0 |
| Classification Module | 2240 | 2082 | 4571 | 16 |
| Total | 13769/49760 | 5702 | 31541/1667 | 32/288 |
DownLoad:
CSV
We chose a desktop processor, two mobile application processors, and the STM-MCU as the evaluation objects. We used a similar algorithm that was run on the Intel Core i7-9750H, Rockchip RK3399 Pro, Raspberry Pi 3B+, and STM32F407. As multiple groups of detection data were input into the FM to measure the operation time, the average values were recorded. The data for each platform are listed in Table 8.
| System on Chip | Architecture | Core Number | Frequency | Latency |
| Proposed | Cortex M0 | 1 | 50MHz | 1.544ms |
| Intel i7-9750H | Coffee Lake | 6 | 2.6GHz | 0.176ms |
| Rockchip RK3399 Pro | Cortex A72 | 6 | 2.0GHz | 1.398ms |
| Raspberry Pi 3B+ | Cortex A53 | 4 | 1.4GHz | 3.232ms |
| STM32F407 | Cortex M4 | 1 | 168MHz | 16.683ms |
DownLoad:
CSV
According to data in Table 8, in this specific scenario and in the range of MCUs, the proposed system has a significant performance, as M4 and M3 cannot even complete the calculations. Moreover, as an MCU, the proposed system outperforms some of the SBC. For example, its performance is more than twice as much of Cortex-A53 (which is usually used in Raspberry Pi), and is close to the high-performance of Cortex-A72.
With this astounding performance, referring to the results in Table 7, the resource consumption of the proposed system is only in line with M3. This accomplishment is of great importance for the design of ASIC and offline high-performance computing.
The coupling modes based on different accelerators are shown in Figure 14. Direct coupling to the AHB BUS was compared to that of the peripheral interface. Because the Cortex-M0 was transplanted to the FPGA through the IP core, only the SPI (Serial Peripheral Interface) and UART (Universal Asynchronous Receiver Transmitter) peripheral interfaces are discussed. Test comparison revealed that delay in the direct coupling and Cortex-M0 is the lowest, which is also determined by the coupling mode. However, for the SPI and UART, the main delay occurs in the data transmission process in the communication protocol of the two. Furthermore, the accelerator would become more universal in the way with a peripheral interface. Low power consumption and less resource consumption are critical factors in designing the accelerator for the MCU.
Since the neural network system architecture is recommended for the application-specific scenario, it is difficult to find the same architecture for comparison. Hence, the power consumption of the CNN and NN, the core part of the accelerator, are compared with those of the other parts (see Table 9). To achieve low power consumption, we adopted methods that minimize the power usage in the design, including the common gated clock and the Gray code encoding for the weight address bit.
| LUTs | Registers | Memory bits | DSP | Power/mW | ||
| CNN | Proposed | 3665 | 1014 | 6726 | 0 | 133.68 |
| [25] | 4901 | 2983 | 4800 | 0 | 380 | |
| [50] | 15285 | 2074 | 57100 | 564 | 607 | |
| [51] | 5717 | 6207 | 3900 | 20 | 370 | |
| Multilayer Perceptron NN | Proposed | 2240 | 2082 | 4571 | 8 | 124.48 |
| [26] | 2047 | / | / | 8 | 34.91 | |
| [52] | 1111 | 408 | 4000 | 6 | 183.27 | |
DownLoad:
CSV
Based on these data, the individual operation frequency of the acceleration module can reach 96 MHz after time sequence analysis and constraint operation. This suggests that the shorter lay is obtained as the accelerator combines with a certain buffer through a peripheral coupling. The FPGA in an Ultra-low-power level of Microsemi IGL00 with only 0.21 MW static power consumption has been previously used [26]. However, the static power consumption of MAX10, which is used here, exceeds 90 MW. Due to the restricted condition, the Cyclone 10LP was used to analyze the power consumption of the Cyclone 10LP chip series using the Power Analyzer Tool of the Quartus. Cyclone 10LP is Intel’s low-power FPGA, with a static power consumption of 30 MW only. Our proposed design is simulated by the Quartus and achieves a lower power consumption index. Therefore, this series of chips should be considered for further testing and applications.
In this paper, we propose a low-cost wearable edge device with a neural network accelerator based on the IMU sensor for gesture recognition. Firstly, the prototype glove is designed to locally collect data, process data, and complete the large volume data calculations off-line. Secondly, with the pre-stage processing module and serial-parallel hybrid method, the device is of low-power and low-latency. As an MCU that consumes at an MCU level, it performs eight times higher than the existing high-performance MCU and outperforms some SBCs. Thirdly, the whole system is a software-hardware co-design that is potentially transferrable to other scenarios. Moreover, a new activation function was designed for the multilayer perception neural network to improve recognition accuracy, and the feature extract process of CNN to complete classification is rather automatic that doesn’t need expert supervision in the process. Finally, all the data are open-sourced and will be continuously updated for other researchers for further use.
Our work promotes embedded systems and accessible edge computing models in the field of hand rehabilitation. However, we recognize that our framework presents four core limitations. The first is related to the amount and the type of hand gestures we tested. Since our intention was to make a technical prototype for locally recognizing medical and healing exercises for hands, we used only ten hand gestures for numbers instead of actual movements that hospitals are currently using. We will continuously work with the medical institute for future development. The second is that the whole device falls short in its undesirable size. It’s not comfortable enough for long-time wearing or in-the-wild uses. Moreover, the power consumption has not reached its lowest, because we didn’t use FPGA devices with the lowest power consumption. Last but not the least, further power analysis and design verification by ASIC tools are needed.
Future studies may focus on the three directions below: (a) Improvement in resource utilization and calculation performance, (b) ASIC implementation of the accelerator with Cortex-M0 processor core via EDA (Electronic Design Automation) tools optimization, (c) More specific and targeted recognition and evaluation schemes for different medical scenarios, such as sign language interpretation, finger movement rehabilitation after stroke and wrist movement evaluation after fracture, etc. Ultimately, we hope our work can be a good reference to improve the device accessibility, usability, versatility for different groups of users.
The authors would like to thank volunteers for participating in data collection, and VeriMake Research for providing developing and testing equipment. This research was partly funded by Industry-University-Research Collaboration Foundation of Fuzhou University grant number 0101/01011919.
The authors declare there is no conflict of interest.
| [1] |
Agostinelli D, Alouges F, DeSimone A (2018) Peristaltic waves as optimal gaits in metameric bio-inspired robots. Front in Robot AI 5: 99. doi: 10.3389/frobt.2018.00099
|
| [2] | Agostinelli D, Lucantonio A, Noselli G, et al. (2019) Nutations in growing plant shoots: The role of elastic deformations due to gravity loading. J Mech Phys Solids 2019: 103702. DOI: 10.1016/j.jmps.2019.103702. |
| [3] |
Agostiniani V, DeSimone A, Lucantonio A, et al. (2018) Foldable structures made of hydrogel bilayers. Mathematics in Engineering 1: 204-223. doi: 10.3934/Mine.2018.1.204
|
| [4] |
Aguilar J, Zhang T, Qian F, et al. (2016) A review on locomotion robophysics: The study of movement at the intersection of robotics, soft matter and dynamical systems. Rep Prog Phys 79: 110001. doi: 10.1088/0034-4885/79/11/110001
|
| [5] | Alberts B, Watson J, Lewis J, et al. (2014) Molecular Biology of the Cell, New York: Garland Science. |
| [6] |
Alouges F, DeSimone A, Giraldi L, et al. (2019) Energy-optimal strokes for multi-link microswimmers: Purcell's loops and Taylor's waves reconciled. New J Phys 21: 043050. doi: 10.1088/1367-2630/ab1142
|
| [7] |
Alouges F, DeSimone A, Lefebvre A (2008) Optimal strokes for low reynolds number swimmers: An example. J Nonlinear Sci 18: 277-302. doi: 10.1007/s00332-007-9013-7
|
| [8] |
Armon S, Efrati E, Kupferman R, et al. (2011) Geometry and mechanics in the opening of chiral seed pods. Science 333: 1726-1730. doi: 10.1126/science.1203874
|
| [9] |
Armon S, Yanai O, Ori N, et al. (2014) Quantitative phenotyping of leaf margins in three dimensions, demonstrated on knotted and tcp trangenics in arabidopsis. J Exp Bot 65: 2071-2077. doi: 10.1093/jxb/eru062
|
| [10] |
Arsuaga J, Vazquez M, McGuirk P, et al. (2005) Dna knots reveal a chiral organization of dna in phage capsids. P Natl Acad Sci 102: 9165-9169. doi: 10.1073/pnas.0409323102
|
| [11] | Aydin YO, Rieser JM, Hubicki CM, et al. (2019) 6-physics approaches to natural locomotion: Every robot is an experiment, In: Walsh S.M. and Stran M.S., Robotic Systems and Autonomous Platforms, Woodhead Publishing in Materials, 109-127. |
| [12] |
Bertinetti L, Fischer FD, Fratzl P (2013) Physicochemical basis for water-actuated movement and stress generation in nonliving plant tissues. Phys Rev Lett 111: 238001. doi: 10.1103/PhysRevLett.111.238001
|
| [13] |
Buskermolen AB, Suresh H, Shishvan SS, et al. (2019) Entropic forces drive cellular contact guidance. Biophys J 116: 1994-2008. doi: 10.1016/j.bpj.2019.04.003
|
| [14] |
Bustamante C (2017) Molecular machines one molecule at a time. Protein Sci 26: 1245-1248. doi: 10.1002/pro.3205
|
| [15] |
Bustamante C, Keller D, Oster G (2001) The physics of molecular motors. Accounts Chem Res 34: 412-420. doi: 10.1021/ar0001719
|
| [16] |
Cavagna A, Giardina I, Grigera TS (2018) The physics of flocking: Correlation as a compass from experiments to theory. Phys Rep 728: 1-62. doi: 10.1016/j.physrep.2017.11.003
|
| [17] | Charras G, Paluch E (2008) Blebs lead the way: How to migrate without lamellipodia. Nat Rev Mol cell bio 9: 730-736. |
| [18] |
Chauvet H, Moulia B, Legué V, et al. (2019) Revealing the hierarchy of processes and time-scales that control the tropic response of shoots to gravi-stimulations. J Exp Bot 70: 1955-1967. doi: 10.1093/jxb/erz027
|
| [19] |
Cheung KJ, Gabrielson E, Werb Z, et al. (2013) Collective invasion in breast cancer requires a conserved basal epithelial program. Cell 155: 1639-1651. doi: 10.1016/j.cell.2013.11.029
|
| [20] |
Cicconofri G, DeSimone A (2015) A study of snake-like locomotion through the analysis of a flexible robot model. P Roy Soc A Math Phys Eng Sci 471: 20150054. doi: 10.1098/rspa.2015.0054
|
| [21] |
Cicconofri G, DeSimone A (2019) Modelling biological and bio-inspired swimming at microscopic scales: Recent results and perspectives. Comput Fluids 179: 799-805. doi: 10.1016/j.compfluid.2018.07.020
|
| [22] |
Danson CM, Pocha SM, Bloomberg GB, et al. (2007) Phosphorylation of wave2 by map kinases regulates persistent cell migration and polarity. J Cell Sci 120: 4144-4154. doi: 10.1242/jcs.013714
|
| [23] | Darwin C (1880) The Power of Movement in Plants, London: John Murray. |
| [24] |
Del Alamo JC, Meili R, Alonso-Latorre B, et al. (2007) Spatio-temporal analysis of eukaryotic cell motility by improved force cytometry. P Natl Acad Sci 104: 13343-13348. doi: 10.1073/pnas.0705815104
|
| [25] |
Doostmohammadi A, Ignes-Mullol J, Yeomans JM et al. (2018) Active nematics. Nat Commun 9: 3246. doi: 10.1038/s41467-018-05666-8
|
| [26] |
Efrati E, Sharon E, Kupferman R (2009) Buckling transition and boundary layer in non-euclidean plates. Phys Rev E 80: 016602. doi: 10.1103/PhysRevE.80.016602
|
| [27] |
Efrati E, Sharon E, Kupferman R (2009) Elastic theory of unconstrained non-euclidean plates. J Mech Phys Solids 57: 762-775. doi: 10.1016/j.jmps.2008.12.004
|
| [28] |
Frangipane G, Dell'Arciprete D, Petracchini S, et al. (2018) Dynamic density shaping of photokinetic E. coli. Elife 7: e36608. doi: 10.7554/eLife.36608
|
| [29] |
Fratzl P, Barth FG (2009) Biomaterial systems for mechanosensing and actuation. Nature 462: 442-448. doi: 10.1038/nature08603
|
| [30] |
Geyer VF, Sartori P, Friedrich BM, et al. (2016) Independent control of the static and dynamic components of the chlamydomonas flagellar beat. Curr Biol 26: 1098-1103. doi: 10.1016/j.cub.2016.02.053
|
| [31] |
Giavazzi F, Malinverno C, Corallino S, et al. (2017) Giant fluctuations and structural effects in a flocking epithelium. J Phys D Appl Phys 50: 384003. doi: 10.1088/1361-6463/aa7f8e
|
| [32] |
Giavazzi F, Paoluzzi M, Macchi M, et al. (2018) Flocking transitions in confluent tissues. Soft matter 14: 3471-3477. doi: 10.1039/C8SM00126J
|
| [33] |
Giedroc DP, Cornish PV (2009) Frameshifting rna pseudoknots: Structure and mechanism. Virus Res 139: 193-208. doi: 10.1016/j.virusres.2008.06.008
|
| [34] |
Giomi L, Bowick MJ, Ma X, et al. (2013) Defect annihilation and proliferation in active nematics. Phys Rev Lett 110: 228101. doi: 10.1103/PhysRevLett.110.228101
|
| [35] |
Giomi L, Bowick MJ, Mishra P, et al. (2014) Defect dynamics in active nematics. Philos T Roy Soc A 372: 20130365. doi: 10.1098/rsta.2013.0365
|
| [36] |
Gladman AS, Matsumoto EA, Nuzzo RG, et al. (2016) Biomimetic 4d printing. Nat Mater 15: 413-418. doi: 10.1038/nmat4544
|
| [37] |
Goldstein RE, van de Meent JW (2015) A physical perspective on cytoplasmic streaming. Interface Focus 5: 20150030. doi: 10.1098/rsfs.2015.0030
|
| [38] |
Guasto JS, Rusconi R, Stocker R (2012) Fluid mechanics of planktonic microorganisms. Annu Rev Fluid Mech 44: 373-400. doi: 10.1146/annurev-fluid-120710-101156
|
| [39] |
Haas PA, Höhn SS, Honerkamp-Smith AR, et al. (2018) The noisy basis of morphogenesis: Mechanisms and mechanics of cell sheet folding inferred from developmental variability. PLoS Biol 16: e2005536. doi: 10.1371/journal.pbio.2005536
|
| [40] |
Hakim V, Silberzan P (2017) Collective cell migration: A physics perspective. Rep Prog Phys 80: 076601. doi: 10.1088/1361-6633/aa65ef
|
| [41] |
Hofhuis H, Moulton D, Lessinnes T, et al. (2016) Morphomechanical innovation drives explosive seed dispersal. Cell 166: 222-233. doi: 10.1016/j.cell.2016.05.002
|
| [42] |
Höhn S, Hallmann A (2011) There is more than one way to turn a spherical cellular monolayer inside out: Type B embryo inversion in volvox globator. BMC biol 9: 89. doi: 10.1186/1741-7007-9-89
|
| [43] |
Höhn S, Hallmann A (2016) Distinct shape-shifting regimes of bowl-shaped cell sheets-embryonic inversion in the multicellular green alga pleodorina. BMC Dev Biol 16: 35. doi: 10.1186/s12861-016-0134-9
|
| [44] |
Höhn S, Honerkamp-Smith AR, Haas PA, et al. (2015) Dynamics of a volvox embryo turning itself inside out. Phys Rev Lett 114: 178101. doi: 10.1103/PhysRevLett.114.178101
|
| [45] |
Hosoi A, Goldman DI (2015) Beneath our feet: Strategies for locomotion in granular media. Annu Rev Fluid Mech 47: 431-453. doi: 10.1146/annurev-fluid-010313-141324
|
| [46] |
Huss JC, Spaeker O, Gierlinger N, et al. (2018) Temperature-induced self-sealing capability of banksia follicles. J R Soc Interface 15: 20180190. doi: 10.1098/rsif.2018.0190
|
| [47] |
Ilievski F, Mazzeo AD, Shepherd RF, et al. (2011) Soft robotics for chemists. Angew Chem Int Edit 50: 1890-1895. doi: 10.1002/anie.201006464
|
| [48] |
Ishimoto K, Gadêlha H, Gaffney EA, et al. (2017) Coarse-graining the fluid flow around a human sperm. Phys Rev Lett 118: 124501. doi: 10.1103/PhysRevLett.118.124501
|
| [49] | Kawaguchi K, Kageyama R, Sano M (2009) Topological defects control collective dynamics in neural progenitor cell cultures. Nature 545: 327-331. |
| [50] |
Keller D, Bustamante C (2000) The mechanochemistry of molecular motors. Biophys J 78: 541-556. doi: 10.1016/S0006-3495(00)76615-X
|
| [51] |
Keller R, Davidson LA, Shook DR (2003) How we are shaped: The biomechanics of gastrulation. Differentiation 71: 171-205. doi: 10.1046/j.1432-0436.2003.710301.x
|
| [52] |
Khalil AA, Ilina O, Gritsenko PG, et al. (2017) Collective invasion in ductal and lobular breast cancer associates with distant metastasis. Clin Exp Metastas 34: 421-429. doi: 10.1007/s10585-017-9858-6
|
| [53] |
Kim J, Hanna JA, Hayward RC, et al. (2012) Thermally responsive rolling of thin gel strips with discrete variations in swelling. Soft Matter 8: 2375-2381. doi: 10.1039/c2sm06681e
|
| [54] |
Kim J, Hanna JA, Byun M, et al. (2012) Designing responsive buckled surfaces by halftone gel lithography. Science 335: 1201-1205. doi: 10.1126/science.1215309
|
| [55] |
Kim S, Laschi C, Trimmer B (2013) Soft robotics: A bioinspired evolution in robotics. Trends Biotechnol 31: 287-294. doi: 10.1016/j.tibtech.2013.03.002
|
| [56] |
Klein Y, Efrati E, Sharon E (2007) Shaping of elastic sheets by prescription of non-euclidean metrics. Science 315: 1116-1120. doi: 10.1126/science.1135994
|
| [57] |
Klein Y, Venkataramani S, Sharon E (2011) Experimental study of shape transitions and energy scaling in thin non-euclidean plates. Phys Rev Lett 106: 118303. doi: 10.1103/PhysRevLett.106.118303
|
| [58] | Laschi C, Mazzolai B (2016) Lessons from animals and plants: The symbiosis of morphological computation and soft robotics. IEEE Robot Autom Mag 23: 107-114. |
| [59] |
Latorre E, Kale S, Casares L, et al. (2018) Active superelasticity in three-dimensional epithelia of controlled shape. Nature 563: 203-208. doi: 10.1038/s41586-018-0671-4
|
| [60] |
Lauga E, Powers TR (2009) The hydrodynamics of swimming microorganisms. Rep Prog Phys 72: 096601. doi: 10.1088/0034-4885/72/9/096601
|
| [61] |
Lewis OL, Zhang S, Guy RD, et al. (2015) Coordination of contractility, adhesion and flow in migrating physarum amoebae. J R Soc Interface 12: 20141359. doi: 10.1098/rsif.2014.1359
|
| [62] |
Malinverno C, Corallino S, Giavazzi F, et al. (2017) Endocytic reawakening of motility in jammed epithelia. Nat Mater 16: 587-596. doi: 10.1038/nmat4848
|
| [63] |
Marchetti MC, Joanny JF, Ramaswamy S, et al. (2013) Hydrodynamics of soft active matter. Rev Mod Phys 85: 1143-1189. doi: 10.1103/RevModPhys.85.1143
|
| [64] |
Marenduzzo D, Micheletti C, Orlandini E, et al. (2013) Topological friction strongly affects viral dna ejection. P Natl Acad Sci 110: 20081-20086. doi: 10.1073/pnas.1306601110
|
| [65] |
Matt G, Umen J (2016) Volvox: A simple algal model for embryogenesis, morphogenesis and cellular differentiation. Dev Biol 419: 99-113. doi: 10.1016/j.ydbio.2016.07.014
|
| [66] |
Michor F, Liphardt J, Ferrari M, et al. (2011) What does physics have to do with cancer? Nat Rev Cancer 11: 657-670. doi: 10.1038/nrc3092
|
| [67] |
Moshe M, Sharon E, Kupferman R (2013) Pattern selection and multiscale behaviour in metrically discontinuous non-euclidean plates. Nonlinearity 26: 3247. doi: 10.1088/0951-7715/26/12/3247
|
| [68] |
Mueller R, Yeomans JM, Doostmohammadi A (2019) Emergence of active nematic behavior in monolayers of isotropic cells. Phys Rev Lett 122: 048004. doi: 10.1103/PhysRevLett.122.048004
|
| [69] |
Noselli G, Arroyo M, DeSimone A (2019) Smart helical structures inspired by the pellicle of euglenids. J Mech Phys Solids 123: 234-246. doi: 10.1016/j.jmps.2018.09.036
|
| [70] |
Noselli G, Beran A, Arroyo M, et al. (2019) Swimming euglena respond to confinement with a behavioural change enabling effective crawling. Nat Phys 15: 496-502. doi: 10.1038/s41567-019-0425-8
|
| [71] |
Olavarrieta L, Hernandez P, Krimer DB, et al (2002) Dna knotting caused by head-on collision of transcription and replication. J Mol biol 322: 1-6. doi: 10.1016/S0022-2836(02)00740-4
|
| [72] |
Palamidessi A, Malinverno C, Frittoli E, et al. (2019) Unjamming overcomes kinetic and proliferation arrest in terminally differentiated cells and promotes collective motility of carcinoma. Nat Mater 18: 1252-1263. doi: 10.1038/s41563-019-0425-1
|
| [73] |
Park JA, Kim JH, Bi D, et al. (2015) Unjamming and cell shape in the asthmatic airway epithelium. Nature Mater 14: 1040-1048. doi: 10.1038/nmat4357
|
| [74] |
Peng C, Turiv T, Guo Y, et al. (2016) Command of active matter by topological defects and patterns. Science 354: 882-885. doi: 10.1126/science.aah6936
|
| [75] |
Plesa C, Verschueren D, Pud S, et al. (2016) Direct observation of dna knots using a solid-state nanopore. Nature Nanotechnol 11: 1093-1097. doi: 10.1038/nnano.2016.153
|
| [76] |
Radszuweit M, Alonso S, Engel H, et al. (2013) Intracellular mechanochemical waves in an active poroelastic model. Phys Rev Lett 110: 138102. doi: 10.1103/PhysRevLett.110.138102
|
| [77] |
Renkawitz J, Kopf A, Stopp J, et al. (2019) Nuclear positioning facilitates amoeboid migration along the path of least resistance. Nature 569: 546-550. doi: 10.1038/s41586-019-1193-4
|
| [78] |
Rosa A, Di Ventra M, Micheletti C (2012) Topological jamming of spontaneously knotted polyelectrolyte chains driven through a nanopore. Phys Rev Lett 109: 118301. doi: 10.1103/PhysRevLett.109.118301
|
| [79] |
Rossi M, Cicconofri G, Beran A, et al. (2017) Kinematics of flagellar swimming in euglena gracilis: Helical trajectories and flagellar shapes. P Natl Acady Sci 114: 13085-13090. doi: 10.1073/pnas.1708064114
|
| [80] |
Sahaf M, Sharon E (2016) The rheology of a growing leaf: Stress-induced changes in the mechanical properties of leaves. J Exp Bot 67: 5509-5515. doi: 10.1093/jxb/erw316
|
| [81] |
Sanchez T, Chen DT, DeCamp SJ, et al. (2012) Spontaneous motion in hierarchically assembled active matter. Nature 491: 431-434. doi: 10.1038/nature11591
|
| [82] |
Sartori P, Geyer VF, Howard J, et al. (2016) Curvature regulation of the ciliary beat through axonemal twist. Phys Rev E 94: 042426. doi: 10.1103/PhysRevE.94.042426
|
| [83] |
Sartori P, Geyer VF, Scholich A, et al. (2016) Dynamic curvature regulation accounts for the symmetric and asymmetric beats of Chlamydomonas flagella. eLife 5: e13258. doi: 10.7554/eLife.13258
|
| [84] |
Saw TB, Doostmohammadi A, Nier V, et al. (2017) Topological defects in epithelia govern cell death and extrusion. Nature 544: 212-216. doi: 10.1038/nature21718
|
| [85] | Sharon E, Roman B, Marder M, et al. (2002) Mechanics: Buckling cascades in free sheets. Nature 419: 579. |
| [86] |
Shishvan SS, Vigliotti A, Deshpande VS (2018) The homeostatic ensemble for cells. Biomech Model Mechan 17: 1631-1662. doi: 10.1007/s10237-018-1048-1
|
| [87] |
Steinbock L, Radenovic A (2015) The emergence of nanopores in next-generation sequencing. Nanotechnology 26: 074003. doi: 10.1088/0957-4484/26/7/074003
|
| [88] |
Suma A, Micheletti C (2017) Pore translocation of knotted dna rings. P Natl Acad Sci 114: E2991-E2997. doi: 10.1073/pnas.1701321114
|
| [89] |
Suma A, Rosa A, Micheletti C (2015) Pore translocation of knotted polymer chains: How friction depends on knot complexity. ACS Macro Lett 4: 1420-1424. doi: 10.1021/acsmacrolett.5b00747
|
| [90] |
Tada M, Heisenberg CP (2012) Convergent extension: Using collective cell migration and cell intercalation to shape embryos. Development 139: 3897-3904. doi: 10.1242/dev.073007
|
| [91] |
Thampi SP, Golestanian R, Yeomans JM (2013) Velocity correlations in an active nematic. Phys Rev Lett 111: 118101. doi: 10.1103/PhysRevLett.111.118101
|
| [92] |
Thampi SP, Yeomans JM (2016) Active turbulence in active nematics. Eur Phys J Spec Top 225: 651-662. doi: 10.1140/epjst/e2015-50324-3
|
| [93] |
Tovo A, Suweis S, Formentin M, et al. (2017) Upscaling species richness and abundances in tropical forests. Sci Adv 3: e1701438. doi: 10.1126/sciadv.1701438
|
| [94] |
Vizsnyiczai G, Frangipane G, Maggi C, et al. (2017) Light controlled 3D micromotors powered by bacteria. Nat Commun 8: 15974. doi: 10.1038/ncomms15974
|
| [95] |
Wolf K, Mazo I, Leung H, et al. (2003) Compensation mechanism in tumor cell migration: mesenchymal-amoeboid transition after blocking of pericellular proteolysis. J Cell Biol 160: 267-277. doi: 10.1083/jcb.200209006
|
| [96] |
Yeh YT, Serrano R, François J, et al. (2018) Three-dimensional forces exerted by leukocytes and vascular endothelial cells dynamically facilitate diapedesis. P Natl Acad Sci 115: 133-138. doi: 10.1073/pnas.1717489115
|
| [97] |
Zhang AT, Montgomery MG, Leslie AGW, et al. (2019) The structure of the catalytic domain of the atp synthase from mycobacterium smegmatis is a target for developing antitubercular drugs. P Natl Acad Sci 116: 4206-4211. doi: 10.1073/pnas.1817615116
|
| [98] |
Zhang S, Guy RD, Lasheras JC, et al. (2017) Self-organized mechano-chemical dynamics in amoeboid locomotion of physarum fragments. J Phys D Appl Phys 50: 204004. doi: 10.1088/1361-6463/aa68be
|
| [99] |
Zhang S, Skinner D, Joshi P, et al. (2019) Quantifying the mechanics of locomotion of the schistosome pathogen with respect to changes in its physical environment. J R Soc Interface 16: 20180675. doi: 10.1098/rsif.2018.0675
|
Figures(4)
Daniele Agostinelli, Roberto Cerbino, Juan C. Del Alamo, Antonio DeSimone, Stephanie Höhn, Cristian Micheletti, Giovanni Noselli, Eran Sharon, Julia Yeomans. MicroMotility: State of the art, recent accomplishments and perspectives on the mathematical modeling of bio-motility at microscopic scales[J]. Mathematics in Engineering, 2020, 2(2): 230-252. doi: 10.3934/mine.2020011
| Layer | The number ofconvolution kernels | Window size | Input | Output |
| Convolution layer C1 | 4 | 3×3 | (50, 12, 1) | (50, 12, 4) |
| Pooling layer P1 | / | 2×2 | (50, 12, 4) | (25, 6, 4) |
| Convolution layer C2 | 6 | 3×3 | (25, 6, 4) | (25, 6, 6) |
| Pooling layer P2 | / | 2×2 | (25, 6, 6) | (12, 3, 6) |
| Convolution layer C3 | 6 | 3×3 | (12, 3, 6) | (12, 3, 6) |
| Pooling layer P3 | / | 2×2 | (12, 3, 6) | (6, 2, 6) |
DownLoad:
CSV
| Activation function | The average accuracy | |
| Hidden Layer 1 | Hidden Layer 2 | |
| ReTanh | ReTanh | 92.23% |
| ReTanh | ReLU | 93.40% |
| ReTanh | Tanh | 92.59% |
| Tanh | ReTanh | 97.15% |
| Tanh | Tanh | 95.56% |
| Tanh | ReLU | 96.89% |
| ReLU | ReTanh | 89.24% |
| ReLU | Tanh | 91.39% |
| ReLU | ReLU | 87.32% |
DownLoad:
CSV
| Numerical Interval | Formula | Absolute Error |
| [0, 1] | y=−0.3275x2+1.0977x−0.0038 | 0.0038 |
| (1, 2] | y=−0.1690x2+0.7021x+0.2324 | 0.0039 |
| (2, 3] | y=−0.0282x2+0.1703x+0.7370 | 0.0055 |
| (3, 4] | y=−0.0039x2+0.0313x+0.9363 | 0.0101 |
| (4, +∞) | 1 | / |
DownLoad:
CSV
| Age | |||||
| 11~20 | 21~40 | 41~60 | 61+ | ||
| Gender | Male | 1 | 2 | 2 | 1 |
| Female | 0 | 1 | 2 | 1 | |
DownLoad:
CSV
| Normal | With some sort of problems or instability | Left handedness |
| 8/10 | 2/10 (male, 41~60, chronic alcoholic, slightly shaking hands when relaxed; female, 21~40, has wound to the ring finger) | 3/10 (Male, 21~40, 41~60; Female, 41~60) |
DownLoad:
CSV
| Classification algorithm | Accuracy of the raw data test | Accuracy of the enhanced data test |
| KNN | 99.09% | 89.1% |
| SVM | 98.88% | 85.12% |
| CNN+NN | 97.15% | 95.12% |
DownLoad:
CSV
| Module | LUTs | Registers | Memory bits | Multiplier elements |
| Cortex-M0 Kernel | 5427 | 1324 | 0 | 6 |
| AHB-Lite BUS | 148 | 46 | 0 | 0 |
| APB BUS | 125 | 24 | 0 | 0 |
| Peripherals | 576 | 495 | 16884 | 4 |
| Preprocessing Module | 1580 | 717 | 3360 | 6 |
| Feature Extraction Module | 3665 | 1014 | 6726 | 0 |
| Classification Module | 2240 | 2082 | 4571 | 16 |
| Total | 13769/49760 | 5702 | 31541/1667 | 32/288 |
DownLoad:
CSV
| System on Chip | Architecture | Core Number | Frequency | Latency |
| Proposed | Cortex M0 | 1 | 50MHz | 1.544ms |
| Intel i7-9750H | Coffee Lake | 6 | 2.6GHz | 0.176ms |
| Rockchip RK3399 Pro | Cortex A72 | 6 | 2.0GHz | 1.398ms |
| Raspberry Pi 3B+ | Cortex A53 | 4 | 1.4GHz | 3.232ms |
| STM32F407 | Cortex M4 | 1 | 168MHz | 16.683ms |
DownLoad:
CSV
| LUTs | Registers | Memory bits | DSP | Power/mW | ||
| CNN | Proposed | 3665 | 1014 | 6726 | 0 | 133.68 |
| [25] | 4901 | 2983 | 4800 | 0 | 380 | |
| [50] | 15285 | 2074 | 57100 | 564 | 607 | |
| [51] | 5717 | 6207 | 3900 | 20 | 370 | |
| Multilayer Perceptron NN | Proposed | 2240 | 2082 | 4571 | 8 | 124.48 |
| [26] | 2047 | / | / | 8 | 34.91 | |
| [52] | 1111 | 408 | 4000 | 6 | 183.27 | |
DownLoad:
CSV
| Layer | The number ofconvolution kernels | Window size | Input | Output |
| Convolution layer C1 | 4 | 3×3 | (50, 12, 1) | (50, 12, 4) |
| Pooling layer P1 | / | 2×2 | (50, 12, 4) | (25, 6, 4) |
| Convolution layer C2 | 6 | 3×3 | (25, 6, 4) | (25, 6, 6) |
| Pooling layer P2 | / | 2×2 | (25, 6, 6) | (12, 3, 6) |
| Convolution layer C3 | 6 | 3×3 | (12, 3, 6) | (12, 3, 6) |
| Pooling layer P3 | / | 2×2 | (12, 3, 6) | (6, 2, 6) |
| Activation function | The average accuracy | |
| Hidden Layer 1 | Hidden Layer 2 | |
| ReTanh | ReTanh | 92.23% |
| ReTanh | ReLU | 93.40% |
| ReTanh | Tanh | 92.59% |
| Tanh | ReTanh | 97.15% |
| Tanh | Tanh | 95.56% |
| Tanh | ReLU | 96.89% |
| ReLU | ReTanh | 89.24% |
| ReLU | Tanh | 91.39% |
| ReLU | ReLU | 87.32% |
| Numerical Interval | Formula | Absolute Error |
| [0, 1] | y=−0.3275x2+1.0977x−0.0038 | 0.0038 |
| (1, 2] | y=−0.1690x2+0.7021x+0.2324 | 0.0039 |
| (2, 3] | y=−0.0282x2+0.1703x+0.7370 | 0.0055 |
| (3, 4] | y=−0.0039x2+0.0313x+0.9363 | 0.0101 |
| (4, +∞) | 1 | / |
| Age | |||||
| 11~20 | 21~40 | 41~60 | 61+ | ||
| Gender | Male | 1 | 2 | 2 | 1 |
| Female | 0 | 1 | 2 | 1 | |
| Normal | With some sort of problems or instability | Left handedness |
| 8/10 | 2/10 (male, 41~60, chronic alcoholic, slightly shaking hands when relaxed; female, 21~40, has wound to the ring finger) | 3/10 (Male, 21~40, 41~60; Female, 41~60) |
| Classification algorithm | Accuracy of the raw data test | Accuracy of the enhanced data test |
| KNN | 99.09% | 89.1% |
| SVM | 98.88% | 85.12% |
| CNN+NN | 97.15% | 95.12% |
| Module | LUTs | Registers | Memory bits | Multiplier elements |
| Cortex-M0 Kernel | 5427 | 1324 | 0 | 6 |
| AHB-Lite BUS | 148 | 46 | 0 | 0 |
| APB BUS | 125 | 24 | 0 | 0 |
| Peripherals | 576 | 495 | 16884 | 4 |
| Preprocessing Module | 1580 | 717 | 3360 | 6 |
| Feature Extraction Module | 3665 | 1014 | 6726 | 0 |
| Classification Module | 2240 | 2082 | 4571 | 16 |
| Total | 13769/49760 | 5702 | 31541/1667 | 32/288 |
| System on Chip | Architecture | Core Number | Frequency | Latency |
| Proposed | Cortex M0 | 1 | 50MHz | 1.544ms |
| Intel i7-9750H | Coffee Lake | 6 | 2.6GHz | 0.176ms |
| Rockchip RK3399 Pro | Cortex A72 | 6 | 2.0GHz | 1.398ms |
| Raspberry Pi 3B+ | Cortex A53 | 4 | 1.4GHz | 3.232ms |
| STM32F407 | Cortex M4 | 1 | 168MHz | 16.683ms |
| LUTs | Registers | Memory bits | DSP | Power/mW | ||
| CNN | Proposed | 3665 | 1014 | 6726 | 0 | 133.68 |
| [25] | 4901 | 2983 | 4800 | 0 | 380 | |
| [50] | 15285 | 2074 | 57100 | 564 | 607 | |
| [51] | 5717 | 6207 | 3900 | 20 | 370 | |
| Multilayer Perceptron NN | Proposed | 2240 | 2082 | 4571 | 8 | 124.48 |
| [26] | 2047 | / | / | 8 | 34.91 | |
| [52] | 1111 | 408 | 4000 | 6 | 183.27 | |
