Early warning for critical transitions using machine-based predictability

1.   Introduction

Complex dynamical systems, including organisms, ecosystems, engineering systems and financial markets, often exhibit sudden transitions from a stable state to an alternative one with different features ^[2,14,15]. Such sharp regime shifts can occur unexpectedly with no explicit trigger, and therefore are hard to predict and manage.

It has been noted that systems near critical points have general properties regardless of differences in detail ^[21]. One of the most important clues is the so-called 'critical slowing down, ' which implies that the system becomes increasingly slow in recovering from small perturbations as it approaches such critical points ^[20]. The slowness of recovery in such natural perturbations near bifurcations leads to an increase in autocorrelation and variance in the fluctuating response ^[1,9]. Other conventional metrics such as spectral density and skewness, have also been used as early-warning indicators (EWIs) for critical transitions ^[2]. When applied to real systems, conventional EWIs often vary in reliability and fail to produce consistent results ^[5]. This implies that detecting critical transitions remains a daunting task, especially when we have insufficient knowledge of the system's transition-generating mechanisms.

In this paper, we propose the concept of predictability for time series data. While the specific value of predictability depends on the machine learning method we choose, it can be used as a simple and flexible EWI for a time series. Roughly put, the concept is based on the observation that complex systems become increasingly unpredictable as they approach tipping points. If we use a time-series predictor with a limited capacity, its prediction error likely grows larger and larger toward a critical point. On the contrary, the metric can also be used to detect the emergence of order for the systems that start to develop a higher level of patterns across a critical transition.

In the following, we present a definition of predictability as a metric throughout the time series and provide a methodological guide for evaluating it as a leading indicator for detecting critical transitions. In many dynamical systems, dramatic changes often occur from a time-dependent change of the parameter sets ^[12,16]. We have applied the proposed metric to the typical empirical models that develop critical transitions by parameter drifts, including a food-chain model, an electric power system and coupled chaotic oscillators. We compare the performance of the predictability to those of other EWIs, such as variance and autocorrelation.

2.   Machine-based predictability

This section introduces the concept of predictability of a time series. By the term predictor, we simply mean a procedure that derives an output time series from an input time series, which we want to naturally interpret as the future and past time series, respectively. For example, linear and quadratic regressions are two elementary predictors. We assume that the length of the input and the output time series can vary for a specific predictor. It is reasonable to expect that the longer the time series is fed to the predictor, the more accurate the prediction. However, such a proportional improvement only occurs in some range in practice due to the nature of the given time series, or due to the limited capacity of the predictor.

Suppose we have a sufficiently reliable predictor that can be used for a general time series. The concept of predictability for complex systems is based on the following hypothesis: the performance of the predictor changes as the system approaches critical transitions. In the situation where the predictor estimates the systems' future behavior with reasonable precision, the hypothesis implies that we will need more data to make a better prediction once the system begins to undergo a critical transition. If the predictor must use the same size input all along, its performance likely degrades near critical points. Considering that we always have limited resources for prediction in reality, the above statement is nothing but a rephrased version of "it is harder to detect critical states earlier than usual states." On the contrary, for systems that consist of many interacting factors, a critical transition induced by synchronization can dramatically change the system's global and long-term behavior into more predictable behavior. For example, the stock price is known to be notoriously hard to forecast with any degree of reliability. However, under some condition, investors in the market start to trade in the same direction and mimic the actions of others, eventually resulting in a large-scale pattern of soaring or crashing of the stock prices. The critical transitions toward synchronization like this can greatly raise the predictability of the system from what was effectively zero before.

For a formal definition of the predictability, let us reformulate the prediction of a time series as a supervised learning problem in machine learning. Define a prediction process $S$ as $S = (M, l_w, l_{tr})$, where $M$ is a predictor, $l_w > 0$ is the length of the sliding window and $l_{tr}$ is the length of the training set. Also, $l_{te} = l_w-l_{tr}$ is the length of the test set. Refer to Figure 1(a) and (b).

To define the predictability of a given time series at $t$, we apply the predictor $M$ to the prior part of the time series. Figure 1(a) shows that the time series in the sliding window $[t-l_{w}, t]$ is used to evaluate the metric. At each time $t$, the corresponding sliding window is split into two parts: the training set and the test set. The portions may vary with time. We use the time series in $[t-l_{w}, t-l_{w}+l_{tr}]$ to train the predictor, and then we have the predictor yield the future time series for the remaining part of the sliding window in $[t-l_{w}+l_{tr}, t]$. The prediction error, $E_S(t)$, can be assessed by comparing this result with the "real" time series from the test set in $[t-l_{w}+l_{tr}, t]$. Note that this error generally decreases as the portion of the training set increases (or, as the portion of the test set decreases, equivalently). Compare Figure 1(b) and (c). At each $t$, we basically want to adjust $l_{tr}$ (or $l_{te}$ equivalently) to keep the error $E_S(t)$ within the tolerance level, $tol$.

Let us define $l_{tr}^{*}$ and $l_{te}^{*}$ such that

The predictability, Pred$(t)$, is now defined as the maximum ratio of the test set to the sliding window that can be predicted within the error tolerance. That is,

For example, suppose the given data are given as a trivial constant time series. Any reasonable predictors can easily capture the constant behavior even from a small training set, that is, with $l_{tr}/l_{w} \ll 1$. This implies Pred$(t)\approx 1$ for the constant time series. Another extreme case is the white noise, which has no time-correlated pattern and therefore allows no meaningful prediction. In that case, no matter how much of the sliding window is used as the training set, the predictor fails to reproduce the time series. This means that we have Pred$(t)\approx 0$ for the white noise.

3.   ARIMA and ESN

While the concept of predictability is introduced to capture the change of the features of a time series, its validity is based on the assumption of unbiased performance of an adopted predictor. Such a requirement can be fulfilled by the most popular time-series predictors. In this work, we use two predictors, one from statistics and the other from machine learning. The autoregressive integrated moving average (ARIMA) is one of the most widely used statistical approaches to time-series forecasting; it incorporates autocorrelation to model temporal structures of the data. We evaluate the predictability using the ARIMA as the predictor in the first numerical example in Section 4. However, since the ARIMA assumes linear non-stationarity in the mean, it is not appropriate for forecasting complex nonlinear systems. We have adopted a more general machine learning method, i.e., echo state networks (ESNs), for other prediction tasks.

ESNs are a special type of recurrent neural network which has two modules: (i) a fixed network called a "reservoir" that reacts to an input nonlinearly and (ii) a trainable linear layer to connect such a response to the desired output. ESNs have been widely used for time-series prediction due to its simple and flexible architecture ^[11,17]. The following is a minimal explanation of the structure of the ESN used in this work. The readers who are more interested in details can refer to ^[6,10]. The ESN is defined as an evolutionary equation such that

where ${\textbf{r}}(k) \in {\mathbb{R}}^{N}$ is a state of the reservoir and ${\textbf{u}}(k) \in {\mathbb{R}}^{L}$ is an input vector. We use $N = 1000$ nodes and a leaking rate $\alpha = 0.1$ throughout all of the examples. ${\textbf{A}} \in {\mathbb{R}}^{N \times N}$ is an internal weight matrix (adjacency matrix between nodes) that is created with the pairwise connection probability, 0.02. ${\textbf{W}}_{in} \in {\mathbb{R}}^{N \times L}$ is an input weight matrix, and its entry is randomly generated from the uniform distribution over $[-0.085, 0.085]$.

Suppose a (training) data set of size $K$ is given as $({\textbf{u}}(1), {\textbf{y}}(1)), ({\textbf{u}}(2), {\textbf{y}}(2)), \cdots, ({\textbf{u}}(K), {\textbf{y}}(K))$, where ${\textbf{u}}(k) \in {\mathbb{R}}^{L_{in}}$ is an input vector and ${\textbf{y}}(k) = [y_{1}(k), y_{2}(k), \cdots y_{L_{out}}(k)]^\text{T} \in {\mathbb{R}}^{L_{out}}$ is a desired output vector for $1\leq k\leq K$. By driving the input series ${\textbf{u}}(1), {\textbf{u}}(2), \cdots {\textbf{u}}(K)$ into Eq. (3.1), we obtain ${\textbf{r}}(1), {\textbf{r}}(2), \cdots, {\textbf{r}}(K)$. Solving the minimization problems,

determines the output weight matrix ${\textbf{W}}_{out}\in {\mathbb{R}}^{L_{out} \times N} = [{\textbf{w}}_{1}, {\textbf{w}}_{2}, \cdots, {\textbf{w}}_{L_{out}}]^\text{T}$.

The matrix ${\textbf{W}}_{out}$ now enables us to approximate an output ${\textbf{y}}$ from an input ${\textbf{u}}$; we first evaluate ${\textbf{r}}$ in Eq (3.1) with respect to ${\textbf{u}}$ and then estimate ${\textbf{y}}$ by $\hat{{\textbf{y}}}$ as

Considering the evolutionary nature of Eq (3.1), an ESN is appropriate, especially for the highly correlated data set. In the examples in Section 4, we collect ${\textbf{u}}(1), {\textbf{u}}(2), \cdots, {\textbf{u}}(K)$ from the dynamics of the models and generate the training set by setting ${\textbf{y}}(k) = {\textbf{u}}(k+1), 1\leq k \leq K.$ Once the corresponding ${\textbf{W}}_{out}$ is obtained in Eq (3.2) as mentioned above, we repeatedly apply Eq (3.3) and generate predictions as $\hat{{\textbf{y}}}(K) = \hat{{\textbf{u}}}(K+1), \, \hat{{\textbf{y}}}(K+1) = \hat{{\textbf{u}}}(K+2), \cdots.$

We use the following error to assess the quality of the predictions in this study:

As a final practical remark, we added small multiplicative noise to the input in Eq (3.1) as follows:

which helps the ESN to avoid overfitting in computations ^[10,22].

4.   Results

In this section, we measure the predictability of the time series generated from various models with parameter drift, including stochastic and chaotic ones. Our goal is to detect an early sign of a sudden change from the predictability before it actually happens. In the following presentation of the numerical experiments, a green vertical dotted line indicates when the system parameter drifts across a bifurcation point. The moment when the system undergoes visible changes is marked with a red vertical dotted line. Note that the predictor in our examples observes the time series of (a part of) the system variables only and cannot access the parameters. At each moment, we use a sliding window that consists of 10,000 sampling data points and evaluate the predictability to the second decimal place. The error tolerance is $tol = 0.1$ for the first example, and $tol = 0.25$ for others. The red solid line is used to plot the moving average of 20 values of the predictability. We compare the predictability to that of other conventional EWIs, variance and autocorrelation, which are evaluated within the same sliding window. The time lag $t$ for autocorrelation is selected among $t = 1, 5, 10, 50$ and $100$, which gives the largest relative deviation in the graph of AR($t$) before the critical transition.

4.1.   Simple bifurcation with a stochastic force

We consider the simple grazing model ^[19,20]:

where $X$ is the biomass of the vegetation. The parameters $K, c$ and $\epsilon$ represent the carrying rate, maximum grazing rate and noise level, respectively. The first term on the right-hand side denotes the growth rate of the vegetation and the second term represents the rate of the total consumption rate. The third term, $\eta(t)$, stands for the stochastic force satisfying $\langle \eta(t) \rangle = 0$ and $\langle \eta(t) \eta(t') \rangle = \delta (t-t').$ The deterministic part of Eq (4.1) undergoes saddle-node bifurcations with respect to, $c$ as in Figure 2. We set $K = 10$ and focus on the "jump-down'' transition occurring at $c^* = 2.60437$.

Figure 3(a) shows the time-series data generated from Eq (4.1). The technical details of the time series are addressed in the caption. The model is known to have the critical slowing down as the parameter $c$ being drifted across a bifurcation point $c^*$ ^[20]. Such a phenomenon is captured in the rise of variance before the collapse, as in Figure 3(c). In order to evaluate the predictability Pred$(t)$, we have adopted ARIMA(0, 1, 1) as a predictor. One can see in Figure 3(b) that the predictability has a notable change in variance before the collapse. This suggests that for a simple system such as that given by Eq (4.1), both the predictability and variance can detect an early sign of collapse, while the autocorrelation fails to do so.

Complex dynamical systems in reality often involve more complicated transition mechanisms than simple bifurcations in the model given by Eq (4.1). Especially, chaotic systems can take some time to react to parameter shifts and the visible changes in the time series tend to follow with some delay. Such a transient period is "the critical window of opportunity'' in which the predictability metric is supposed to detect the structural change and issue an early-warning signal.

In the next three examples, we apply an ESN as a predictor to compute the predictability of complex dynamical systems. It is shown in ^[4,7,13] that an ESN has the ability to forecast the corresponding systems, and it can even estimate the critical values of the parameters if the values of the parameters are provided during the training. However, note again that the predictor in our examples uses a time series of the system variables only without information on the parameters.

4.2.   Food-chain model

Abrupt extinction of species from the earth has been occurring and is increasing rapidly, likely due to a shift in the parameters of the ecological systems. We consider a food-chain model of three species ^[18]:

Here $R, C$ and $P$ are the population densities of the resource, consumer and predator species, respectively. The values of the other parameters $x_c, y_c, x_p, y_p, R_0$ and $C_0$ were taken from ^[18] and are presented in the caption of Figure 4. $K$ is the environmental capacity of the resource species, which is the parameter that can induce a catastrophic bifurcation at the critical value $K = K_c = 0.99976$. A bifurcation diagram with respect to $K$ is shown in Figure 4. For a $K$ slightly greater than $K_c$, the food-chain model tends to show transient chaos leading to a sudden species extinction.

To generate a time series in $P$ that displays a sudden collapse, we slowly change $K$ across $K_c$ in the middle of the simulation. Figure 5(a) shows the resulting time series. The technical details of the time series are addressed in the caption. Note that the period of this transient chaos before collapse (between the green and red lines) is widely varied and sensitively depends on the systems' initial configuration.

Figure 5(b) shows the evolution of the predictability Pred$(t)$, as marked with blue circles. It fluctuates over time, likely due to the chaotic nature of the data. The predictability notably drops from 0.7 to 0.2 between the green and the red vertical dotted lines. This implies that the predictability successfully captures the structural change in the system before its effect explicitly appears as a local extinction in $P$. This contrasts with the variance and autocorrelation in Figure 5(c) and (d), respectively, both of which do not show many notable changes.

4.3.   Electric power system

In the electric power system, bifurcations linked with the onset of voltage collapse are of great interest for secure and stable operation. It is observed that the voltage's sudden collapse is often preceded by chaotic fluctuations for some duration. A possible explanation is that disturbances lead to a shift in the system parameters through the critical value ^[3], bringing the system into transient chaos, where it eventually settles down to a voltage collapse later. We consider the model of the electrical power system with voltage collapse in ^[3,23]:

where $\delta_m, \omega, \delta$ and $V$ are the generator voltage phase angle, the rotor speed, the load voltage phase angle and the voltage, respectively. $P$ and $Q$ are the real and reactive powers supplied by the load, and they are defined as

where

The values of all other constants were taken from ^[3,23] and are presented in the caption of Figure 6.

We take $Q_1$ as the main parameter since it can induce a catastrophic bifurcation in $V$ at the critical value $Q_1 = Q_{1c} = 2.9898256$. A bifurcation diagram with respect to $Q_1$ is shown in Figure 6. For a $Q_1$ slightly greater than $Q_{1c}$, the power system tends to show transient chaos leading to sudden voltage collapse.

To generate a time series in $V$ that displays a sudden collapse, we slowly change $Q_1$ across $Q_{1c}$ in the middle of the simulation. Figure 7 shows the resulting time series and the corresponding EWIs in a similar manner to the previous examples. Considering the complexity of the power system, we constructed an ESN based on a four-dimensional vector time series for $(\delta_m, \omega, \delta, V)$. Once again, the predictability in Figure 7(b) shows a notable drop from 0.7 to 0.2 between the green and the red vertical dotted lines, while the variance and autocorrelation in Figure 7(c) and (d) fail to report any essential changes. This implies that the predictability is functioning as a more reliable EWI for the electric power system.

4.4.   Coupled chaotic oscillators

The final example is the coupled chaotic oscillators, which undergo a critical transition as a result of synchronization. It is generally of interest to detect the onset of synchronization. The task is challenging in practical situations where only a part of the variables is available. It is, in essence, to detect the transitions between high-dimensional chaos and lower-dimensional chaos. The Lyapunov exponent is not a good indicator in this case, since its evaluation requires a large data set from both asynchronous and synchronous regimes. We consider the case of the system composed of coupled Rössler-like chaotic oscillators ^[8]:

where $\epsilon$ is the coupling coefficient, $a = 0.2, b = 0.2$ and $c = 5.7$. We assume that the only observable is the summation of two variables, i.e., $o(t) = x_1(t)+x_2(t)$, and the values of other variables are not available. While generating time series in $o(t)$, we have $\epsilon$ slowly drifted so that two oscillators synchronize at some point. While the value of $o(t)$ in Figure 8(a) seems to make no qualitative change, the graph of the phase difference $|x_1(t)-x_2(t)|$ in Figure 8(b) reveals that complete synchronization between two oscillators occurs at $t = 4280$ (marked with the green dotted line).

We evaluated the predictability of $o(t)$ in order to find out if it timely reacts to the structural change of the system (synchronization). One can see in Figure 8(c) that the synchronization that occurs at $t = 4280$ in the time series readily impacts the predictability. The reduction of complexity of the system leads to the rise of the predictability from what was effectively zero before. The phenomenon is the opposite of three previous examples and can be understood by considering the system escapes from an asynchronous state which is highly chaotic and does not allow reliable predictions for the predictor. It is obvious in Figure 8(d) that the variance is by far a crude way to capture such a transition.

5.   Discussion

In this paper, we presented a new approach to detect critical transitions in a time series from complex dynamical systems. Our approach is based on the metric called predictability which, at each time, indicates the level of precision of the prediction up to that moment made by a machine predictor with a fixed capacity. Different from other conventional EWIs that assume an underlying crisis-generating mechanism, the predictability rather takes a black-box approach. The metric indirectly shows the structural changes of the system in terms of how much predictability we gain or lose every moment. In other words, we argue that a change in predictability reflects a shift in the system parameters. Hence, it can be interpreted as an early-warning signal during the transient period until the structural change starts to build up into a more explicit form in the outcome.

While we were performing numerical experiments, we found that there is a subtle difference between two types of systems undergoing critical transitions. In a stochastic system with a simple transition-generating mechanism, the collapse immediately follows as the system parameter crosses a critical point. On the other side, complex systems tend to take some time to react to such a parameter shift. In either case, we observe the changes in the predictability to become clear before the collapse occurs, whether or not there is temporal discrepancy between the critical shift of the parameter and the actual collapse of the system.

The fluctuation of the predictability has been observed around the transitions, especially on the side of the higher predictable regime. That is, if the system undergoes a sudden collapse, then the predictability tends to fluctuate before the collapse occurs. If the system makes a sudden synchronization, the fluctuation starts to build up thereafter. This fluctuation may degrade the reliability of the predictability. However, we observed that the magnitude of the fluctuation depends not only on the nature of the time-series data, but also on the choice of a predictor, including its hyperparameters, such as the sliding window size and the tolerance level. We will leave this issue for future research.

The framework of predictability has flexibility in its implementation. Any general and model-free methods from statistics and machine learning can be used as a predictor. Some of those, like the ARIMA and ESN used in this work, can even be upgraded and sharpened in their ability to detect critical transitions by taking other EWIs as extra inputs. However, in essence, the crisis is what we cannot predict properly in advance. The predictability helps to capture such aspects of critical transitions and quantify them as a metric for the structural change.

Acknowledgments

P. Kim was supported by a National Research Foundation of Korea (NRF) grant funded by the South Korean government (MSIT) (No. NRF-2021R1A4A1032924).

Conflict of interest

There is no conflict of interest associated with any of the authors who contributed their efforts to this manuscript.