A robust circular RNA-based prognostic signature for postoperative recurrence in stage II/III colon cancer

1.   Introduction

Support vector machines (SVMs) are robust computational tools for supervised learning, commonly employed in classification and regression tasks. With foundations in statistical learning theory and Bayesian principles, SVMs aim to identify an optimal separating hyperplane that maximizes the margin between positive and negative examples ^[1]. This approach has proven effective in diverse applications such as particle identification, face recognition, text categorization, and bioinformatics.

Expanding on the conventional SVM framework, Mangasarian and Wild ^[2] introduced the generalized eigenvalue proximal support vector machine (GEPSVM), which addresses binary classification by identifying two distinct hyperplanes, one for each category. The closest hyperplane is assigned to data points based on proximity, transforming the problem into a generalized eigenvalue formulation. The solutions are obtained using eigenvectors associated with the smallest eigenvalues.

To further enhance classification efficiency, Jayadeva et al. ^[3] proposed the twin support vector machine (TWSVM), which also constructs two non-parallel hyperplanes. Unlike GEPSVM, TWSVM involves solving two smaller quadratic programming problems (QPPs) instead of a single large-scale QPP. This design reduces computational complexity, as shown in experimental evaluations that demonstrate TWSVM's superior performance over GEPSVM and standard SVMs on datasets from the University of California, Irvine (UCI) machine learning repository ^[4].

Focusing on computational simplicity and scalability, Kumar and Gopal ^[5] developed the least squares twin support vector machine (LSTSVM) as an extension of TWSVM ^[6]. LSTSVM reformulates the primal QPPs of TWSVM using least squares principles, replacing inequality constraints with equality constraints. Consequently, the optimization reduces to solving two systems of linear equations, bypassing the need for external optimizers. This approach effectively accommodates nonlinear kernels while maintaining computational efficiency. Empirical comparisons across various UCI and artificial datasets confirm LSTSVM's faster training time and competitive classification accuracy relative to TWSVM and traditional LSTSVM.

Despite these advantages, traditional LSTSVM minimizes the $ L_2 $-norm of error variables, making it sensitive to outliers. This sensitivity can degrade classification accuracy, especially in noisy or imbalanced datasets. To address this, several enhancements have been proposed to improve the robustness of LSTSVM and related models. Furthermore, the standard LSTSVM reduces the coefficients of irrelevant features without eliminating any of them entirely. As a result, if the dataset contains many irrelevant features, using the standard LSTSVM may lead to a complex model with numerous included features since none of the irrelevant coefficients are reduced to zero.

Gao et al. ^[7] introduced the $ L_1 $-norm least squares twin support vector machine, which replaces the $ L_2 $-norm with the $ L_1 $-norm in the objective function to promote robustness and handle outliers effectively. This reformulation also promotes sparsity and feature suppression. By converting the constrained problem into an unconstrained convex quadratic form, they solve it efficiently using a generalized Newton method.

Yan et al. ^[8] proposed the $ L_1 $-norm-based least squares twin bounded support vector machine, replacing all conventional $ L_2 $-norms with $ L_1 $-norms to reduce outlier influence. The optimization problems are addressed through an iterative reweighting technique.

Wang et al. ^[9] introduced the robust capped $ L_1 $-norm twin support vector machine with privileged information, which incorporates the learning using privileged information framework. The capped $ L_1 $-norm enhances robustness, while upper and lower bound constraints on both main and privileged features control noise sensitivity. An alternating minimization approach is used to solve the optimization problems.

In a related line of work, the robust capped $ L_1 $-norm projection twin support vector machine (CPTSVM) was proposed to improve the outlier resistance of PTSVM models by Yang et al. ^[10]. By replacing the squared $ L_2 $-norm with the capped $ L_1 $-norm, the CPTSVM formulation increases classifier robustness in the presence of noise and outliers. Though the resulting problems are non-convex and non-smooth, an iterative algorithm with proven convergence is employed to solve them. Experiments on artificial and benchmark datasets demonstrate the model's robustness and effectiveness.

These related works inform the design of our proposed method, which integrates $ L_1 $-norm regularization into the LSTSVM framework and leverages the alternating direction method of multipliers (ADMM) ^[11,12,13] for scalable optimization. In contrast to prior methods, our approach introduces acceleration mechanisms and guard conditions to ensure both robustness and fast convergence on large and noisy datasets.

2.   Preliminaries

This section briefly reviews the fundamental concepts of TWSVM, LSTSVM, ADMM, and the Lasso technique.

2.1.   Twin support vector machine (TWSVM)

TWSVM is a classification technique developed to reduce the computational burden of traditional SVM. Instead of finding a single hyperplane to separate two classes, TWSVM constructs two non-parallel hyperplanes. Each hyperplane is positioned closer to one class while maintaining the maximum possible distance from the other. Given a dataset $ D $ with $ m_1 $ and $ m_2 $ training points labeled $ +1 $ and $ -1, $ respectively, in $ \mathbb{R}^n $, the data points for class $ +1 $ are represented by matrix $ A \in \mathbb{R}^{m_1\times n} $, while matrix $ B \in \mathbb{R}^{m_2\times n} $ represents class $ -1. $ The linear TWSVM is defined by:

and

where $ w_1, w_2 \in \mathbb{R}^n $ are normal vectors, and $ b_1, b_2 \in \mathbb{R} $ are bias terms. These hyperplanes are obtained by solving two separate optimization problems, each associated with one class:

and

where $ c_1, c_2 > 0 $ are parameters, $ {\mathbf \xi}_1 \in \mathbb{R}^{m_{1}} $, $ {\mathbf \xi}_2 \in \mathbb{R}^{m_{2}} $ are slack vectors, and $ e_1 $ and $ e_2 $ are vectors of ones of appropriate dimensions.

Using the Lagrangian dual method on (2.1) and (2.2), and introducing the Lagrange multipliers $ \alpha \in \mathbb{R}^{m_2} $ and $ \beta \in \mathbb{R}^{m_1} $, the resulting Wolfe dual formulations are:

where

and

And

where

and

Solving (2.3) and (2.4) yields the two non-parallel hyperplanes:

In comparison to SVM, the QPP in TWSVM has fewer parameters than that of SVM since (2.3) and (2.4) each involve only $ m_1 $ and $ m_2 $ parameters, unlike SVM's QPP, which depends on $ m_1+m_2 $ parameters.

In cases involving nonlinear data, the kernel functions are introduced, allowing the two hyperplanes of TWSVM in the kernel space to be represented as follows:

where $ K $ is a properly selected kernel,

and $ u_1, u_2 \in \mathbb{R}^n. $ The nonlinear TWSVM optimization problem can be expressed:

and

To compute the hyperplanes for nonlinear TWSVM, the dual forms of (2.7) and (2.8) are derived and solved to obtain the hyperplanes in (2.6). However, solving nonlinear TWSVM involves handling two QPPs and requires inverting two matrices of sizes $ (m_1 \times m_1) $ and $ (m_2 \times m_2) $, which becomes computationally demanding for large datasets. This adds computational complexity compared to the linear case.

2.2.   Least squares twin support vector machine (LSTSVM)

The LSTSVM is a binary classification technique that creates two non-parallel hyperplanes, which is proposed by Kumar and Gopal. LSTSVM addresses two modified primal QPPs from TWSVM, applying a least squares approach in which inequality constraints are replaced by equalities in (2.1) and (2.2) as:

and

where $ c_1, c_2 $ are the regularization parameters and $ \xi_1, \xi_2 $ are slack variables.

In (2.9), the QPP incorporates the $ L_2 $-norm of the slack variable $ \xi_2 $ with a weight of $ \frac{c_1}{2} $ rather than the $ L_1 $-norm weighted by $ c_1 $ as used in (2.1). This change renders the constraint $ \xi_2 \geq 0 $ unnecessary. As a result, solving (2.9) reduces to solving a system of linear equations. By substituting the equality constraints directly into the objective function, the problem is rewritten as:

Setting the gradient of (2.11) with respect to $ w_1 $ and $ b_1 $ to zero yields a closed-form solution for QPP (2.9):

where

and

Similarly, solving for the second hyperplane (2.10) gives:

Therefore, the two nonparallel hyperplanes of LSTSVM can be obtained by inverting two matrices of dimension $ (n+1) \times(n+1), $ where $ n $ is the number of features, which is significantly smaller than the total number of training samples. This makes LSTSVM more computationally efficient than TWSVM while also improving generalization. The nonlinear case follows the same approach, replacing linear terms with kernel functions.

2.3.   Alternating direction method of multipliers (ADMM)

ADMM was first introduced by Glowinski and Marroco ^[11] and Gabay and Mercier ^[12]. ADMM is an iterative optimization algorithm designed to decompose complex problems into manageable subproblems, which are then solved alternately. This approach ensures computational efficiency and makes ADMM particularly suitable for distributed and large-scale problems. ADMM solves optimization problems of the form:

where $ f $ and $ g $ are convex functions, and $ A $, $ B $, and $ c $ are given matrices/vectors. Since (2.14) is a constrained minimization problem, we can write the related augmented Lagrangian:

ADMM operates through a sequence of iterations

where $ \rho > 0 $ is the Lagrangian dual variable, which is also called the penalty parameter. The algorithm involves three key steps. First, an $ x $-minimization step optimizes $ x $ using the augmented Lagrangian function $ L_\rho $ while keeping $ z $ and the dual variable $ y $ fixed. Next, a $ z $-minimization step updates $ z $ similarly. Finally, the dual variable $ y $ is updated using a step size proportional to the augmented Lagrangian parameter $ \rho. $

The Lasso. Lasso regularization, which utilizes the $ L_1 $-norm, is an optimization and machine learning approach designed to reduce overfitting and promote sparsity in model parameters and is often solved using the ADMM because ADMM is well-suited for convex optimization problems with separable objective functions and constraints. The corresponding Lasso formulation is expressed as:

where $ y \in \mathbb{R}^n, X \in \mathbb{R}^{n \times p}, $ and $ \tau > 0 $ is a scalar regularization parameter that controls the strength of the penalty.

ADMM introduces an auxiliary variable $ z $ to separate the least squares and $ L_1 $ penalty terms:

ADMM solves this using the augmented Lagrangian:

where $ u $ is the scaled dual variable and $ \rho > 0 $ is the penalty parameter controlling convergence. ADMM then alternates between updating $ \beta, z, $ and $ u $:

(1) Update $ \beta $ (least squares step):

where $ \left(X^TX+\rho I\right) $ is always invertible, since $ \rho > 0. $

(2) Update $ z $ (soft-thresholding step):

where $ S_{\tau/\rho} $ is the soft-thresholding operator, which recall is defined as:

(3) Update $ u $ (dual variable update):

3.   Research methodology

Lasso LSTSVM by ADMM. Building on the foundational concepts discussed earlier, this section presents the formulation for solving LSTSVM with $ L_1 $-norm regularization, also known as the Lasso technique, using the ADMM framework. ADMM is chosen over traditional optimization techniques due to its strong scalability and its ability to decompose complex objectives—particularly those involving non-smooth terms like the $ L_1 $-norm—into simpler subproblems. This makes it especially effective for high-dimensional, large-scale datasets.

Linear case. In this work, we replace the $ L_2 $-norm of the penalty term in LSTSVM (2.9) and (2.10) with the $ L_1 $-norm, allowing the problem to be reformulated in a manner similar to the Lasso technique. This modification promotes sparsity in the slack variables and enables the use of ADMM to decompose the problem into simpler subproblems, improving computational efficiency. The modified optimization problems are formulated as follows:

and

where $ c_1, c_2 $ are given positive parameters.

To facilitate efficient computation, we reformulate the problems using auxiliary variables:

and

where

and

The augmented Lagrangian for the first reformulated problem (e.g., for $ x_1 $ and $ \xi_2 $) is given by:

where $ \rho > 0 $ serves as a penalty parameter and $ y_1 $ is the dual variable. The ADMM update rules for solving this problem are as follows:

$ (1) $ $ x_1 $-update: Solve for $ x_1^{(k+1)} $:

resulting in:

$ (2) $ $ \xi_2 $-update: Solve for $ \xi_2^{(k+1)} $:

which simplifies to the soft-thresholding operation:

where $ S_{\tau_1/\rho}(\cdot) $ is the soft-thresholding operator.

$ (3) $ Dual variable update: Update the dual variable $ y_1 $:

The iterative steps for solving the problem using ADMM are summarized in Algorithm 1:

Similarly, the second reformulated problem can be solved using the same concepts as the first problem, as previously demonstrated. The iterative steps for updating $ x_2, \xi_1, $ and $ y_2 $ are summarized in Algorithm 2:

Nonlinear case. In real-world scenarios, the linear kernel method is not always applicable, as large-scale datasets often exhibit higher complexity. Therefore, we extend the algorithm to nonlinear Lasso LSTSVM using kernel techniques ^[14]. We modify the optimization problems (3.1) and (3.2) as follows:

and

Where

is a kernel for any $ \textbf{x} $ and $ \textbf{y} $, where function $ \phi $ maps data points $ \textbf{x} $ from $ \mathbb{R}^n $ to $ \mathbb{R}^m (n < m). $

The model can be reformulated as the following:

and

where

and

The solution is similar to the linear kernel case: construct the augmented Lagrange function and follow the ADMM framework to update $ x, \xi $ and dual variables $ y. $

As for complexity, solving the linear system contributes $ O(n^3) $. Our proximal operator is simple. Its complexity is $ O(n) $. If $ k $ iterations are required, the total complexity is $ O(k \cdot (n^3+n)) $. For convex problems, $ k $ scales as $ O(1/\epsilon) $, so the total complexity becomes $ O\left(\frac{n^3}{\epsilon}\right). $ The criteria that ensures that our ADMM converges are the feasibility of the primal and dual. It is important to have feasibility in both primal and the dual (Lagrangian variables).

Accelerated Lasso LSTSVM with guard condition. In this work, we employ an accelerated ADMM framework ^[15] designed to ensure a consistent reduction in the combined residual $ \gamma $ by introducing a guard condition. This condition regulates the acceleration process, allowing it to proceed when met and reverting to the standard ADMM iteration otherwise. This adaptive mechanism prevents potential instability or divergence that might arise from direct acceleration.

Let $ u $ denote the target vector we aim to compute. The approximation of $ u $ at the $ k- $th iteration is represented by $ u^{(k)}. $ For clarity, $ \bar{u}^{(k)} $ represents the approximation of $ u $ computed using the standard ADMM method, while $ \hat{u}^{(k)} $ denotes the vector derived from an acceleration strategy applied to the ADMM process. In an acceleration step, the next iteration $ \hat{u}^{(k+1)} $ is calculated by combining current and previous approximations of $ \bar{u}, $ often as follows:

where $ \alpha^{(k)} $ is an adaptive parameter governing the acceleration. The proposed accelerated ADMM method, incorporating the guard condition, is formally outlined in Algorithm 3.

In our algorithm, the guard condition is designed to monitor the progress of the accelerated ADMM updates and decide whether to accept the acceleration or revert to standard ADMM updates. To make this decision robust and effective, we adopt the parameter selection strategy proposed in ^[15]. Specifically, the threshold parameter $ \eta \in (0, 1) $ is used to determine whether the accelerated update yields sufficient improvement in the combined residual. If the reduction is less than a factor of $ \eta, $ the algorithm rejects the acceleration step and reverts to the previous iterate.

In this work, we set $ \eta = 0.85, $ following the empirical recommendation in ^[15]. This choice reflects a balanced trade-off between acceleration and stability. Additionally, the momentum parameter $ \alpha^{(k)} $ is used to control the acceleration step size. Following the stationary acceleration approach proposed in ^[15], we fix

for all iterations, rather than using a dynamically updated rule. This simplification reduces computational overhead and avoids potential oscillations caused by increasing momentum.

In particular, ^[15] provides a convergence proof under the condition that

ensuring the stability of the accelerated scheme. Based on this, we select a fixed value of $ \alpha $ within this bound, e.g.,

to maintain theoretical convergence guarantees while benefiting from the improved convergence speed that acceleration offers.

This approach achieves faster convergence by leveraging the vectors computed from standard ADMM iterations as a reference for acceleration steps. Notably, selecting appropriate parameters for the guard condition is crucial to optimizing the method's overall efficiency.

4.   Results and discussion

This section presents a comparative study evaluating the effectiveness of the accelerated Lasso LSTSVM using ADMM with both linear and nonlinear kernels. We assess the classification accuracy and computational efficiency of the standard LSTSVM, Lasso LSTSVM, and its accelerated variant.

For experiments with linear kernels, we utilize datasets from the UCI machine learning repository ^[4], including ionosphere, breast cancer, Pima Indians, dry bean, satellite, predict students' dropout and academic success (PSDA), and QSAR biodegradation. For nonlinear classification, we employ EEG eye state and magic telescope (also from the UCI repository), along with the moon dataset—a synthetic dataset from Kaggle ^[16]—and the electricity dataset ^[17], a real-world dataset obtained from OpenML.

All datasets are designed for binary or multi-class classification tasks and span diverse domains such as medicine, physics, and social sciences. Their varying characteristics in terms of sample size, feature dimension, and complexity provide a robust foundation for evaluating the generalization performance of the proposed models across different scenarios.

As illustrated in Figure 1, a synthetic dataset was constructed to evaluate model performance on linearly separable data. The dataset consists of 500 samples with 2 features and 2 classes. Orange points represent Class 1, while blue points represent Class 2. To simulate challenging conditions, we introduced outliers by mislabeling 10% of the data—an intentionally high proportion, considering that outliers typically account for less than 5% of real-world datasets.

We then evaluated the models by splitting the dataset into 80% training and 20% testing data. The results show that Lasso LSTSVM achieved an accuracy of 94%, compared to 92% from the standard LSTSVM. This highlights the Lasso model's robustness to label noise and its ability to generalize better in the presence of significant outlier contamination.

The classification results are influenced by the selection of penalty parameters, which are set to specific values for linear and nonlinear kernels to ensure a fair comparison. For nonlinear classifiers, the Gaussian RBF kernel is utilized, with the kernel function defined as

with the kernel parameter $ \sigma = 1 $ due to its well-established capability to model complex, nonlinear relationships in high-dimensional feature spaces. It is a widely used choice in kernel-based learning methods because it introduces locality and smoothness in decision boundaries, which is beneficial for datasets with intricate structures, such as EEG eye state and magic telescope.

In our experiments, we fixed the kernel parameter to $ \sigma = 1 $ for consistency and to avoid additional hyperparameter tuning that could overshadow the performance comparison between models. However, we acknowledge that the performance of RBF-based models is sensitive to the choice of $ \sigma. $ A very small $ \sigma $ may lead to overfitting by capturing noise, while a large $ \sigma $ can oversmooth decision boundaries and underfit the data. The experiment results are summarized in Tables 1–4, where $ m $ and $ n $ indicate the number of training samples and feature dimensions, respectively.

The datasets used in this study exhibit varying degrees of outlier presence. Based on the interquartile range (IQR) method ^[18], most datasets were found to contain a moderate to high number of outliers. Notably, the Pima Indians and QSAR datasets have a particularly high concentration of outliers. For Pima Indians, this is primarily due to missing or implausible values treated as real numbers—features such as insulin, BMI, skin thickness, and blood pressure often contain zeros, which are physiologically impossible and act as placeholders for missing data but are not properly handled in the raw dataset. The QSAR dataset, on the other hand, naturally includes outliers due to its chemical diversity, high dimensionality, and non-normal feature distributions.

Table 1 compares the classification accuracy of the standard LSTSVM and the Lasso LSTSVM using a 10-fold cross-validation approach. Results show that Lasso LSTSVM outperforms the standard LSTSVM on datasets such as ionosphere, Pima Indians, PSDA, and QSAR, all of which contain relatively high levels of outliers. In contrast, for datasets like breast cancer, dry bean, and satellite, which exhibit fewer outliers, the performance gap between the models is narrower.

As shown in Table 2, aside from the moon dataset, which is synthetic and generated from Kaggle, the remaining three datasets (EEG eye state, magic telescope, and electricity) contain a considerable number of outliers. Interestingly, the standard LSTSVM performs better than the Lasso LSTSVM on the Moon dataset, while Lasso LSTSVM shows higher accuracy for EEG eye state and magic telescope. However, there are cases—such as the electricity dataset—where the standard LSTSVM still performs better despite the presence of outliers. This can be attributed to the nature of the outliers in this dataset, which result from natural fluctuations rather than sensor errors or data entry mistakes. In such cases, the $ L_2 $-norm of the standard LSTSVM may better capture the underlying data structure, whereas the $ L_1 $-norm regularization of Lasso LSTSVM might overly penalize these variations, potentially missing broader trends.

Tables 3 and 4 compare the computational time between the Lasso LSTSVM and the Accelerated Lasso LSTSVM models. The results clearly demonstrate that, for both linear and Gaussian RBF kernels, the accelerated Lasso LSTSVM significantly reduces computation time while maintaining classification accuracy comparable to that of the standard Lasso LSTSVM. This highlights the effectiveness of the proposed acceleration strategy, particularly for large-scale or high-dimensional datasets where computational efficiency is critical.

It is widely understood that in regression, ridge regression is used when many predictors or independent variables may be relevant, particularly in multicollinear contexts. In contrast, Lasso regression is employed when we suspect that only a few predictors are significant, allowing for effective feature selection. However, ridge regularization is more sensitive to outliers because it minimizes squared errors, which can lead to biased coefficients that are heavily influenced by those outliers. This principle applies to both the standard LSTSVM models and the Lasso LSTSVM models. Consequently, existing LSTSVM models are likely more sensitive to outliers than the Lasso LSTSVM models.

However, in some instances, the standard LSTSVM models outperformed the Lasso LSTSVM. This may be due to the standard LSTSVM's ability to handle a larger number of relevant predictors or independent variables more effectively. The standard LSTSVM shrinks the coefficients of irrelevant features without eliminating any coefficients entirely. As a result, if the dataset contains many irrelevant features, using the standard LSTSVM may lead to a complex model with numerous included features since none of the irrelevant coefficients are reduced to zero. In contrast, the standard LSTSVM might perform better than the Lasso LSTSVM when the dataset includes relevant independent variables, as shown in some of ours examples.

5.   Conclusions

This research presents a comparative analysis between three classification models: the standard LSTSVM, the Lasso LSTSVM, and the proposed accelerated Lasso LSTSVM, using the ADMM framework. The study compares the accuracy of the standard LSTSVM with Lasso LSTSVM and compares the computational time between Lasso LSTSVM and Accelerated Lasso LSTSVM. The inclusion of $ L1 $-norm regularization helps make the model more robust to outliers, enabling it to adapt well to datasets with a high number of outliers. Additionally, the acceleration step in the ADMM process helps reduce computation time without compromising classification accuracy.

Experimental results on several benchmark datasets, both linear and nonlinear, show that the proposed model outperforms the standard LSTSVM in terms of accuracy. Although there are cases where the standard LSTSVM achieves better accuracy than the Lasso type, this can be attributed to various factors revealed through analysis. However, there remains potential for improvement in computational efficiency, particularly for large-scale datasets. The accelerated model effectively reduces the number of iterations and training time compared to the non-accelerated version. Furthermore, the guard condition applied with the acceleration step ensures the algorithm's stability and guarantees reliable results. This study demonstrates that combining robustness to outliers with ADMM tuning and acceleration steps produces an efficient model that is well-suited for real-world data applications. Future research could extend this work by developing adaptive acceleration techniques and investigating theoretical convergence guarantees to achieve even faster and more reliable algorithms for practical use.

Author contributions

Thanasak Mouktonglang: conceptualization, methodology, validation, formal analysis, investigation, data curation, writing review and editing, supervision, funding acquisition; Rujira Fongmun: methodology, software, validation, investigation, data curation, writing original draft, writing review and editing. All authors have read and agreed to the published version of the manuscript.

Use of Generative-AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This research was partially supported by the Chiang Mai University, Faculty of Science, Chiang Mai University. The first author was supported by the Development and Promotion of Science and Technology Talents Project (DPST) scholarship.

Conflict of interest

The authors declare no conflicts of interest.