Assessment of a three-dimensional baroclinic circulation model of the Tagus estuary (Portugal)

1.   Introduction

In recent years, the software industry has expanded significantly owing to the increase in demand for software products for non-face-to-face services for various applications (conferences, education, delivery, shopping, etc.). The development and growth of software industries have led to digital innovation, which has increased the use of technology for both work and leisure ^[1,2,3,4]. For example, let us consider a store where more than 70% of the daily sales occur via card payments. If the card payment machine does not work due to internet failure, the delivery applications will not receive any processing applications, further affecting the sales severely. Internet failures can occur for various reasons, but users using the software will feel very uncomfortable. Like this, software has become an integral part of our lives. Various software programs are used in different industries, and it is difficult and complicated to develop related software. However, the main focus of software development must be to provide high-quality services to customers by improving reliability and stability.

Various methods are being used to improve software reliability. Among them, software testing is the most widely used method, and software testing is one of the methods to increase software reliability through a series of procedures (code, algorithm, optimization, etc.) to find and improve software faults ^[5,6]. In the past, software testing was defined as an activity of finding faults to confirm faults in order to check whether an application program or system operates normally and was nothing more than a business activity performed after development. However, in recent years, it refers to the entire process of making decisions through numerical data based on detected faults, such as the early stage of software development and the correction stage. As such, the number of software faults and the time interval between each fault have a significant impact on software reliability, and software reliability can be evaluated more easily by using a software reliability model, i.e., a mathematical model using the number of software faults and the time interval between faults. It can measure the number of software failures, software failure interval and software reliability. Also, as seen in the previous example for internet failure, software faults can occur for a number of reasons, including independent software faults such as code errors and software freezes, and dependent (secondary) causes that lead to other software faults due to code errors.

Many software reliability models (SRMs) have been developed from the past to the present, among the existing SRMs, the Goel-Okumoto (GO) ^[7] model is the most preferred model, which is based on the non-homogeneous Poisson process (NHPP). The GO model defines a mean value function (MVF) with an intensity function using an exponential distribution. In this model, the reliability during the mission time is determined by estimating the number of failures that occur while removing the faults remaining in the software. Since then, several researchers have extended the SRM based on the NHPP by, for example, indicating that the cumulative failure number of software failures increases in an S-shape by considering testing efforts or assuming imperfect debugging ^[8]. Pham and Zhang ^[9] proposed a generalized NHPP SRM in which the basic assumption is that the rate of change in the number of software faults is proportional to the content of the remaining faults. However, because the developed software has a very complex structure, one failure may affect other failures and thus increase the probability of causing another failure, causing software failures to occur dependently ^[10]. Pham and Pham ^[11] proposed an SRM with a dependency relationship assuming incomplete debugging. Lee et al. ^[12] proposed an SRM whereby dependent faults would be caused by prior software failures, and Kim et al. ^[13] proposed an SRM whereby software failures occur in a dependent manner. The previously proposed SRMs considered that, even if dependent faults occur, they occur constantly.

In recent decades, open-source software (OSS) development has been recognized and accepted by the industry and has gradually become an alternative name for software development. Many large companies, such as Microsoft, Google, Baidu and Alibaba, have their own OSS development projects. The development process for this type of software mainly involves the release of new versions of software or open-source projects by software developers, which enables the users to use and test the software to detect any faults in it during the usage process. The detected faults can be either removed by the users or sent to developers via email so that the developers can verify and remove such failures. Moreover, the effective evaluation of the reliability of OSS is a challenging problem. Various SRMs that utilize OSS have been developed. A reliability model used for effectively evaluating the reliability of OSS ^[14], a multi-release OSS reliability model with dependent fault detection ^[15], an OSS reliability growth model that considers change points ^[16] and an integrated OSS reliability model ^[17] have been used to evaluate the OSS in various ways.

In this study, when most software-dependent failures occur, dependent defects are considered on the assumption that the probability of rapidly increasing failures increases from the beginning, and, thereafter, the finite number of defects that result in failures occurs at a slow rate until the maximum number of failures is reached. Based on this, we propose a new SRM. In other words, we propose a new SRM that considers the number of finite and dependent defects. In addition, in order to see that the proposed SRM using defect data from OSS, which is a recent trend, is superior to the existing SRMs, the suitability is reviewed based on various criteria.

The MVF for the new SRM is derived in Section 2 by using the maximum number of fault contents in the software and the time-dependent fault detection rate function. The criteria for model comparisons and the selection of the best model are discussed in Section 3, and the results of each criteria value (CV) and model comparison are discussed in Section 4. Finally, in Section 5, conclusions and remarks are presented.

2.   Dependent SRM

2.1.   Generalized SRM of existing research

By assuming that software failure follows the NHPP, $N\left(t\right)$ ($t\ge 0$) is the Poisson probability density function with the parameter $m\left(t\right)$, where m(t) is the MVF, which is the expected number of faults detected at time $t$, and it can be expressed as follows:

The MVF $m\left(t\right)$ with the failure intensity $\lambda \left(t\right)$ is expressed as follows:

The reliability function R(t) of software representing the probability that a software error will not occur within the interval [0, t] is expressed as in Equation (1):

If t+x is given, then the software reliability can be expressed as a conditional probability R(x|t), as expressed in Equation (2).

The MVF $m\left(t\right)$ of the generalized NHPP SRM can be obtained by solving the differential equation expressed in Equation (3):

where $a\left(t\right)$ is the expected total number of software failures, and $b\left(t\right)$ is the fault detection rate function. Many existing NHPP SRMs have been developed on the premise that software faults occur independently (remove faults immediately if they occur). However, since the software is used in different environments, software faults occur very differently, depending on where the software is operated, and new faults may occur due to existing faults.

The MVF of the NHPP SRM considering dependent faults can be obtained using the differential equation (4) as shown below.

2.2.   Novel SRM with dependent faults

As discussed in Section 2.1, many existing SRMs consider that dependent errors persist, if they do occur. However, when most software-dependent faults occur, there is a high probability that faults will increase rapidly from the beginning, and, subsequently, they occur slowly until the maximum number of faults is reached.

In this study, we considered a finite number of software faults and dependent faults.

Thus, the MVF $m\left(t\right)$ can be obtained by solving the differential equation (5), with the initial condition $m\left(0\right)\ne 0$ ^[18]:

Here, $a\left(t\right)$ and $b\left(t\right)$ represent the maximum number of fault contents in the software and the time-dependent fault detection rate function, respectively.

In this study, the following $a\left(t\right)$ and $b\left(t\right)$ are considered:

We obtained a new NHPP SRM that considers a finite number of faults and dependent faults with the initial condition $m\left(0\right) = k$, which can be expressed as

3.   Criteria for model comparison and average value of criteria

3.1.   Existing NHPP SRMs for comparison

Table 1 summarizes the MVFs of existing NHPP SRMs and the proposed new NHPP SRM. Existing NHPP SRMs 8, 9, 10 and 11 consider the dependency.

Table 1.  Existing NHPP SRMs.

No.	Model	$m\left(t\right)$
1	GO [7]	$m\left(t\right)=a\left(1-{e}^{-bt}\right)$
2	HD-GO [19]	$\mathrm{m}(\mathrm{t})=\log \left[\left(\mathrm{e}^{\mathrm{a}}-\mathrm{c}\right) /\left(\mathrm{e}^{\mathrm{ae}-{\mathrm{bt}}}-\mathrm{c}\right)\right]$
3	Y-DS [20]	$m\left(t\right)=a\left(1-\left(1+bt\right){e}^{-bt}\right)$
4	O-IS [21]	$m\left(t\right)=\frac{a\left(1-{e}^{-bt}\right)}{1+\beta {e}^{-bt}}$
5	Y-Exp [22]	$m\left(t\right)=a\left(1-{e}^{-\gamma \alpha \left(1-{e}^{-\beta t}\right)}\right)$
6	Y-Ray [22]	$m\left(t\right)=a\left(1-{e}^{-\gamma \alpha \left(1-{e}^{-\beta {t}^{2}/2}\right)}\right)$
7	PZ-IFD [23]	$m\left(t\right)=a\left(1-{e}^{-bt}\right)\left(1+\left(b+d\right)t+bd{t}^{2}\right)$
8	P-DP 1 [24]	$m\left(t\right)=\alpha \left(\gamma t+1\right)$)
9	P-DP 2 [24]	$\begin{array}{l} m\left(t\right)={m}_{0}\left(\frac{\gamma t+1}{\gamma {t}_{0}+1}\right){e}^{-\gamma \left(t-{t}_{0}\right)} \\ \ \ \ \ \ \ \ \ \ \ \ \ +\alpha \left(\gamma t+1\right)\left(\gamma t-1+\left(1-\gamma {t}_{0}\right){e}^{-\gamma \left(t-{t}_{0}\right)}\right) \end{array}$
10	P-DP 3 [18]	$m\left(t\right)=\frac{a}{1+d\left(\frac{1+\beta }{\beta +{e}^{bt}}\right)}$
11	L-DP [12]	$m\left(t\right)=\frac{a}{1+\frac{a}{h}{\left(\frac{b+c}{c+b{e}^{bt}}\right)}^{\frac{a}{b}}}$
12	New model	$m\left(t\right)=\frac{a}{1+\left(\frac{a}{k}-1\right)\left(1+bt\right){e}^{-bt}}$

---

 | Show Table

DownLoad: CSV

3.2.   Criteria for model comparison

In order to compare the performance of the model, we use 12 criteria for model comparison. They are the mean squared error (MSE), the predicted relative variation (PRV), the root mean square prediction error (RMSPE), the sum of absolute errors (SAE), the mean absolute error (MAE), the mean error of prediction (MEOP), the Theil statistics (TS), the predictive ration risk (PRR), Pham's information criterion (PIC), Pham's criterion (PC), ${R}^{2}$ and the adjusted ${R}^{2}$ (${AdjR}^{2}$). The CVs are obtained by using the difference between the actual value and the predicted value. Some criteria are sensitive to outliers and others are not. This shows the superiority of a given model by comparing many scales without using one scale for various reasons.

In Table 2, ${y}_{i}$ and $\widehat{m}\left({t}_{i}\right)$ represent the total number of failures and the estimated cumulative number of failures, respectively. $n$ and m denote the total number of observations and the number of unknown parameters in the model, respectively. In Table 2, from 1–10, the smaller those values, the better the model performance (close to zero). From 11–12, the higher those values, the better the model performance (close to one).

Table 2.  Criteria for model comparison.

No.	Criteria	$function$
1	MSE	$MSE=\frac{{\sum }_{i=1}^{n}{\left(\widehat{m}\left({t}_{i}\right)-{y}_{i}\right)}^{2}}{n-m}$
2	PRV [25]	$PRV=\sqrt{\frac{{\sum }_{i=1}^{n}{\left({y}_{i}-\widehat{m}\left({t}_{i}\right)-Bias\right)}^{2}}{n-1}}$
3	RMSPE [26]	$RMSPE=\sqrt{{Variance}^{2}+{Bias}^{2}}$
4	SAE [27]	$SAE=\sum\limits_{i=1}^{n}\left|\widehat{m}\left({t}_{i}\right)-{y}_{i}\right|$
5	MSE [27]	$MAE=\frac{{\sum }_{i=1}^{n}\left|\widehat{m}\left({t}_{i}\right)-{y}_{i}\right|}{n-m}$
6	MEOP [28]	$MEOP=\frac{{\sum }_{i=1}^{n}\left|\widehat{m}\left({t}_{i}\right)-{y}_{i}\right|}{n-m+1}$
7	TS [29]	$TS=100*\sqrt{\frac{{\sum }_{i=1}^{n}{\left({y}_{i}-\widehat{m}\left({t}_{i}\right)\right)}^{2}}{{\sum }_{i=1}^{n}{{y}_{i}}^{2}}}{\%}$
8	PRR [30]	$PRR=\sum\limits_{i=1}^{n}{\left(\frac{\widehat{m}\left({t}_{i}\right)-{y}_{i}}{\widehat{m}\left({t}_{i}\right)}\right)}^{2}$
9	PIC [30]	$PIC=\sum\limits_{i=1}^{n}{\left(\widehat{m}\left({t}_{i}\right)-{y}_{i}\right)}^{2}+m\left(\frac{n-1}{n-m}\right)$
10	PC [30]	$PC=\left(\frac{n-m}{2}\right)log\left(\frac{{\sum }_{i=1}^{n}{\left(\widehat{m}\left({t}_{i}\right)-{y}_{i}\right)}^{2}}{n}\right)+m\left(\frac{n-1}{n-m}\right)$
11	${R}^{2}$ [31]	${R}^{2}=1-\frac{{\sum }_{i=1}^{n}{\left(\widehat{m}\left({t}_{i}\right)-{y}_{i}\right)}^{2}}{{\sum }_{i=1}^{n}{\left({y}_{i}-\overline{{y}_{i}}\right)}^{2}}$
12	Adj ${R}^{2}$ [31]	${AdjR}^{2}=1-\frac{\left(1-{R}^{2}\right)\left(n-1\right)}{n-m-1}$

---

 | Show Table

DownLoad: CSV

The variance and bias are given:

3.3.   Average of the normalized criteria and ranking of model

We described 12 criteria in Section 3.1 to compare the performance of NHPP SRMs. Since there are various comparison criteria, it is difficult to select a criterion first and check the performance of the SRM. Therefore, the criterion method to integrate them is needed. Therefore, a new integrated criterion was proposed by considering the value and ranking of each criterion. Earlier, Li and Pham ^[31] described a criterion for ranking the best models by using the distance of the regularization criterion method.

Because the CVs and rankings for model comparison are different, we have proposed an integrated comparison criterion method that considers the average of each criterion and the average of the rankings of each criterion. The average value of the normalized criteria and ranking (AC value) of the model is defined as follows:

where $s$ is the total number of models; ${C}_{ij}$ denotes the criterion value of the ith model of the jth criterion, where $i = \mathrm{1, 2}, \dots, s$; $d$ is the total number of certain criteria (MSE, PRV, RMSPE, MAE, MEOP, TS, PIC and PC); $f$ is the total number of the remaining criteria (${R}^{2}$ and ${AdjR}^{2}$). $k = 1$ denotes the $d$ and $f$ values of the criteria, and $k = 2$ denotes the $d$ and $f$ rankings of the criteria.

4.   Numerical examples

4.1.   Data information

We downloaded all of the fixed issues for products, namely, Apache IoTDB (IoTDB, database for Internet of Things) of the Apache open-source project from the Apache issue tracking system (https://iotdb.apache.org/). Bugs (Bugs), new features (NFs) and feature improvements (IPMs) have been presented with different symbols. Only those issues that did not duplicate and were reproducible for others were selected. Briefly, the Apache IoTDB is an integrated data management engine designed for time-series data. Additional information can be found at https://iotdb.apache.org/. Data were collected for different issue types on a monthly basis from January 2019 to January 2022. The proposed model only considered dependent faults; therefore, we did not consider the independent issue, but reflected the dependent issue and constructed the dataset, as listed in Table 3. In Table 3, Dataset 1 has a cumulative number of failures of 3, 5, ..., 260 at t = 1, 2, ..., 36. Datasets 2–4 have the cumulative number of failures $\mathrm{2, 2}, \dots, 353$; $\mathrm{2, 5}, \dots, 377$; $\mathrm{2, 5}, \dots, 495$, at $t = \mathrm{1, 2}, \dots, 37$, respectively.

Table 3.  Information of datasets.

Dataset	Index	Explanation
1	Months	Sum of Bugs and NFs
2	Months	Sum of Bugs and IPMS
3	Months	Sum of NFs and IPMs
4	Months	Sum of total issues (Bugs + NFs + IPMs)

---

 | Show Table

DownLoad: CSV

4.2.   Results of parameter estimation for models and comparison

We estimated the parameters of all models listed in Table 1 for Datasets 1–4 based on the least square estimation method using MATLAB (version 2021a) and R (version 4.2.2) programs; the parameter estimates of model are listed in Table 4. Tables 5–8 list the criteria obtained using the estimated parameters listed in Table 4; the best value for each criterion is indicated in bold font.

Table 4.  Parameter estimation of model for Datasets 1–4.

No.	Model	Dataset 1	Dataset 2	Dataset 3	Dataset 4
1	GO	$\begin{array}{c} \widehat{a}= 15579.370\\ \widehat{b}= 0.000237\end{array}$	$\begin{array}{c} \widehat{a}= 30074.804\\ \widehat{b}= 0.00016\end{array}$	$\begin{array}{c} \widehat{a}=26543.88 \\ \widehat{b}= 0.00023\end{array}$	$\begin{array}{c} \widehat{a}= 35704.462 \\ \widehat{b}= 0.00020\end{array}$
2	HD-GO	$\begin{array}{c} \widehat{a}=709.783 \\ \widehat{b}=0.005496 \\ \widehat{c}=1.27898 \end{array}$	$\begin{array}{c} \widehat{a}=709.783 \\ \widehat{b}=0.00745 \\ \widehat{c}=1.5662 \end{array}$	$\begin{array}{c} \widehat{a}=709.783 \\ \widehat{b}=0.00954 \\ \widehat{c}=0.7354 \end{array}$	$\begin{array}{c} \widehat{a}=709.783 \\ \widehat{b}=0.01151 \\ \widehat{c}=175.9297 \end{array}$
3	Y-DS	$\begin{array}{c} \widehat{a}=141419.032 \\ \widehat{b}=0.001433 \end{array}$	$\begin{array}{c} \widehat{a}=113361.757 \\ \widehat{b}=0.00182 \end{array}$	$\begin{array}{c} \widehat{a}=92734.061 \\ \widehat{b}=0.00223 \end{array}$	$\begin{array}{c} \widehat{a}=206592.161 \\ \widehat{b}=0.00163 \end{array}$
4	O-IS	$\begin{array}{c} \widehat{a}=265191.291 \\ \widehat{b}=0.129382 \\ \widehat{\beta }=108796.118 \end{array}$	$\begin{array}{c} \widehat{a}=55395.720 \\ \widehat{b}=0.11982 \\ \widehat{\beta }=13622.8014 \end{array}$	$\begin{array}{c} \widehat{a}=31054.349 \\ \widehat{b}=0.08524 \\ \widehat{\beta }=1964.09 \end{array}$	$\begin{array}{c} \widehat{a}=41668.713 \\ \widehat{b}=0.10680 \\ \widehat{\beta }=4506.2851 \end{array}$
5	Y-Exp	$\begin{array}{c} \widehat{a}=6651.999 \\ \widehat{\alpha }=13.4697 \\ \widehat{\beta }=0.000097 \\ \widehat{\gamma }=0.426491 \end{array}$	$\begin{array}{c} \widehat{a}=23542.721 \\ \widehat{\alpha }=0.2863 \\ \widehat{\beta }=0.000015 \\ \widehat{\gamma }=48.3835 \end{array}$	$\begin{array}{c} \widehat{a}=9574.09 \\ \widehat{\alpha }=4.3317 \\ \widehat{\beta }=0.0000081 \\ \widehat{\gamma }=18.2711 \end{array}$	$\begin{array}{c} \widehat{a}=8894.355 \\ \widehat{\alpha }=0.0063 \\ \widehat{\beta }=0.000229 \\ \widehat{\gamma }=571.5637 \end{array}$
6	Y-Ray	$\begin{array}{c} \widehat{a}=18930.468 \\ \widehat{\alpha }=0.06186 \\ \widehat{\beta }=0. 000004\\ \widehat{\gamma }=60.355006 \end{array}$	$\begin{array}{c} \widehat{a}=7937.291 \\ \widehat{\alpha }=0.5458 \\ \widehat{\beta }=0.0000014 \\ \widehat{\gamma }=42.4752 \end{array}$	$\begin{array}{c} \widehat{a}=5198.476 \\ \widehat{\alpha }=1.9516 \\ \widehat{\beta }=0.0000009 \\ \widehat{\gamma }=48.8568 \end{array}$	$\begin{array}{c} \widehat{a}=10748.070 \\ \widehat{\alpha }=2.4634 \\ \widehat{\beta }=0. 0000163 \\ \widehat{\gamma }=1.2584 \end{array}$
7	PZ-IFD	$\begin{array}{c} \widehat{a}=1.427 \\ \widehat{b}=0.017955 \\ \widehat{d}=4.621768 \end{array}$	$\begin{array}{c} \widehat{a}=1.410 \\ \widehat{b}=0.01901 \\ \widehat{d}=5.6567 \end{array}$	$\begin{array}{c} \widehat{a}=4.145 \\ \widehat{b}=0.01651 \\ \widehat{d}=2.6920 \end{array}$	$\begin{array}{c} \widehat{a}=3.282 \\ \widehat{b}=0.01787 \\ \widehat{d}=3.7908 \end{array}$
8	P-DP1	$\begin{array}{c} \widehat{\alpha }=125.5344 \\ \widehat{\gamma }=0.03869 \end{array}$	$\begin{array}{c} \widehat{\alpha }=172.8202 \\ \widehat{\gamma }=0.0374 \end{array}$	$\begin{array}{c} \widehat{\alpha }=266.7805 \\ \widehat{\gamma }=0.0334 \end{array}$	$\begin{array}{c} \widehat{\alpha }=261.0303 \\ \widehat{\gamma }=0.0369 \end{array}$
9	P-DP2	$\begin{array}{c} \widehat{\alpha }=244.7919 \\ \widehat{\gamma }=0.029573 \\ \widehat{{t}_{0}}=9.839177 \\ \widehat{{m}_{0}}=1.5332 \end{array}$	$\begin{array}{c} \widehat{\alpha }=330.5071 \\ \widehat{\gamma }=0.0288 \\ \widehat{{t}_{0}}=16.1134 \\ \widehat{{m}_{0}}=32.0163 \end{array}$	$\begin{array}{c} \widehat{\alpha }=247.5648 \\ \widehat{\gamma }=0.0346 \\ \widehat{{t}_{0}}=4.7037 \\ \widehat{{m}_{0}}=2.7852 \end{array}$	$\begin{array}{c} \widehat{\alpha }=428.8474 \\ \widehat{\gamma }=0.0301 \\ \widehat{{t}_{0}}=9.3036 \\ \widehat{{m}_{0}}=6.5048 \end{array}$
11	P-DP3	$\begin{array}{c} \widehat{a}=2613.915 \\ \widehat{b}=0.156904 \\ \widehat{\beta }=8.2482 \\ \widehat{d}=285.0215 \end{array}$	$\begin{array}{c} \widehat{a}=529633.047 \\ \widehat{b}=0.13081 \\ \widehat{\beta }=2.0695 \\ \widehat{d}=64815.3057 \end{array}$	$\begin{array}{c} \widehat{a}=4946.888 \\ \widehat{b}=0.10157 \\ \widehat{\beta }=0.5079 \\ \widehat{d}=372.2134 \end{array}$	$\begin{array}{c} \widehat{a}=69831.344 \\ \widehat{b}=0.11404 \\ \widehat{\beta }=0.1537 \\ \widehat{d}=8676.8881 \end{array}$
10	L-DP	$\begin{array}{c} \widehat{a}=1792.175 \\ \widehat{b}=0.00494 \\ \widehat{c}=71.7287 \\ \widehat{h}=2.2845 \end{array}$	$\begin{array}{c} \widehat{a}=5034.263 \\ \widehat{b}=0.00033 \\ \widehat{c}=13.2127 \\ \widehat{h}=3.3307 \end{array}$	$\begin{array}{c} \widehat{a}=1966.294 \\ \widehat{b}=0.00152 \\ \widehat{c}=30.1387 \\ \widehat{h}=9.7228 \end{array}$	$\begin{array}{c} \widehat{a}=2485.048 \\ \widehat{b}=0.00004 \\ \widehat{c}=0.8902 \\ \widehat{h}=5.8794 \end{array}$
12	New	$\begin{array}{c} \widehat{a}=87694025.6 \\ \widehat{b}=0.1622 \\ \widehat{k}=5.1345 \end{array}$	$\begin{array}{c} \widehat{a}=899987.667 \\ \widehat{b}=0.15304 \\ \widehat{k}=7.9320 \end{array}$	$\begin{array}{c} \widehat{a}=1835.516 \\ \widehat{b}=0.13584 \\ \widehat{k}=17.2829 \end{array}$	$\begin{array}{c} \widehat{a}=658742.756 \\ \widehat{b}=0.14264 \\ \widehat{k}=15.3712 \end{array}$

---

 | Show Table

DownLoad: CSV

Table 5.  CVs for model comparison on Dataset 1.

No.	Model	MSE	PRV	RMSPE	SAE	MAE	MEOP	PRR	TS	PIC	PC	R2	Adj R2
1	GO	1749.0858	39.7658	41.1806	1160.8790	34.1435	33.1680	11.6572	46.0613	59538.9169	128.0236	0.6312	0.6089
2	HD-GO	1918.5570	41.2864	42.4974	1186.6781	35.9599	34.9023	12.2289	47.5265	63364.8815	126.4751	0.6074	0.5706
3	Y-DS	606.9791	23.4641	24.2601	650.4427	19.1307	18.5841	459.0569	27.1343	20707.2882	110.0315	0.8720	0.8643
4	O-IS	36.0128	5.5591	5.8198	165.6282	5.0190	4.8714	119.988	6.5114	1240.9221	60.8801	0.9926	0.9919
5	Y-Exp	1868.2746	39.9998	41.2932	1157.2915	36.1654	35.0694	11.6917	46.1835	59831.4523	123.0148	0.6293	0.5815
6	Y-Ray	630.7619	23.1734	23.9915	643.7168	20.1161	19.5066	479.2857	26.8349	20231.0471	105.6413	0.8748	0.8587
7	PZ-IFD	542.3169	21.8688	22.5922	599.3037	18.1607	17.6266	472.5235	25.2682	17948.9582	105.6277	0.8890	0.8786
8	P-DP1	454.8300	20.3569	21.0017	550.7308	16.1980	15.7352	1072.91	23.4885	15534.2202	105.1258	0.9041	0.8983
9	P-DP2	425.5850	19.7243	19.7257	553.6523	17.3016	16.7773	288.4161	22.0424	13665.3862	99.3459	0.9156	0.9047
10	P-DP3	32.6231	5.3460	5.4582	159.8638	4.9957	4.8444	2.6718	6.1028	1090.6046	58.2508	0.9935	0.9927
11	L-DP	33.0958	5.4048	5.4982	140.5448	4.3920	4.2589	1.9926	6.1469	1105.7307	58.4809	0.9934	0.9926
12	New	24.5575	4.8076	4.8118	114.1580	3.4593	3.3576	0.7847	5.3770	862.8971	54.5629	0.9950	0.9945

---

 | Show Table

DownLoad: CSV

Table 6.  CVs for model comparison on Dataset 2.

No.	Model	MSE	PRV	RMSPE	SAE	MAE	MEOP	TS	PRR	PIC	PC	R2	Adj R2
1	GO	2908.3079	51.1206	53.1200	1533.2673	43.8076	42.5908	44.0904	13.0524	101862.7752	140.6529	0.6561	0.6359
2	HD-GO	3288.0496	54.0665	55.6817	1593.5985	46.8705	45.5314	46.2061	13.8233	111847.6852	139.4058	0.6223	0.5880
3	Y-DS	935.3733	29.0943	30.1279	785.6199	22.4463	21.8228	25.0044	100.3033	32810.0655	120.8012	0.8894	0.8829
4	O-IS	55.8942	7.0373	7.2595	200.3325	5.8921	5.7238	6.0244	11.7649	1954.4024	70.1378	0.9936	0.9930
5	Y-Exp	3087.2469	51.2011	53.1445	1531.5532	46.4107	45.0457	44.1096	13.0542	101927.1487	135.0539	0.6558	0.6128
6	Y-Ray	973.7958	28.7239	29.8466	780.6954	23.6574	22.9616	24.7731	104.5938	32183.2611	116.0157	0.8914	0.8779
7	PZ-IFD	813.8860	26.6544	27.6965	711.3360	20.9216	20.3239	22.9885	110.3514	27726.1230	115.6699	0.9065	0.8980
8	P-DP1	676.7092	24.7943	25.6270	639.2695	18.2648	17.7575	21.2679	244.5784	23756.8221	115.1364	0.9200	0.9153
9	P-DP2	626.2336	23.9577	23.9592	653.5447	19.8044	19.2219	19.8662	239.2390	20713.7086	108.7313	0.9302	0.9215
10	P-DP3	46.1444	6.4682	6.5028	202.5889	6.1391	5.9585	5.3927	3.7242	1570.7650	65.7002	0.9949	0.9942
11	L-DP	53.8884	6.9832	7.0271	193.2495	5.8560	5.6838	5.8277	1.6682	1826.3172	68.2600	0.9940	0.9932
12	New	41.6728	6.2699	6.2735	192.7933	5.6704	5.5084	5.2018	3.3775	1470.8747	65.1464	0.9952	0.9948

---

 | Show Table

DownLoad: CSV

Table 7.  CVs for model comparison on Dataset 3.

No.	Model	MSE	PRV	RMSPE	SAE	MAE	MEOP	TS	PRR	PIC	PC	R2	Adj R2
1	GO	2287.4481	44.9701	47.1005	1377.519	39.3577	38.2644	33.0486	8.4481	80132.683	136.4505	0.7719	0.7584
2	HD-GO	2788.2933	49.6130	51.2713	1478.073	43.4727	42.2306	35.9627	9.3023	94855.972	136.6031	0.7298	0.7053
3	Y-DS	388.0583	19.1464	19.4162	468.6592	13.3903	13.0183	13.6121	91.6956	13654.041	105.4049	0.9613	0.9590
4	O-IS	99.0515	9.6336	9.6710	238.4523	7.0133	6.8129	6.7782	1.8966	3421.7495	79.8649	0.9904	0.9895
5	Y-Exp	2445.8270	45.1553	47.2919	1383.307	41.9184	40.6855	33.1829	8.4914	80760.2899	131.2112	0.7700	0.7412
6	Y-Ray	408.4608	19.1300	19.3441	461.4523	13.9834	13.5721	13.5605	97.1207	13527.2073	101.6804	0.9616	0.9568
7	PZ-IFD	311.0582	17.0022	17.1362	388.9742	11.4404	11.1135	12.0117	84.0768	10629.9796	99.3187	0.9699	0.9671
8	P-DP1	246.5689	15.4693	15.4825	355.2497	10.1500	9.8680	10.8504	229.1409	8701.9111	97.4684	0.9754	0.9740
9	P-DP2	261.0926	15.4687	15.4704	363.6438	11.0195	10.6954	10.8417	760.0981	8664.0564	94.2963	0.9754	0.9724
10	P-DP3	124.6118	10.5118	10.6830	327.8676	9.9354	9.6432	7.4900	3.2155	4160.1880	82.0917	0.9883	0.9868
11	L-DP	119.1020	10.4128	10.4478	296.8038	8.9941	8.7295	7.3225	2.3600	3978.3649	81.3455	0.9888	0.9874
12	New	141.9586	11.4880	11.5765	358.5639	10.5460	10.2447	8.1145	3.5573	4880.5910	85.9831	0.9862	0.9850

---

 | Show Table

DownLoad: CSV

Table 8.  CVs for model comparison on Dataset 4.

No.	Model	MSE	PRV	RMSPE	SAE	MAE	MEOP	TS	PRR	PIC	PC	R2	Adj R2
1	GO	5149.9542	67.9784	70.6857	2052.4628	58.6418	57.0129	40.3687	11.0273	180320.3972	150.6527	0.6949	0.6770
2	HD-GO	6215.2439	74.6340	76.5627	2184.1801	64.2406	62.4051	43.7097	12.1963	211372.2939	150.2299	0.6424	0.6099
3	Y-DS	1383.9061	35.5586	36.6507	964.0537	27.5444	26.7793	20.9265	60.6451	48508.7130	127.6563	0.9180	0.9132
4	O-IS	133.4095	11.0178	11.2193	293.6791	8.6376	8.3908	6.4039	6.0626	4589.9226	84.9272	0.9923	0.9916
5	Y-Exp	5531.7658	68.3112	71.1327	2071.6176	62.7763	60.9299	40.6255	11.1302	182596.2704	144.6772	0.6911	0.6524
6	Y-Ray	1461.7003	35.3994	36.5725	964.3966	29.2241	28.3646	20.8831	62.4636	48284.1104	122.7172	0.9184	0.9082
7	PZ-IFD	1183.7263	32.5196	33.4115	854.8952	25.1440	24.4256	19.0754	62.6249	40300.6951	122.0382	0.9319	0.9257
8	P-DP1	956.5578	29.8019	30.4771	748.3795	21.3823	20.7883	17.3980	153.2968	33551.5228	121.1931	0.9433	0.9400
9	P-DP2	927.9433	29.1631	29.1652	770.7883	23.3572	22.6702	16.6390	197.9840	30670.1293	115.2199	0.9482	0.9417
10	P-DP3	114.1997	10.2304	10.2314	268.6371	8.1405	7.9011	5.8371	1.6640	3816.5911	80.6520	0.9936	0.9928
11	L-DP	158.4102	11.9760	12.0483	299.3494	9.0712	8.8044	6.8748	1.1627	5275.5382	86.0515	0.9912	0.9900
12	New	115.5631	10.3995	10.4459	305.9183	8.9976	8.7405	5.9602	3.0198	3983.1460	82.4859	0.9934	0.9927

---

 | Show Table

DownLoad: CSV

In Table 5, we can observe that the MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PRR, PIC and PC values for the proposed model were the lowest, and that the R² and Adj R² values for the proposed model were the largest among all models. As listed in Table 5, the values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PRR, PIC and PC for the proposed model were 24.5575, 4.8076, 4.8118,114.1580, 3.4593, 3.3576, 0.7847, 5.3770,862.8971 and 54.5629, respectively, which are lower. The proposed models for the values of ${R}^{2}$ and Adj ${R}^{2}$ were 0.9950 and 0.9945, respectively, which are higher. Next, the P-DP3 model's MSE, PRV, RMSE, TS, PIC and PC values were 32.6231, 5.3460, 5.4582, 6.1028, 1090.6046 and 58.2508, respectively, which are the second lowest. Moreover, the P-DP3 model's values of R² and Adj R² were 0.9935 and 0.9927, respectively, which are the second largest. The SAE, MAE, MEOP and PRR values for the L-DP model were 140.5448, 4.3920, 4.2589 and 1.9926, respectively, which are the second lowest. Figure 1 depicts graphical representations of the MVFs for all models based on Dataset 1. Figure 2 depicts the graphical representation of the relative error values (REVs) of all models for Dataset 1. It is evident that the proposed model is closer to zero at each point of time index compared to other models.

Figure 1.  MVFs of the 12 models for Dataset 1.

DownLoad: Full-Size Img PowerPoint

Figure 2.  REVs of the 12 models for Dataset 1.

DownLoad: Full-Size Img PowerPoint

In Table 6, we can observe that the MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PIC and PC values for the proposed model are the lowest, and that the R² and Adj R² values for the proposed model are the largest among all the models compared. As listed in Table 6, the values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PIC and PC for the proposed model were 41.6728, 6.2699, 6.2735,192.7933, 5.6704, 5.5084, 5.2018, 1470.8747 and 65.1464, respectively, which are lower. The proposed model's values of R² and Adj R² were 0.9952 and 0.9948, respectively, which are higher. The PRR value for the L-DP model was 1.6682, which is the lowest. Next, the P-DP3 model's values of MSE, PRV, RMSE, TS, PIC and PC were 46.1444, 6.4682, 6.5028, 5.3927, 1570.7650 and 65.7002, respectively, which are the second lowest. Furthermore, the P-DP3 model's values of R² and Adj R² were 0.9949 and 0.9942, respectively, which are the second largest. The SAE, MAE and MEOP values for the L-DP model were 192.7933, 5.8560 and 5.6838, respectively, which are the second lowest. Figure 3 depicts graphical representations of the MVFs for all models based on Dataset 2. Figure 4 depicts the graphical representation of the REVs for all models for Dataset 2.

Figure 3.  MVFs of the 12 models for Dataset 2.

DownLoad: Full-Size Img PowerPoint

Figure 4.  REVs of the 12 models for Dataset 2.

DownLoad: Full-Size Img PowerPoint

In Table 7, we can observe that the MSE, PRV, RMSE, TS, PRR, PIC and PC values for the proposed model were the fourth lowest, and that the values of SAE, MAE and MEOP for the proposed model were the fifth lowest when compared to those of the other models. The R² and Adj R² values for the proposed model were the fourth largest among all of the models compared. As listed in Table 7, the values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PRR, PIC and PC for the proposed model were 141.9586, 11.4880, 11.5765,358.5639, 10.5460, 10.2447, 8.1145, 3.5573, 4880.5910 and 85.9831, respectively. The values of R² and Adj R² for the proposed model were 0.9862 and 0.9850, respectively. The values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PRR, PIC and PC for the O-IS model were 99.0515, 9.6336, 9.6710,238.4523, 7.0133, 6.8129, 6.7782, 1.8966, 3421.7495 and 79.8649, respectively, which are the lowest. In addition, the values of R² and Adj R² for the O-IS model were 0.9904 and 0.9895, respectively, which are the largest. The values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PRR, PIC and PC for the L-DP model were 119.1020, 10.4128, 10.4478,296.8038, 8.9941, 8.7295, 7.3225, 2.3600, 3978.3649 and 81.3455, respectively, which are the second lowest. The values of R² and Adj R² for the L-DP model were 0.9888 and 0.9874, respectively, which are the second largest.

Figure 5 depicts graphical representations of the MVFs for all models based on Dataset 3. Figure 6 depicts the graphical representation of the REVs of all models for Dataset 3.

Figure 5.  MVFs of the 12 models for Dataset 3.

DownLoad: Full-Size Img PowerPoint

Figure 6.  REVs of 12 models for Dataset 3.

DownLoad: Full-Size Img PowerPoint

In Table 8, we can observe that the MSE, PRV, RMSE, TS, PIC and PC values for the proposed model were the second lowest among all of the models compared. The values of MAE, MEOP and PRR for the proposed model were the third lowest when compared to other models. The R² and Adj R² values for the proposed model were the second largest among all of the models compared. As listed in Table 8, the values of MSE, PRV, RMSE, TS, PIC and PC for the proposed model were 115.5631, 10.3995, 10.4459, 5.9602, 3983.1460 and 82.4859, respectively, which are the second lowest. The MAE, MEOP and PRR values for the proposed model were 8.9976, 8.7405 and 3.0198, respectively, which are the third lowest. The values of R² and Adj R² for the proposed model were 0.9934 and 0.99927, respectively, which are the second largest values. The values of MSE, PRV, RMSE, SAE, MAE, MEOP, TS, PIC and PC for the P-DP3 model were 114.1997, 10.2304, 10.2314,268.6371, 8.1405, 7.9011, 5.8371, 3816.5911 and 80.6520, respectively, which are the lowest.

Furthermore, the values of R² and Adj R² for the P-DP3 model were 0.9936 and 0.9928, respectively, which are the largest. The SAE, MAE and MEOP values for the O-IS model were 293.6791, 8.6376 and 8.3908, respectively, which are the second lowest. The goodness-of-fit of the proposed model was excellent when considering the values of the overall criteria. Figure 7 depicts graphical representations of the MVFs for all models based on Dataset 4. Figure 8 depicts the graphical representation of the REVs of all models for Dataset 4.

Figure 7.  MVFs of all 12 models for Dataset 4.

DownLoad: Full-Size Img PowerPoint

Figure 8.  REVs of 12 models for Dataset 4.

DownLoad: Full-Size Img PowerPoint

4.3.   Results of the average of the normalized criteria and ranking for models

In Table 9, we can observe that the AC values for the proposed model on Datasets 1–2 were the lowest among all of the models that were compared. The AC value for the proposed model on Dataset 3 was the fourth lowest. In addition, the AC value for the proposed model on Dataset 4 was the second lowest. For Dataset 3, the AC value for the O-IS model was the lowest. For Dataset 4, the AC value for the P-DP3 model was the lowest. From the AC values, it is evident that the goodness-of-fit of the proposed model is excellent. Figure 9 depicts the graphical representation of the three-dimensional plots for the model, AC values and ranks of the 12 models listed in Table 9 for Datasets 1–4.

Table 9.  AC values for model comparison on Datasets 1–4.

No.	Model	Dataset 1	Dataset 2	Dataset 3	Dataset 4
1	GO	0.14288	0.14586	0.15089	0.14702
2	HD-GO	0.15684	0.16297	0.17542	0.16900
3	Y-DS	0.10143	0.09779	0.08547	0.09339
4	O-IS	0.03764	0.03318	0.01930	0.02850
5	Y-Exp	0.14963	0.15136	0.15869	0.15445
6	Y-Ray	0.10133	0.09713	0.08474	0.09400
7	PZ-IFD	0.08942	0.08613	0.07121	0.08188
8	P-DP1	0.08810	0.08270	0.06424	0.07774
9	P-DP2	0.07100	0.07817	0.08379	0.07844
10	P-DP3	0.02331	0.02485	0.03470	0.01621
11	L-DP	0.02513	0.02500	0.02752	0.03406
12	New	0.01329	0.01488	0.04402	0.02532

---

 | Show Table

DownLoad: CSV

Figure 9.  Three-dimensional plots of the model, AC values and ranks of 12 models listed in Table 9 for Dataset 1–4.

DownLoad: Full-Size Img PowerPoint

4.4.   Results of the sensitivity analysis for software reliability

In this section, we present a sensitivity analysis (SA) to examine the effect of each parameter of the proposed model on software reliability. We analyzed how the software reliability values change when each estimated parameter value from Datasets 1 to 4 changes by 5% from -15% to +15% using Equation (2). Figures 10 depicts the SA for three parameters ($a$, $b$, $k$) in the proposed model based on Dataset 1. As depicted in Figure 10, the estimated software reliability values can be viewed according to each estimated parameter. Parameter $a$ shows that it does not significantly affect the reliability. Parameter $b$ has a greater influence than the other parameters, and it is determined that parameter $k$ also has a slight effect. As can be seen in Figures 11–13, the results of the SA of the reliability are similar.

Figure 10.  SA of the reliability of the proposed model on Dataset 1.

DownLoad: Full-Size Img PowerPoint

Figure 11.  SA of the reliability of the proposed model on Dataset 2.

DownLoad: Full-Size Img PowerPoint

Figure 12.  SA of the reliability of the proposed model on Dataset 3.

DownLoad: Full-Size Img PowerPoint

Figure 13.  SA of the reliability of the proposed model on Dataset 4.

DownLoad: Full-Size Img PowerPoint

5.   Conclusions

In the era of the fourth industrial revolution, since most core technologies are implemented in software, it is very important to reduce the possibility of software failure and maintain a high level of reliability. Many large companies are performing their own OSS developments, and the development process for OSS is primarily because users find software defects during the usage process and communicate them to the developer, who checks and removes them. We discussed a new SRM that considers the number of finite faults and dependent faults, and we examined the goodness-of-fit based on several criteria by using OSS datasets to show that the new SRM is superior to existing models. Since it is difficult to select the first criterion to confirm the performance excellence of the SRM due to the variety of comparison criteria, we proposed a new integrated criterion considering the value and ranking of each criterion, and the results also demonstrated that the proposed model is superior to other models. In addition, SA was conducted by using the amount of change in each parameter to determine the extent to which the parameter affects software reliability, and it was found that the parameter $b$ had a greater effect than other parameters.

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation (NRF) of Korea, funded by the Ministry of Education (NRF-2021R1F1A1048592 and NRF-2021R1I1A1A01059842).

Conflict of interest

The authors declare that there is no conflict of interest.