Saddlepoint approximation for the p-values of some distribution-free tests

1.   Introduction

Asymptotic approximations of the saddlepoint type for the p-value of the test statistics of two distribution-free tests are considered. The first test addresses the scale problem and it has been introduced by Mathur and Dolo ^[1] as a good alternative to some famous scale tests such as the Siegel-Tukey test ^[2], Levene test ^[3], Klotz test ^[4] and Fligner and Killeen test ^[5]. The second test addresses the location problem and is a bivariate signed-ranked test ^[6]. This test is characterized by its statistic being independent of the correlation between two variables, making it easy to note the marginal effect of a single variable on the test statistic. Moreover, it is a scale-invariant test, ensuring that the value of the statistic does not change when the scale of the observations has been changed. Furthermore, it is robust to outliers and more robust than its counterparts under the non-normal distribution, even under very small changes in location. Mathur and Sepehrifar ^[6] have proven that their test is competitive with several analogues, such as the Mardia test ^[7], Wilcoxon one-sample bivariate rank sum test, ^[8] and the Peters and Randles test ^[9]. It should be noted that the statistics of the two tests considered here, whether the scale or the location test, depend on the ratio of the two variables. This technique was also used by many statisticians in the formation of their test statistics, such as Blumen ^[10], and Sen and Mathur ^[11].

The saddlepoint approximation is fundamentally a method for approximating a probability density or mass function using its corresponding moment-generating function or cumulant generating function ^[12]. It is a frequently used statistical approximation method in approximating many statistical and probabilistic functions. The theoretical and applied statistics are full of approximation methods to solve many problems that do not have an exact solution. The accuracy of the approximation method that each method provides is what distinguishes one method from another. In the case of approximating the statistical functions, such as the distribution, mass, and density functions, we can refer to the asymptotic normal approximation method, which depends on the central limit theorem, and to the saddlepoint approximation method, which is considered a generalization to Laplace's method for approximating integrals. The saddlepoint approximation method offers several significant benefits in statistical inference, particularly in hypothesis testing and p-value approximation. One of the primary advantages is their high accuracy, especially for small sample sizes, where traditional asymptotic methods often fall short. Unlike standard approximations, the saddlepoint method produces precise tail probabilities, which are critical in accurately assessing the significance of test statistics. Furthermore, this approximation is highly versatile and can be applied to a wide range of complex distributions, including nonparametric tests and ratio-based statistics. The computational efficiency of the saddlepoint method also makes it a practical alternative to the simulation methods, reducing the need for extensive simulations. Overall, the saddlepoint approximation method is a powerful and flexible tool that can outperform traditional methods, particularly in challenging scenarios where exact solutions are impractical or unavailable. In this article, a comparison is made between the accuracy of the two methods for approximating the p-value of the two statistics of the considered non-parametric tests, clarifying that the saddlepoint approximation method is more accurate than the normal approximation method. The origin of the saddlepoint method in statistics can be traced back to 1954 by the work presented by Daniels ^[13] as an approximation of the probability density function for the mean of $n$ random variables. Daniels's work served as the starting point for many scholars and statisticians to present many approximations of statistical and probability functions, such as the approximation of the distribution function, the conditional distribution function, and the bivariate distribution function by Lugannani and Rice ^[14], Skovgaard ^[15], and Wang ^[16], respectively. Subsequently, contributions were made to this topic, and its applications have spread to many branches of statistics. In this regard, we can refer to several references, for example, ^{[12,17,18,19,20]}. Gatto and Jammalamadaka ^[21] introduced a saddlepoint approximation for the distribution function of a $M$ statistic, conditioned on another $M$ statistic. Abd-Elfattah and Butler ^[22,23] derived the permutation distribution of the weighted log-rank class of tests using the saddlepoint approximation method. Abd-Elfattah ^[24,25] proposed the saddlepoint approximation method to approximate p-values for weighted log-rank class of tests considering truncated binomial and randomized block designs, respectively. Abd El-Raheem and Abd-Elfattah ^[26,27] extended the results of ^[22,25] for the clustered censored data. Abd El-Raheem et al. ^[28] approximated the tail probabilities for multivariate sign and signed-rank tests using the saddlepoint approximation method. Readers may refer to recent works on the saddlepoint approximation, such as ^{[29,30,31,32]}.

This article aims to enhance the accuracy of p-value calculations in small sample sizes, where traditional methods, such as the normal approximation, can fail to provide precise results. The accuracy of the proposed approach (saddlepoint approximation) is assessed by comparing the approximated p-values to exact p-values obtained by the simulation method (permutation-based, so time-consuming). The relative absolute error is used to evaluate the precision and reliability of the approximation across various scenarios.

The article is organized as follows: In Sections 2 and 3, we provide the saddlepoint results and show the other asymptotic approximations for the considered tests. Finally, Sections 4 and 5 are devoted to the numerical comparisons between the saddlepoint and normal approximation methods using real and simulated data, respectively.

2.   Dispersion test

Let $y_{i}$, $i = 1, 2, ..., n$ and $x_{j}$, $j = 1, 2, ..., m$ be independent random samples from continuous populations with distribution functions $F_{Y}(y)$ and $F_{X}(\sigma y)$. To test the hypothesis $H_{0}: F_{Y}(y) = F_{X}(\sigma y)$, for all $y$ and $\sigma = 1$ against $H_{1}: F_{Y}(y) \neq F_{X}(\sigma y)$, for all $y$ and $\sigma \neq 1$, $\sigma > 0$. Mathur and Dolo ^[1] introduced the dispersion test statistic as follows:

where $r_{k} = \frac{x_{j}-M }{y_{i}-M}$, $M$ is the median of the combined samples, $R(r_{k})$ is the rank of $r_{k}$, and

The expectation and variance of the test statistic in (2.1) under $H_{0}$ are $E(W|H_{0}) = mn(mn + 1)/4$, and $V(W|H_{0}) = mn(mn + 1)(2mn + 1)/24$. When sample sizes are large

Saddlepoint approximation

In this subsection, the saddlepoint approximation method is applied to approximate the p-value of the statistic in (2.1).

Depending on the permutation distribution of the statistic $W$ in (2.1), then the moment generating function of $W$ is given by:

then, the cumulant generating function, CGF, of the statistic $W$ is

The first, second, and third derivatives of the CGF, $K_{W}$, in (2.2) are

and

The saddlepoint approximation of the cumulative distribution function of the statistic $W$, $F_W(w)$ is given by ^[14]:

where

Here, $\Phi$ and $\phi$ represent the standard normal distribution function and density function, respectively, and $\mu_{W} = E(W|H_{0})$ is the mean of the distribution. The symbol $\text{sgn}(\tilde{s})$ denotes the sign of $\tilde{s}$. The saddlepoint $\tilde{s}$ is the unique solution to the equation $K_W'(\tilde{s}) = w$.

The saddlepoint approximation of the exact p-value for the statistic $W$ is given by:

where $w_0$ is the observed value of the statistic $W$.

3.   Signed rank test for bivariate location problem

Let a random sample $(x_i, y_i)$ for $i = 1, 2, \ldots, n$, consisting of $n$ independent pairs taken from a bivariate population with a continuous distribution function $F_{X, Y}(x + \mu_1, y + \mu_2)$. Assume the population is elliptically symmetric around its median $(\mu_1, \mu_2)$. We aim to test the null hypothesis $H_0: (\mu_1, \mu_2) = (0, 0)$ against the alternative hypothesis $H_1: (\mu_1 \neq 0, \mu_2 \neq 0)$. For this purpose, assume $\vartheta_{i} = tan^{-1}(d_i)$, where $d_i = y_i /x_{i}$ for $i = 1, 2, \ldots, n$ are the tangents of the projected angles corresponding to $S_1, S_2, \ldots, S_n$, respectively, where $S_i = (X_i, Y_i)'$ for $i = 1, 2, \ldots, n$. Sign tests are fundamental in non-parametric methods, and numerous researchers have worked on developing non-parametric tests based on this concept. When the underlying population exhibits elliptical symmetry, it seems intuitive and logical to assess the distance of each observation from the origin. Mathur and Sepehrifar ^[6] used the ranks of these distances along with the directions of the observations to construct a signed rank statistic for the bivariate location problem as follows:

and

where

Under the the null hypothesis $H_0$, $P(\delta_i = -1) = P(\delta_i = 1) = 1/2$, the statistic $U_1$ has normal distribution with $E(U_1 |H_{0}) = 0$ and $V(U_1 |H_{0}) = \frac{1}{3}\sum_{i = 1}^{n} \cos^{2}\left(\frac{ i \pi }{n}\right)$, and the $U_2$ has normal distribution with $E(U_2 |H_{0}) = 0$ and $V(U_2 |H_{0}) = \left(n-\sum_{i = 1}^{n} \cos^{2}\left(\frac{i \pi }{n}\right)\right)/3$. Thus, the bivariate signed rank statistic is given by:

where $C_i = \frac{1}{n}{R}(d_i) \left(\cos\left(\frac{i \pi }{n}\right) +\sin\left(\frac{i \pi }{n}\right) \right)$. The statistic $U$ in (3.3) is normally distributed with mean $E(U |H_{0}) = 0$ and variance $V(U |H_{0}) = 1/3$. Let $\eta_i = \frac{ \delta_i +1}{2}$, then $\eta_i = 0$ or $\eta_i = 1$. Thus, the statistic in (3.3) becomes

Saddlepoint approximation

This subsection applies the saddlepoint approximation method to approximate the p-value of the statistic in (3.4).

The moment generating function of the statistic $U$ in (3.4) is determined by its permutation distribution and is given by:

then, the CGF of the statistic $U$ is

The first, second, and third derivatives of the CGF, $K_{U}$, in (3.5) are

The saddlepoint approximation of the cumulative distribution function of the statistic $U$, $F_U(u)$, is given by ^[14]:

where

The saddlepoint $\tilde{s}$ is the unique solution to the equation $K_U'(\tilde{s}) = u$.

The saddlepoint approximation of the exact p-value for the statistic $U$ is given by:

where $u_0$ is the observed value of the statistic $U$.

4.   Numerical comparisons

Analyzing real data is essential for validating the saddlepoint approximation and comparing it to the normal approximation. By applying the saddlepoint approximation to real data, we can assess its accuracy in approximating the exact p-value and its effectiveness across various scenarios, thereby confirming its robustness and reliability.

4.1.   Numerical comparisons for the dispersion test

Example 1: Prehistoric Native Americans used pipes for ceremonial purposes, which were typically made of carved stone or clay ceramics. Clay pipes were easier to produce, while stone pipes required careful drilling using a hollow bone and special stone drills. According to one anthropologist, the easier manufacturing process of clay pipes resulted in greater variation in their construction. The diameters of ceramic pipe bowls and stone pipes (cm) from the Wind Mountain archaeological area were measured to evaluate this claim. These data are presented in Table 1, and the source of these data is the reference ^[33]. The dispersion test is used to evaluate the null hypothesis, which states no difference in variance, against the alternative hypothesis supporting the anthropologist's claim that clay pipes exhibit greater variance. The approximated p-value using the simulation (permutation-based method), saddlepoint, and normal approximation methods are calculated and listed in Table 2.

Example 2: A key indicator of a company's productivity is the relative annual return to its total assets. This metric shows the return generated from assets over a year, thoroughly assessing the company's financial efficiency and profitability. It is useful for comparing competing firms. Table 3 displays the percentage returns based on assets for a random sample of prominent companies from France and Germany. The source of these data is the reference ^[34]. The dispersion test is used to evaluate the claim that there is a difference in the population variance of percentage yields between leading companies in France and Germany. The approximated p-value using the simulation, saddlepoint, and normal approximation methods are obtained and listed in Table 2.

Table 2 shows that the saddlepoint approximation technique offers higher accuracy and reliability than the traditional method, as it is significantly closer to that obtained using the simulation method.

4.2.   Numerical comparisons for the bivariate signed rank test

Example 1: In 2010, the average minimum temperatures recorded for counties Roscommon and Meath in Ireland showed notable variations throughout the year. These temperatures, measured in degrees Celsius (℃), reflect the coldest daily temperatures, averaged over the year. Table 4 shows the average minimum temperatures (℃) recorded for counties Roscommon and Meath, Ireland 2010; see ^[35] for more details. We aim to test the null hypothesis, $H_0: (\mu_1, \mu_2) = (0, 0)$ against the alternative hypothesis $H_1: (\mu_1 \neq 0, \mu_2 \neq 0)$.

Example 2: The data were collected to study the development of the immune system in 36 HIV-positive newborns. These children were given Ritonavir therapy, and their CD45RA and CD45RO T cell counts were measured at birth and again after 24 weeks of treatment. Table 5 shows the difference between CD45RA T cells and CD45RO T cells at 24 weeks of treatment and at birth. The source of these data is the reference ^[36]. The bivariate signed rank test was conducted to determine whether these cell counts were significantly changed over the treatment period. The p-value of this test is calculated using simulation, saddlepoint, and normal approximation methods and presented in Table 6.

Table 6 demonstrates that the saddlepoint approximation technique provides greater accuracy and reliability than the traditional method, as its results are much closer to those obtained using the simulation method.

5.   Simulation study

When analytical solutions are intractable or impractical, statistical approximation methods, such as asymptotic normal approximation or saddlepoint approximation methods, are often employed to make inferences about complex models or datasets. Understanding the relative performance of these methods is crucial for selecting the most appropriate technique for a given context. This section uses simulation studies to evaluate and compare the performance of different approximation methods under various conditions.

5.1.   Simulation study for the dispersion test

For the dispersion test, a simulation study is performed to assess the consistency of the saddlepoint p-value approximation across various sample sizes, distributions, and location parameter values. The simulations involved two distributions: logistic and extreme value. For each distribution, 1,000 data sets are generated with total sample sizes of $N = 16, 24, 32$, and $42$, where $m = n = N/2$. The simulated mid-p-value for each of the 1,000 data sets is calculated using $10^6$ randomized sequences for the indicators $\phi_k$. Let $\sigma = \sigma_x / \sigma_y$ represent the dispersion parameter, where $\sigma_x$ and $\sigma_y$ are the scale parameters for the populations $X$ and $Y$, respectively. Let $M_x$ and $M_y$ be the medians of the two populations $X$ and $Y$, respectively. The data sets are generated with $M_y = 0$, $\sigma_y = 1$, $M_x = c \sigma_y$, where $c = 0, 1$, and $2$, and $\sigma_x$ is selected to ensure that the mean of the simulated mid-p-values for the 1,000 data sets is approximately 0.05. To compare the saddlepoint and normal approximation methods, we calculate the following quantities: saddlepoint approximation proportion (Sap.Prop.) refers to the proportion of the saddlepoint method to the simulation method, relative absolute error of saddlepoint (Rel.Abs.Err.Sap.) indicated the accuracy of the saddlepoint method compared to the simulation method, and relative absolute error of normal (Rel.Abs.Err.Nor.) represented the accuracy of the normal method compared to the simulation method. The mathematical definition of the quantities Sap.Prop., Rel.Abs.Err.Sap., and Rel.Abs.Err.Nor. are given by:

where $I()$ denotes the indicator function, $P_{i}(Sap)$ represents the saddlepoint p-value, $P_{i}(Nor)$ represents the normal approximation p-value, and $P_{i}(Sim)$ represents the simulated p-value.

and

Results of the simulation study for comparing saddlepoint and normal approximation techniques are presented in Table 7. It also shows that the average of the Sap.Prop. is approximately $87.56\%$ (this value is the average of all fourth-column values in Table 7). This high percentage indicates that the saddlepoint approximation method was more accurate in $87.56\%$ of the considered cases. Furthermore, The average of the Rel.Abs.Err.Sap. is approximately 0.049 (this value is the average of all fifth-column values in Table 7), and the corresponding value for the normal approximation is approximately 0.1193 (this value is the average of all sixth-column values in Table 7). The large difference between the relative absolute error of the two methods also shows that the saddlepoint method is more accurate than the normal approximation method. We can illustrate the superiority of the saddlepoint approximation method over the normal approximation method by plotting the relative absolute error of the two methods. Figures 1 and 2 display the relative absolute error of the saddlepoint and normal approximation methods for logistic and extreme value distributions when $N = 16$ and $c = 0$. It is clearly evident that the relative absolute error resulting from the saddlepoint method is much less than that resulting from the normal approximation method.

5.2.   Simulation study for the bivariate signed rank test

Bivariate data are generated from the bivariate normal, logistic, and extreme value distributions to compare the different approximation methods used to approximate the exact p-value of the bivariate signed rank test statistic. 1,000 samples are generated from each distribution. The p-value is calculated using the three approximation methods for each sample of data, and then we calculate the average of the thousand p-values of each approximation method. Results of the simulation study for comparing saddlepoint and normal approximation techniques are displayed in Table 8.

It also shows that the average of the Sap.Prop. is approximately $97\%$. This high percentage indicates that the saddlepoint approximation method was more accurate in $97\%$ of the considered cases. Furthermore, the average of the Rel.Abs.Err.Sap. is approximately 0.0258, and the corresponding value for the normal approximation is approximately 0.6145. The large difference between the relative absolute error of the two methods also shows that the saddlepoint method is more accurate than the normal approximation method. Figures 3 and 4 illustrate the relative absolute error of approximating the p-value of the bivariate signed rank test using the normal approximation (shown in red) and the saddlepoint approximation (shown in blue). From Figures 3 and 4, it is evident that the saddlepoint approximation consistently achieves lower error rates across all sample indices compared to the normal approximation. The red lines corresponding to the normal approximation show frequent and large spikes in error, indicating that the normal approximation tends to produce larger deviations from the true p-values. In contrast, the blue lines representing the saddlepoint approximation remain much closer to zero, with minimal variation, highlighting its superior accuracy.

6.   Conclusions

The article highlights the effectiveness of using the saddlepoint approximation for calculating p-values in distribution-free tests, particularly for the signed rank test and a nonparametric scale test. The study compares the saddlepoint approximation method to the traditional asymptotic normal approximation method, showing that the saddlepoint approximation consistently provides lower error rates in p-value approximation. Through numerical comparisons and practical examples, the findings demonstrate that the proposed method offers greater accuracy and can serve as a reliable alternative to traditional approaches in nonparametric statistics. This suggests that the saddlepoint approximation has practical advantages in improving the precision of statistical tests.

Author contributions

A. M. Abd El-Raheem: Conceptualization, Methodology, Investigation, Software, Writing – review & editing, Visualization, Resources, Software, Writing – original draft; M. Hosny: review & editing, Funding acquisition, Project administration. All the authors have agreed and given their consent for the publication of this research paper.

Use of Generative-AI tools declaration

The authors declare that they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The second author thanks the Deanship of Scientific Research and Graduate Studies at King Khalid University for funding this work through a Large Research Project under grant number RGP2/398/45.

Conflict of interest

The authors declare no conflicts of interest.