Most non-inferiority (NI) clinical trial is conducted with large-scale enrollment. With this scale, the parametric testing method is used to confirm the non-inferiority in general. Hida and Tango (2011) introduced a parametric t-test to assess the NI trial. In this approach, the mean of each treatment group is used as the measure of the hypotheses, and the test statistic follows a Student’s t-distribution and degrees of freedom vary according to variance’s homogeneity. However, in the case of rare diseases, it is difficult to recruit patients who meet the eligibility criteria. In this case, a clinical trial is terminated due to insufficient enrollment, and research on the rare disease could not persist. Therefore, there have been some research on NI trials with nonparametric methods, which are less restrictive than parametric methods. The nonparametric method is not affected by the normality of data and the homogeneity of variance.

In situations where nonparametric methods are appropriate, enrollment is small scale or normality of data is not assured, there have been several methods. In this paper, we consider the three-arm design non-inferiority clinical trial. The three-arm design indicates that the clinical trial with experimental, reference drug, and placebo. The importance of three-arm design is highlighted in various research since we can show the assay sensitivity in the presence of a placebo arm. In the ICH E10 guideline (2000), assay sensitivity is defined as the property of a clinical trial defined as the ability to distinguish an effective treatment from a less effective or inefficient treatment. Thus, in many non-inferiority clinical trials, demonstrating assay sensitivity takes precedence over establishing non-inferiority. Park and Kim (2014) proposed a ratio-shape formulation for the non-inferiority hypotheses. They utilized a Hodges-Lehmann estimator to test this non-inferiority hypothesis. However, there is an alternative approach to assessing nonparametric NI trials known as the ‘relative effect’ which uses the relative effect as a measure of the hypothesis instead of the mean effect. Munzel (2009) introduced a ratio-shaped hypothesis with relative effect.

In this paper, we introduce a modification of the method by Park and Kim (2014) applicable in a three-arm NI trial, along with a modification of Munzel (2009) method utilizing unweighted relative effect—a measure calculated with pseudo rank, offering advantages over traditional weighted rank. While various NI trial testing methods have been proposed, comprehensive comparisons of their performance remain scarce. Our primary focus lies in introducing and applying these methods. To rigorously assess their performance and effectiveness, we conduct a comprehensive comparison with existing testing methods through a carefully designed simulation study. By evaluating empirical levels and statistical powers, we aim to provide valuable insights into the practical utility of these methods in non-inferiority clinical trials.

This paper is structured as follows. Section 2 introduces existing NI trial testing methods, encompassing both parametric and nonparametric approaches. In Section 3, we present the modification of the method by Park and Kim for three-arm clinical trials and propose the utilization of unweighted relative effect in Munzel’s method. Section 4 presents the simulation results, where we evaluate the performance of the testing methods. Finally, Section 5 concludes our results and initiates a discussion on the implications of our findings.

2.1. Parametric method

Hida and Tango (2010) suggested a parametric testing method when a NI trial includes a single experimental, reference treatment and placebo. Assume that the primary endpoint under the three-arm is X_{i j}, i = E, R, P, j = 1, . . . , n_i, respectively. X_Ej , X_Rj and X_Pj are mutually independent and normally distributed with unknown but common variance σ², that is, X_Ej N(μ_E, σ²), j = 1, . . . , n_E, X_Rj N(μ_R, σ²), j = 1, . . . , n_R and X_Pj N(μ_P, σ²), j = 1, . . . , n_P. The total sample size is N = n_E + n_R + n_P. Each sample size of treatment group is not necessarily identical. The non-inferiority trial and assay sensitivity null hypotheses are H_0E: μ_E – μ_R ≤ −M₂ and A₀: μ_R – μ_P ≤ M₁. H_0E is the hypothesis of NI trial and A₀ is the hypothesis of assay sensitivity. The corresponding alternative hypotheses are H_1E: μ_E – μ_R > −M₂ and A₁: μ_R – μ_P > M₁ respectively. M₁ is the entire effect size of reference drug and M₂ is the largest clinically acceptable difference, i.e., non-inferiority margin. It is required that M₁ ≥ M₂ = r × M₁, where 0 < r ≤ 1. Choosing the proper value of M₁ and M₂ is found in FDA guidance for non-inferiority trials (2016).

To test the null hypotheses H_0E and A₀, Student t-test is used. T_E is the test statistic for NI trial

and T_A is the test statistics for proving assay sensitivity.

where X¯i=(1/ni) Σj=1nj Xij, i = E, R, P, j = 1, . . . , n_i. And the homogeneous variance σ̂_ER and σ̂_RP are defined σ^ER2=((nE-1)sE2+(nR-1)sR2)/nE+nR-2 and σ^RP2=((nR-1)sR2+(np-1)sP2)/nR+nP-2 where s_E², s_R² and s_P² denote a sample variance of experimental, reference and placebo treatments, respectively. The test statistic T_E follows a t-distribution with degrees of freedom (n_E + n_R – 2) and T_A follows a t-distribution with degrees of freedom (n_R + n_P – 2). And corresponding 100 × (1 – α)% confidence intervals are

and

respectively.

In case of heterogenous variance, the identical confidence interval is applied and the only difference is degrees of freedom. Detailed formulas are found in Huang et al. (2015).

2.2. Nonparametric methods

As mentioned in the introduction, we can divide nonparametric NI trial testing methods into two categories. The first category involves NI hypotheses testing with mean effect, while the other category focuses on NI hypothesis testing with relative effect. We first introduce the method by Park and Kim (2014). However, in our comparison scenario assuming the clinical trial with a three-arm design, Park-Kim method is not included because Park-Kim method is applied in case of two-arm clinical trial which contains only a single experimental drug and reference drug, not a placebo.

3.1. Modified Park-Kim method

In this section, we modify the Park-Kim method by reformulating it in three-arm design case. We add placebo arm in the hypothesis, and the NI hypothesis H₀: (μ_E – μ_R)/μ_R – μ_P ≤ −γ vs. (μ_E – μ_R)/μ_R – μ_P > −γ, where r is pre-specified margin. We have retained the numerator and made a modification to the denominator estimator, changing it to two sample of Hodges-Lehmann estimator.

Suppose Δ̂ = median{(X_Rj – W_Ps), j = 1, . . ., n_R, s = 1, . . ., n_P}, where X_Rj is the j^th response of reference treatment and W_Ps is the sth response of placebo treatment. And then define the order statistic of (X_Rj–W_Ps) as H₍₁₎, H₍₂₎, . . ., H_(nRnP). When n_Rn_P is odd, n_Rn_P = 2b+1 then b = (n_Rn_P–1)/2, the Hodges-Lemann estimator would be Δ̂ = H_(b+1), and when n_Rn_P is even, n_Rn_P = 2b then b = (n_Rn_P)/2, the Hodges-Lemann estimator would be Δ̂ = (H_(b) + H_(b+1))/2. Therefore, the lower and upper limit of modified nonparametric 100 × (1 – α)% confidence interval of (μ_E – μ_R)/μ_R – μ_P is

respectively.

3.2. Modified Munzel method using unweighted relative effect

In section 2.2.2, we introduced the concept of a ‘usual rank’, which is calculated based on the total number of observations. Pseudo rank, on the other hand, is slightly different. It represents an unweighted rank, where we consider the total number of groups instead of the total number of observations. We refer to relative effects calculated using pseudo rank as ‘fixed relative effects’ because they are not influenced by the number of observations. The concept of unweighted relative effect was suggested by Brunner and Puri (1996) for the first time, and then asymptotic properties of unweighted relative effect was demonstrated by Domhof (2001). Brunner et al. (2021) cautioned that the weighted relative effect can vary according to the sample ratio, making it an unstable measure for use in non-inferiority trials. Therefore, we opted to use the pseudo rank method, which remains stable and is not affected by the sample ratio.

Let U=(1/K) Σi=1kFi denote the unweighted mean distribution. As mentioned above, it is not affected by total number of observations. Similar to usual rank, respective empirical version of function U is U^=(1/K) Σi=1kF^i. And let R_{i j}^ϕ represents the pseudo rank of X_{i j} among all k treatment groups, then R_{i j}^ϕ is defined as

where U^(x)=(1/k) Σr=1kFr(x) denotes the unweighted mean of empirical distribution functions. It can be also estimated consistently by the simple plug-in estimator

where R¯i.ϕ=(1/ni)Σj=1niRijϕ and U^=(1/k)Σr=1kF^r denotes the unweighted mean of empirical distributions F̂₁, . . ., F̂_r. The value piϕ quantifies an effect of the distribution F_i with respect to the unweighted mean distribution U. Fixed relative effect p^iϕ is also related to mean of the pseudo rank R¯iϕ. The only difference between relative effect and fixed relative effect lies in the replacement of Ŵ(X_{i j}) with Û(X_{i j}). Therefore, the application unweighted relative effect to Munzel method is simply substituting (weighted) relative effect to unweighted relative effect. The null hypotheses are H_0E: (p^ϕ_E – p^ϕ_R)/P^ϕ_R – p^ϕ_p ≤ –δ and A₀: P^ϕ_R – P^ϕ_P ≤ Q₁. The corresponding alternative hypotheses are H_1E: (p^ϕ_E – p^ϕ_R)/p^ϕ_R – p^ϕ_p > −δ and A₁: P^ϕ_R – P^ϕ_P > Q₁, respectively. The two-sided (1–α)×100% confidence interval for ratio (p^ϕ_E – p^ϕ_R)/p^ϕ_R – p^ϕ_P is exactly identical with just substituting p_i to p^ϕ_i, i = E, R, P.

4.1. Simulation scheme

To assess the performance of the testing methods introduced in the previous sections, we conducted an extensive simulation study. The criteria for demonstrating its performance are empirical level and statistical power. We aim to assess how the performance of the testing methods vary under various situations. We generate the data from three distributions (normal, gamma and exponential distribution). For the normal distribution, we consider both equal variance and unequal variance cases. In case of the gamma distribution, we used two different shape parameters (α) while keeping the rate parameter fixed (β = 2). Similarly, for the exponential distribution, we employed two rate parameters (λ). We also consider the four types of sample ratios reflecting the real-world situations. Additionally, sample size is also an important factor, and thus we have set the ratio 1 as equivalent to 20 and 50 people in small sample and large samples, respectively. Sample ratio and sample size variation is displayed in Table 2. The distribution of data used in simulation is displayed in Table 1.

For normal distribution case, we consider a total of four scenarios considering both homogeneous variance and heterogeneous variance. For gamma and exponential distribution cases, two scenarios are used in simulation study.

The first step of evaluating performance of testing methods is checking the empirical level. We calculated and compared to nominal level before proceeding to the statistical power comparison. We iterate a total of 10,000 times for each testing method, and thus the empirical level is considered valid when if falls within the interval [0.0457, 0.0543]. If the empirical level deviates from this interval, the comparison of statistical power becomes unreliable. After confirming that the empirical level is satisfied, statistical power is calculated. As in Hinda and Tango (2011), the power of the testing procedure is defined as

Setting the NI trial and assay sensitivity margin is also crucial in NI trial. The margins of parametric testing methods denoted by M₁ and M₂ are the same as in the previous studies. Subsequently, the corresponding margins of nonparametric testing methods, labeled as Q₁ and Q₂, are chosen based on the well-known property

It means that Φ is almost linear around 0.5, i.e., it is approximately linearly connected to set the nonparametric effect similar to parametric effect. Thus, gamma and exponential distribution are also applied to its property using normal approximation. The margins for each testing method and distribution are demonstrated in Table 3.

Additionally, an assay sensitivity test must be conducted before initiating an NI trial. The NI trial is performed only after confirming assay sensitivity. If assay sensitivity is not established, the NI trial is not conducted and the simulation is terminated. Thus, the simulation steps are follows:

• Generate data from each distribution with predefined mean, variance, or parameters.

• Apply each testing method to the data and determine whether the hypothesis is rejected or accepted.

• Repeat these steps 10,000 times for each method.

• Calculate the empirical level and statistical power of each testing method.

All simulation studies were conducted using programming version 4.1.2 (R project homepage: http://www.r-project.org). The author developed the codes for generating rank statistics and implementing all testing methods from scratch.

4.2. Simulation results

To improve the readability of the tables, the abbreviations have been employed in the simulation result tables. Abbreviations are as follows: ‘HT-P’ (Hida-Tango parametric method), ‘M-PK’ (modified Park-Kim method), ‘M-UR’ (Munzel method using usual rank), ‘MM-PR’ (modified Munzel method using pseudo rank).

We have presented various NI trial methods, including both parametric and nonparametric approaches. Additionally, we modified Park-Kim method using two sample Hodes-Lehmann estimator. Also, in nonparametric method using relative effect, we applied unweighted relative effect which uses the pseudo rank. Pseudo rank is calculated only with the number of the treatment groups, not affected by the sample ratio of each treatment group. So, it is a fixed measure while usual rank is changed its value according to the sample ratio and thus yields unstable measure.

Now we summarize the major findings from our simulation studies.

• Parametric testing methods demonstrate superiority under normal distribution, while among non-parametric methods, MM-PR exhibits the highest statistical power. Conversely, in non-normal scenarios, parametric methods falter, while MM-PR consistently display superior performance.

• Notably high statistical power is observed in scenarios with a ratio of 2 for experimental drugs and the reference drug, and a ratio of 1 for the placebo.

• Testing method based on unweighted relative effect (MM-PR) consistently outperform those based on weighted relative effect (M-UR).

• Statistical power tends to be higher in situations with larger sample sizes. Particularly, the MM-PR method shows comparable performance to parametric methods in normal cases, while excelling in both normal and non-normal scenarios.

• In small sample scenarios, the MM-PR method exhibits satisfactory statistical power, highlighting its effectiveness, especially in trials with limited sample availability, such as those for rare diseases.

For ease of comparison, we converted Table 7 into a bar graph (Figure 1). Scenarios 1 to 8 represent different distributions listed in Table 7. For example, Scenario 1 is a normal distribution with means (12, 14, 0) and variances (2, 2, 2), while Scenario 6 is a gamma distribution with shape parameters (5, 5, 1), means (10, 10, 2), and variances (20, 20, 4). Each method is represented by different colors. The HT-P method (light grey) performed well in Scenarios 1–4 (normal distributions) but was inferior in Scenarios 5–8 (non-normal distributions). The MM-PR method maintained its position as the second best in Scenarios 1–4 and outperformed in Scenarios 5–8.

We anticipate that our proposed MM-PR method will be particularly useful in rare disease clinical trials, where sample sizes are limited, owing to its minimal susceptibility to data distribution. Additionally, our ongoing research into nonparametric testing methods for cases involving multiple experimental drugs aims to provide further insights into the complexities of relative effect estimation and variance-covariance estimation. All simulation studies were conducted using programming version 4.1.2 (R project homepage: http://www.r-project.org). The author developed the codes for generating rank statistics and implementing all testing methods from scratch.

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00208882, No. NRF-2022M3J6A1063595). This research was also supported and funded by the Korean National Police Agency [Project Name: Advancing the Appraisal Techniques of Forensic Entomology / Project Number: PR10-04-000-22].