What is the probability that statistical significance will be achieved in a second test?

Question

Assume I'm thinking of investing in a small biotech company. In a phase 2 study with 120 patients split 80/40 between the new drug and old drug, the progression free survival (PFS) rate of the patients taking the new drug was greater than that of those taking the old drug at the 1% level (P = 0.01).

Now, the company is conducting a phase 3 trial with 300 patients split 200/100 between the new drug and the old drug. What is the probability that the PFS will be greater for the patients taking the new drug than for those taking the old drug at the 5% level (P = 0.05) or better?

In other words, what is the probability that the phase 3 trial will be successful so that the FDA will presumably approve the drug? I want to know if the drug company is a good investment, which of course also depends upon the market capitalization of the company and the size of the potential market, but the first question is what the probability is of the drug trial being successful? Is there any way to get a handle on that based only on the P2 results?

Edit: P.S. This is not a homework problem, though it (or something like it) would seem to be a standard and useful problem.

Answer 1

It seems reasonable to assume that the tests involved in both Phase 2 and Phase 3 are tests that compare 'survival' fractions $p_1 = X_1/n_1$ (new drug) with $p_2 = X_2/n_2$ (old drug) to see if the former significantly exceeds the latter.

The test statistic $z = (p_1 - p_2)/d$, where $d^2 = p_1(1-p_1)/n_1 + p_2(1 - p_2)/n_2.$ We would say that the new drug is significantly better than the old at if $z > z^*$, where the critical value is $z^* = 2.3263$ for a test at the 1% level and $z^* = 1.6449$ at the 5% level.

For the Phase 2 test, we had $n_1 = 80$ and $n_2 = 40.$ The surviving numbers $X_1$ and $X_2$ with the new and old drugs, respectively, are unknown. To start, we assume that $X_2 = 20$ so that $p_2 = 0.5$. Then we ask what $X_1$ must have been in order to achieve significance at the 1% level. Algebra would be a little messy, but a straightforward grid search for the smallest $X_1$ that would give the desired result is $X_1 = 58$ so that $p_1 = 58/80 = 0.725,$ $z = 2.4064 > 2.3263.$

Then for a the Phase 3 test (assuming the same protocol for choosing patients, drug dosages, etc.) we suppose we would be choosing $X_1 \sim Binom(200, 0.725)$ and $X_2 \sim Binom(100, 0.5).$ Finding the corresponding $p_1, p_2,$ and $z,$ we wonder what proportion of the $z$'s would exceed $1.6449$ in order to have significance at the 5% level, and (we hope) approval by the FDA. A simple simulation shows that proportion to be about 98.4%.

Fortunately, the outlook is about as promising, if we assume $p_2 = 0.3$ in the the Phase 2 trial; and even more promising (proportion about 99.5%), if we assume the the old drug has $p_2 = 0.8$.

By making additional, possibly realistic, assumptions, this problem might be solved satisfactorily without using a computer. If this is a homework problem, I will leave that to you.

---

In case you are interested in my computation and simulation in R, the code is shown below. To avoid confusion in programming, I used $X$'s and $p$'s in Phase 2 and $Y$'s and $q$'s in Phase 3. (I do not claim my code is optimal or elegant.)

 # Phase 2: Grid search for required p1
 n1 = 80;  n2 = 40;  x2 = 20;  z = numeric(n1)
 for(x1 in 1:80)  {
    p1 = x1/n1;  p2 = .5
    den = sqrt( p1*(1-p1)/n1 + p2*(1-p2)/n2 )
    z[x1] = (p1 - p2)/den }
 crit = qnorm(.99)
 i = 1:n1
 pp1 = min(i[z > crit])/n1 ; pp1
 ## 0.725

 Phase 3: Proportion of expts signif at 5%
 m = 10^5;  n1 = 200;  n2 = 100
 y1 = rbinom(m, n1,  pp1);  y2 = rbinom(m, n2,  p2)
 q1 = y1/n1;  q2 = y2/n2
 den = sqrt( q1*(1-q1)/n1 + q2*(1-q2)/n2 )
 z = (q1 - q2)/den
 mean(z > qnorm(.95))
 ## 0.98436