4
$\begingroup$

I am doing an experiment in which I measure success or failure of a series of $k$ trials, yielding a proportion $P$. I repeat the experiment several times, yielding a set of $N$ proportions: $P_1, \ldots, Pn$, and would like to compute a mean and confidence interval for these proportions. There are a few things that make this tricky:

  • The experiments are independent, but there might be systematic errors that affect all of the trials within any one experiment (in other words, the trials are pseudo-replicates).

  • The number of trials can vary by as much as a factor of $10$ from experiment to experiment.

  • For reasons related to the design of this specific experiment, I have more confidence in experiments where more trials were conducted.

So far, rather than computing a simple average:

$$P_{est} = \frac1N \sum_{i=1}^N P_i$$

I compute a weighted average proportion:

$$P_{est} = \frac{ \sum_{i=1}^N k_i \cdot P_i}{\sum_{i=1}^N k_i}$$

Which is equivalent to the total number of successes divided by the total number of trials across all experiments.

Now, how do I calculate a confidence interval for this? It is not a simple binomial proportion because of the pseudo-replication issue. Nor is it the simple confidence interval for the mean of experiments, because that doesn't take into account the weighting.

Thanks for any suggestions.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Drawing from survey sampling methods, I might look at estimating the $p_{est}$ as you've already done but then estimate the standard error by treating those trials within one experiment as a cluster. It would be a simple one-stage cluster where your clusters are the individual experiments and they need not be balanced. Once you have your estimate $\hat{p}_{est}$ and your standard error of $\hat{p}_{est}$, you can generate your confidence interval in the usual $\hat{p}_{est}\pm t^*\times SE[\hat{p}_{est}]$. However the calculation of the standard error will be different from your typical calculation, as you lose a bit of precision due to the clusters. The positive side is that you accurately capture those different experiments.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your suggestion. Can you point me to a good textbook or online reference that describes how to calculate an estimated standard error for unbalanced clusters? I found several different formulae online, but I honestly don't know enough to evaluate which ones are applicable to this kind of situation. $\endgroup$
    – Dan
    Commented Oct 23, 2015 at 12:55
  • $\begingroup$ On this website (stattrek.com/survey-research/…), there is a formula about halfway down the page that deals with one-stage cluster samples. Your estimate should be $\hat{p}_{est} = \frac{N}{n*M}\sum{M_i*P_i}$ and your standard error should be $SE[\hat{p}_{est}] = \frac{1}{M}*\sqrt{\frac{N^2*(1-n/N)}{n}*\frac{(\sum{M_i*P_i-\bar{t}_{prop}/N})^2}{n-1}}$. The website will specify what each of the variables means (there isn't enough room here). Once you get your estimate and your standard error, you can generate the CI as you generally would. $\endgroup$
    – Matt Brems
    Commented Oct 23, 2015 at 13:46
  • $\begingroup$ In this formula for standard error I am unsure what value to use for the population size N, since my experiments are not samples from a fixed population size. I could set N=n (the actual number of experiments I performed), but then the first term under the radical evaluates to zero (because n/N = 1). Alternatively I could assume N is very large (I could in theory perform infinitely many experiments if I had an infinite amount of time). If N>>n then it looks like all of the N terms will cancel (n/N ~= 0, and N^2 in the first term cancels with 1/N^2 in the second term). Is this correct? $\endgroup$
    – Dan
    Commented Oct 23, 2015 at 14:55
  • $\begingroup$ I would opt for assuming N is very large. For computational purposes, you might let N = 100000000 (or something similar) but it may be better to analytically derive what happens when N is very large. At first glance, the summation on the right-hand side of the square root argument may be a bit trickier than simply canceling out. $\endgroup$
    – Matt Brems
    Commented Oct 23, 2015 at 15:28
  • 1
    $\begingroup$ You're right, the N's don't cancel in a straightforward way, and SE ends up scaling as N when N>>n. If N is infinite, then SE is also infinite, which is clearly not correct. What I ended up doing was using the formula given here and here to calculate a weighted variance, from which I derive a confidence interval in the usual way. This is the best I've come up with after a lot of research, but I'm very much interested if someone knows of a better way to do it. $\endgroup$
    – Dan
    Commented Oct 23, 2015 at 19:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.