Cochran's Q Test#

Cochran's Q test is the extension of the McNemar test to three or more related samples with a binary dependent variable (yes/no, pass/fail, correct/incorrect). When you want to know whether the success rate of the same individuals differs across multiple conditions, Cochran's Q is your test.

When to Use#

• You have three or more conditions and the same individuals are measured under each condition
• Your dependent variable is binary (0/1, yes/no, pass/fail)
• You want to test whether the proportions (e.g., pass rates) differ across conditions
• You have a repeated measures design with categorical data — Cochran's Q is the nonparametric counterpart to RM-ANOVA for binary outcomes
• With only two conditions, use the McNemar test instead

Assumptions#

• Binary (dichotomous) dependent variable (0 or 1)
• Related samples — the same individuals under all conditions
• Sufficiently large sample size (rule of thumb: n x k >= 24)

Tip: Cochran's Q is an omnibus test — it only tells you that at least one condition differs, not which ones. For pairwise comparisons, use post-hoc McNemar tests with Bonferroni correction.

Formula#

The test statistic Q approximately follows a chi-squared distribution with $k - 1$ degrees of freedom:

$Q = \frac{k(k-1) \sum_{j=1}^{k}(G_j - \bar{G})^2}{k \sum_{i=1}^{n} R_i - \sum_{i=1}^{n} R_i^2}$

Where:

• $k$ = number of conditions
• $G_j$ = sum of successes in condition $j$
• $\bar{G}$ = mean of column totals
• $R_i$ = row sum (number of successes) for individual $i$

Example#

Effect Size#

The Cochran's Q coefficient can be reported as an effect size, normalizing the Q value:

$\text{Effect size} = \frac{Q}{n(k-1)}$

This value ranges from 0 to 1. Alternatively, Serlin's S can be used, which indicates the proportion of the maximum possible Q value.

Tip: Always conduct post-hoc tests when the result is significant. Without them, you don't know which conditions actually differ.