The DSM‐IV diagnoses generated by the fully structured lay‐administered Composite International Diagnostic Interview Version 3.0 (CIDI 3.0) in the WHO World Mental Health (WMH) surveys were compared to diagnoses based on follow‐up interviews with the clinician‐administered non‐patient edition of the Structured Clinical Interview for DSM‐IV (SCID) in probability subsamples of the WMH surveys in France, Italy, Spain, and the US. CIDI cases were oversampled. The clinical reappraisal samples were weighted to adjust for this oversampling. Separate samples were assessed for lifetime and 12‐month prevalence. Moderate to good individual‐level CIDI‐SCID concordance was found for lifetime prevalence estimates of most disorders. The area under the ROC curve (AUC, a measure of classification accuracy that is not influenced by disorder prevalence) was 0.76 for the dichotomous classification of having any of the lifetime DSM‐IV anxiety, mood and substance disorders assessed in the surveys and in the range 0.62–0.93 for individual disorders, with an inter‐quartile range (IQR) of 0.71–0.86. Concordance increased when CIDI symptom‐level data were added to predict SCID diagnoses in logistic regression equations. AUC for individual disorders in these equations was in the range 0.74–0.99, with an IQR of 0.87–0.96. CIDI lifetime prevalence estimates were generally conservative relative to SCID estimates. CIDI‐SCID concordance for 12‐month prevalence estimates could be studied powerfully only for two disorder classes, any anxiety disorder (AUC = 0.88) and any mood disorder (AUC = 0.83). As with lifetime prevalence, 12‐month concordance improved when CIDI symptom‐level data were added to predict SCID diagnoses. CIDI 12‐month prevalence estimates were unbiased relative to SCID estimates. The validity of the CIDI is likely to be under‐estimated in these comparisons due to the fact that the reliability of the SCID diagnoses, which is presumably less than perfect, sets a ceiling on maximum CIDI‐SCID concordance. Copyright � 2006 John Wiley & Sons, Ltd.