Skip to main content

Advertisement

SpringerLink
Log in

The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Predictors of disagreement varied by domain; significant predictors indexed higher clinical severity or social disadvantage. Results support the CASA as a reliable and robust assessment of pediatric service use, but administrators should be alert when assessing vulnerable populations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Angold, A., Costello, E. J., Farmer, E. M. Z., Burns, B. J., & Erkanli, A. (1999). Impaired but undiagnosed. Journal of the American Academy of Child & Adolescent Psychiatry, 38, 129–137. https://doi.org/10.1097/00004583-199902000-00011.

    Article  CAS  Google Scholar 

  • Angold, A., Costello, E. J., Pickles, A., Winder, F., & Silver, D. (1987). The development of a questionnaire for use in epidemiological studies of depression in children and adolescents. Unpublished manuscript, London University.

  • Ascher, B. H., Farmer, E. M. Z., Burns, B. J., & Angold, A. (1996). The child and adolescent services assessment (CASA): Description and psychometrics. Journal of Emotional and Behavioral Disorders, 4, 12–20. https://doi.org/10.1177/106342669600400102.

    Article  Google Scholar 

  • Beardslee, W. R., Brent, D. A., Weersing, V. R., Clarke, G. N., Porta, G., Hollon, S. D.,… Garber, J. (2013). Prevention of depression in at-risk adolescents: Longer-term effects. JAMA Psychiatry, 70, 1161–1170. https://doi.org/10.1001/jamapsychiatry.2013.295.

    Article  PubMed  PubMed Central  Google Scholar 

  • Birmaher, B., Khetarpal, S., Brent, D., Cully, M., Balach, L., Kaufman, J. & Neer, S. M. (1997). The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 545–553. https://doi.org/10.1097/00004583-199704000-00018.

    Article  CAS  Google Scholar 

  • Burns, B. J., Angold, A., & Costello, E. J. (1992). Measuring child, adolescent, and family service use. In L. Bickman & D. J. Rog (Eds.), Evaluating mental health services for children (pp. 17–29). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Burns, B. J., Costello, E. J., Angold, A., Tweed, D., Stangle, D., Farmer, E. M. Z., & Erkanli, A. (1995). Children’s mental health service use across service sectors. Health Affairs, 14, 147–159. https://doi.org/10.1377/hlthaff.14.3.147.

    Article  CAS  PubMed  Google Scholar 

  • Burns, B. J., Phillips, S. D., Wagner, H. R., Barth, R. P., Kolko, D. J., Campbell, Y., & Landsverk, J. (2004). Mental health need and access to mental health services by youths involved with child welfare: A national survey. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 960–970. https://doi.org/10.1097/01.chi.0000127590.95585.65.

    Article  Google Scholar 

  • Bussing, R., Mason, D. M., Leon, C. E., & Sinha, K. (2003). Agreement between CASA parent reports and provider records of children’s ADHD services. The Journal of Behavioral Health Services & Research, 30, 462–469. https://doi.org/10.1007/BF02287433.

    Article  Google Scholar 

  • Corral, I., & Landrine, H. (2010). Methodological and statistical issues in research with diverse samples: The problem of measurement equivalence. In H. Landrine & N. F. Russo (Eds.), Handbook of diversity in feminist psychology (pp. 83–121). New York: Springer.

    Google Scholar 

  • Costello, E. J., Angold, A., Burns, B. J., Erkanli, A., Stangl, D. K., & Tweed, D. L. (1996). The great smoky mountains study of youth: Functional impairment and serious emotional disturbance. Archives of General Psychiatry, 53, 1137–1143. https://doi.org/10.1001/archpsyc.1996.01830120077013.

    Article  CAS  PubMed  Google Scholar 

  • Costello, E. J., Burns, B. J., Costello, A. J., Edelbrock, C., Dulcan, M., & Brent, D. (1988). Service utilization and psychiatric diagnosis in pediatric primary care: The role of the gatekeeper. Pediatrics, 82, 435–441.

    CAS  PubMed  Google Scholar 

  • Costello, E. J., He, J., Sampson, N. A., Kessler, R. C., & Merikangas, K. R. (2014). Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey-Adolescent. Psychiatric Services, 65, 359–366. https://doi.org/10.1176/appi.ps.201100518.

    Article  PubMed  Google Scholar 

  • Domino, M. E., Burns, B. J., Silva, S. G., Kratochvil, C. J., Vitiello, B., Reinecke, M. A.,… March, J. S. (2008). Cost-effectiveness of treatments for adolescent depression: Results from TADS. American Journal of Psychiatry, 165, 588–596. https://doi.org/10.1176/appi.ajp.2008.07101610.

    Article  PubMed  Google Scholar 

  • Domino, M. E., Foster, E. M., Vitiello, B., Kratochvil, C. J., Burns, B. J., Silva, S. G.,… March, J. S. (2009). Relative cost-effectiveness of treatments for adolescent depression: 36-Week results from the TADS randomized trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 711–720. https://doi.org/10.1097/CHI.0b013e3181a2b319.

    Article  Google Scholar 

  • Farmer, E. M. Z., Angold, A., Burns, B. J., & Costello, E. J. (1994). Reliability of self-reported service use: Test-retest consistency of children’s responses to the Child and Adolescent Services Assessment (CASA). Journal of Child and Family Studies, 3, 307–325. https://doi.org/10.1007/BF02234688.

    Article  Google Scholar 

  • Farmer, E. M. Z., Burns, B. J., Angold, A., & Costello, E. J. (1997). Impact of children’s mental health problems on families: Relationships with service use. Journal of Emotional and Behavioral Disorders, 5, 230–238. https://doi.org/10.1177/106342669700500406.

    Article  Google Scholar 

  • Farmer, E. M. Z., Mustillo, S. A., Wagner, H. R., Burns, B. J., Kolko, D. J., Barth, R. P., & Leslie, L. K. (2010). Service use and multi-sector use for mental health problems by youth in contact with child welfare. Children and Youth Services Review, 32, 815–821. https://doi.org/10.1016/j.childyouth.2010.01.019.

    Article  Google Scholar 

  • Farmer, E. M. Z., Stangl, D. K., Burns, B. J., Costello, E. J., & Angold, A. (1999). Use, persistence, and intensity: Patterns of care for children’s mental health across one year. Community Mental Health Journal, 35, 31–46.

    Article  CAS  Google Scholar 

  • Garber, J., Clarke, G. N., Weersing, V. R., Beardslee, W. R., Brent, D. A., Gladstone, T. R. G.,… Iyengar, S. (2009). Prevention of depression in at-risk adolescents: A randomized controlled trial. JAMA, 301, 2215–2224.

    Article  CAS  Google Scholar 

  • Geisinger, K. F. (2013). Reliability. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 21–42). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Ghandour, R. M., Kogan, M. D., Blumberg, S. J., Jones, J. R., & Perrin, J. M. (2012). Mental health conditions among school-aged children: Geographic and sociodemographic patterns in prevalence and treatment. Journal of Developmental & Behavioral Pediatrics, 33, 42–54. https://doi.org/10.1097/DBP.0b013e31823e18fd.

    Article  Google Scholar 

  • Griffin, J. A., Cicchetti, D., & Leaf, P. J. (1993). Characteristics of youths identified from a psychiatric case register as first-time users of services. Hospital and Community Psychiatry, 44, 62–65. https://doi.org/10.1176/ps.44.1.62.

    Article  CAS  PubMed  Google Scholar 

  • Guy, W. (1976). Clinical global impression scale. Rockville, MD: National Institute of Mental Health.

    Google Scholar 

  • Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34.

    Article  Google Scholar 

  • Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329–343.

    Google Scholar 

  • Hurlburt, M. S., Leslie, L. K., Landsverk, J., Barth, R. P., Burns, B. J., Gibbons, R. D.,… Zhang, J. (2004). Contextual predictors of mental health service use among children open to child welfare. Archives of General Psychiatry, 61, 1217–1224. https://doi.org/10.1001/archpsyc.61.12.1217.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hwang, W., Myers, H. F., Abe-Kim, J., & Ting, J. Y. (2008). A conceptual paradigm for understanding culture’s impact on mental health: The cultural influences on mental health (CIMH) model. Clinical Psychology Review, 28, 211–227. https://doi.org/10.1016/j.cpr.2007.05.001.

    Article  PubMed  Google Scholar 

  • Jensen, P. S., Hinshaw, S. P., Swanson, J. M., Greenhill, L. L., Conners, C. K., Arnold, L. E.,… Wigal, T. (2001). Findings from the NIMH Multimodal Treatment Study of ADHD (MTA): Implications and applications for primary care providers. Developmental and Behavioral Pediatrics, 22, 60–73.

    Article  CAS  Google Scholar 

  • Kazdin, A. E., Holland, L., Crowley, M., & Breton, S. (1997). Barriers to treatment participation scale: Evaluation and validation in the context of child outpatient treatment. Journal of Child Psychology and Psychiatry, 38, 1051–1062. https://doi.org/10.1111/j.1469-7610.1997.tb01621.x.

    Article  CAS  PubMed  Google Scholar 

  • Kutner, M. H., Nachtsheim, C. J., & Neter, J. (2004). Applied linear regression models (4th edn.). Irwin: McGraw-Hill.

    Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://doi.org/10.2307/2529310.

    Article  CAS  Google Scholar 

  • Leslie, L. K., Hurlburt, M. S., James, S., Landsverk, J., Slymen, D. J., & Zhang, J. (2005). Relationship between entry into child welfare and mental health service use. Psychiatric Services, 56, 981–987. https://doi.org/10.1176/appi.ps.56.8.981.

    Article  PubMed  Google Scholar 

  • Lynch, F. L., Dickerson, J. F., Clarke, G., Vitiello, B., Porta, G., Wagner, K. D., … Brent, D. (2011). Incremental cost-effectiveness of combined therapy vs medication only for youth with Selective Serotonin Reuptake Inhibitor–resistant depression: Treatment of SSRI-Resistant Depression in Adolescents Trial findings. Archives of General Psychiatry, 68, 253–262.

    Article  Google Scholar 

  • Mayr, U., & Kliegl, R. (2000). Task-set switching and long-term memory retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 5, 1124–1140.

    Google Scholar 

  • Merikangas, K. R., He, J., Burstein, M. E., Swendsen, J., Avenevoli, S., Case, B.,… Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the National Comorbidity Survey Adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 50, 32–45. https://doi.org/10.1016/j.jaac.2010.10.006.

    Article  Google Scholar 

  • Nilchaikovit, T., Hill, J. M., & Holland, J. C. (1993). The effects of culture on illness behavior and medical care. Asian and American differences. General Hospital Psychiatry, 15, 41–50.

    Article  CAS  Google Scholar 

  • Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.

    Article  Google Scholar 

  • Sanchez, A. L., Cornacchio, D., Poznanski, B., Golik, A. M., Chou, T., & Comer, J. S. (2018). The effectiveness of school-based mental health services for elementary-aged children: A meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 57, 153–165. https://doi.org/10.1016/j.jaac.2017.11.022.

    Article  Google Scholar 

  • Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. https://doi.org/10.1037//0003-066X.54.2.93.

    Article  Google Scholar 

  • Simon, A. E., Pastor, P. N., Reuben, C. A., Huang, L. N., & Goldstrom, I. D. (2015). Use of mental health services by children ages six to 11 with emotional or behavioral difficulties. Psychiatric Services, 66, 930–937. https://doi.org/10.1176/appi.ps.201400342.

    Article  PubMed  Google Scholar 

  • Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the state-trait anxiety inventory. Palo Alto, CA: Consulting Psychologists Press.

    Google Scholar 

  • U.S. Department of Health and Human Services. (2001). Mental health: Culture, race, and ethnicity—a supplement to mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.

    Book  Google Scholar 

  • Walker, L. S., & Garber, J. (2003). Manual for the children’s somatization inventory. Nashville, TN: Vanderbilt University Medical Center.

    Google Scholar 

  • Weersing, V. R., Brent, D. A., Rozenman, M. S., Gonzalez, A., Jeffreys, M., Dickerson, J. F.,… Iyengar, S. (2017). Brief behavioral therapy for pediatric anxiety and depression in primary care: A randomized clinical trial. JAMA Psychiatry. https://doi.org/10.1001/jamapsychiatry.2017.0429.

    Article  PubMed  PubMed Central  Google Scholar 

  • Weersing, V. R., Gonzalez, A., Campo, J. V., & Lucas, A. N. (2008). Brief behavioral therapy for pediatric anxiety and depression: Piloting an integrated treatment approach. Cognitive and Behavioral Practice, 15, 126–139.

    Article  Google Scholar 

  • Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671–689. https://doi.org/10.1037/0003-066X.61.7.671.

    Article  PubMed  Google Scholar 

  • Wilk, M. B., & Gnanadesikan, R. (1968). Probability plotting methods for the analysis of data. Biometrika, 55, 1–17. https://doi.org/10.1093/biomet/55.1.1.

    Article  CAS  PubMed  Google Scholar 

  • Wood, A., Kroll, L., Moore, A., & Harrington, R. (1995). Properties of the mood and feelings questionnaire in adolescent psychiatric outpatients: A research note. Child Psychology & Psychiatry & Allied Disciplines, 36, 327–334.

    Article  CAS  Google Scholar 

Download references

Funding

This study was funded by the National Institute of Mental Health (5R01MH084935; 5R01MH084916); however, the funding source had no role in any aspect of study creation, implementation, or preparation of this manuscript.

Author information

Authors and Affiliations

  1. San Diego State University/University of California, San Diego Joint Doctoral Program in Clinical Psychology, 6363 Alvarado Court, Suite 103, San Diego, CA, 92120, USA

    Karen T. G. Schwartz & V. Robin Weersing

  2. Department of Psychology, San Diego State University, San Diego, CA, USA

    Amanda A. Bowling & V. Robin Weersing

  3. Center for Health Research, Kaiser Permanente Northwest, Portland, OR, USA

    John F. Dickerson & Frances L. Lynch

  4. Department of Psychiatry, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA

    David A. Brent

  5. Western Psychiatric Institute and Clinic, University of Pittsburgh Medical Center, Pittsburgh, PA, USA

    David A. Brent & Giovanna Porta

  6. Department of Statistics, University of Pittsburgh, Pittsburgh, PA, USA

    Satish Iyengar

Authors
  1. Karen T. G. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amanda A. Bowling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John F. Dickerson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Frances L. Lynch
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David A. Brent
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Giovanna Porta
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Satish Iyengar
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. V. Robin Weersing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karen T. G. Schwartz.

Ethics declarations

Conflict of interest

Dr. Brent receives research support from NIMH, royalties from Guilford Press, royalties from the electronic self-rated version of the C-SSRS from ERT, Inc., royalties from performing duties as an UptoDate Psychiatry Section Editor, and consulting fees from Healthwise. He has no other conflicts of interest to report. Ms. Schwartz, Ms. Bowling, Dr. Dickerson, Dr. Lynch, Ms. Porta, Dr. Iyengar and Dr. Weersing declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent and assent were obtained from all adult and youth participants, respectively, included in the study, in accordance with the San Diego State University and University of California, San Diego institutional review boards.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schwartz, K.T.G., Bowling, A.A., Dickerson, J.F. et al. The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement. Adm Policy Ment Health 45, 944–957 (2018). https://doi.org/10.1007/s10488-018-0876-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-018-0876-8

Keywords

Search

Navigation