A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence
- PMID: 37089997
- PMCID: PMC9978628
- DOI: 10.1177/26334895211002474
A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence
Abstract
Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out.
Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) "cited-by" searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed.
Results: Measures were identified in only seven constructs: Structural characteristics (n = 13), Networks and communication (n = 29), Engaging (n = 1), Opinion leaders (n = 5), Champions (n = 5), Planning (n = 5), and Reflecting and evaluating (n = 5). No quantitative assessment measures of Formally appointed implementation leaders, External change agents, or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from "-2" to "10" out of a total of "36."
Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures.
Plain language summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention's use in practice and how to take the intervention to scale. Unfortunately, measures of these factors-stakeholders, their networks and communication, and their implementation activities-have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.
Keywords: Implementation science; change agents; communications; measure; networks; opinion leaders; process; reliability; structural characteristics; validity.
© The Author(s) 2021.
Conflict of interest statement
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Figures
Similar articles
-
Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review.Implement Res Pract. 2021 Jun 21;2:26334895211018862. doi: 10.1177/26334895211018862. eCollection 2021 Jan-Dec. Implement Res Pract. 2021. PMID: 37090009 Free PMC article. Review.
-
Measuring characteristics of individuals: An updated systematic review of instruments' psychometric properties.Implement Res Pract. 2021 Mar 23;2:26334895211000458. doi: 10.1177/26334895211000458. eCollection 2021 Jan-Dec. Implement Res Pract. 2021. PMID: 37090010 Free PMC article. Review.
-
Measuring readiness for implementation: A systematic review of measures' psychometric and pragmatic properties.Implement Res Pract. 2020 Aug 26;1:2633489520933896. doi: 10.1177/2633489520933896. eCollection 2020 Jan-Dec. Implement Res Pract. 2020. PMID: 37089124 Free PMC article. Review.
-
Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review.Implement Res Pract. 2021 Mar 23;2:2633489521994197. doi: 10.1177/2633489521994197. eCollection 2021 Jan-Dec. Implement Res Pract. 2021. PMID: 37090003 Free PMC article. Review.
-
Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
Cited by
-
Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review.Implement Res Pract. 2022 Dec 4;3:26334895221141116. doi: 10.1177/26334895221141116. eCollection 2022 Jan-Dec. Implement Res Pract. 2022. PMID: 37091091 Free PMC article. Review.
-
A pragmatic context assessment tool (pCAT): using a Think Aloud method to develop an assessment of contextual barriers to change.Implement Sci Commun. 2023 Jan 11;4(1):3. doi: 10.1186/s43058-022-00380-5. Implement Sci Commun. 2023. PMID: 36631914 Free PMC article.
References
-
- American Psychological Association. (2020). Construct. https://dictionary.apa.org/construct
Publication types
LinkOut - more resources
Full Text Sources