Linda Raftree

Large language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text-based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep-learning AI techniques, where language models are trained on extremely large text data from the… Show more

Large language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text-based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep-learning AI techniques, where language models are trained on extremely large text data from the internet and then re-used for downstream tasks with limited fine-tuning required. They offer exciting opportunities for evaluators to automate and accelerate time-consuming tasks involving text analytics and text generation. We estimate that over two-thirds of evaluation tasks will be affected by LLMs in the next 5 years. Use-case examples include summarizing text data, extracting key information from text, analyzing and classifying text content, writing text, and translation. Despite the advances, the technologies pose significant challenges and risks. Because LLM technologies are generally trained on text from the internet, they tend to perpetuate biases (racism, sexism, ethnocentrism, and more) and exclusion of non-majority languages. Current tools like ChatGPT have not been specifically developed for monitoring, evaluation, research, and learning (MERL) purposes, possibly limiting their accuracy and usefulness for evaluation. In addition, technical limitations and challenges with bias can lead to real world harm. To overcome these technical challenges and ethical risks, the evaluation community will need to work collaboratively with the data science community to co-develop tools and processes and to ensure the application of quality and ethical standards. Show less

In this paper, we explore how digital approaches and interventions could be safely and feasibly incorporated into the different layers of the World Health Organization’s MHPSS Pyramid (used by a majority of humanitarian organizations to frame their MHPSS work). Based on this exploration, we suggest an adapted MHPSS Pyramid that includes potential digital interventions with some of their benefits and drawbacks for adolescents in forced displacement contexts. We also provide a summary of the… Show more

In this paper, we explore how digital approaches and interventions could be safely and feasibly incorporated into the different layers of the World Health Organization’s MHPSS Pyramid (used by a majority of humanitarian organizations to frame their MHPSS work). Based on this exploration, we suggest an adapted MHPSS Pyramid that includes potential digital interventions with some of their benefits and drawbacks for adolescents in forced displacement contexts. We also provide a summary of the existing evidence base and inputs from UNHCR staff to lay out the case and the caveats for digital MHPSS for forcibly displaced and stateless adolescents. This leads us to a set of core insights into the key benefits of digital MHPSS at the different levels of the MHPSS Pyramid alongside the barriers, limitations, and risks.

Following the exploration of these benefits and challenges, we highlight good practices for designing and
implementing digital MHPSS programming with forcibly displaced and stateless adolescents and make
recommendations for further action by UNHCR at strategic, advocacy, policy, monitoring, evaluation,
research, operational, and guidance levels. Rounding off the report is a checklist for practitioners to follow when designing and implementing digital MHPSS approaches and interventions. Show less

The COVID-19 pandemic led to an increase in the use of digital platforms at the individual and organizational level. This affected how organizations conduct social and behavior change communication (SBCC) efforts. In this paper we explore how the pre-pandemic digital SBCC experiences and strategies of four organizations served as an asset for addressing emerging pandemic realities. The report can guide others who are working in MERL of SBCC and adjacent fields.

Part 1: Overview of Data Governance focuses on theoretical aspects of data governance with particular emphasis on personal and sensitive data, Africa’s data ecosystem, and the M&E practices of various actors involved. It considers the African contextual reality in which no single, common law exists to govern data practices and states have varying levels of data capacity and regulation. Finally it summarizes privacy-related debates in the framework of M&E and the data revolution and points… Show more

Part 1: Overview of Data Governance focuses on theoretical aspects of data governance with particular emphasis on personal and sensitive data, Africa’s data ecosystem, and the M&E practices of various actors involved. It considers the African contextual reality in which no single, common law exists to govern data practices and states have varying levels of data capacity and regulation. Finally it summarizes privacy-related debates in the framework of M&E and the data revolution and points towards alternative data governance models that might be suited for the continent. https://merltech.org/wp-content/uploads/2022/01/Responsible-Data-Governance-for-ME_part-1.pdf

Part 2: Practical Guidance on Responsible Data Governance in M&E provides orientation for practitioners on responsible data governance of personal and sensitive data in Africa in the context of monitoring and evaluation. Following the ‘data lifecycle,’ it offers tips and guidance as well as case studies to support M&E professionals to manage data responsibly in their work. https://merltech.org/wp-content/uploads/2022/01/Responsible-Data-Governance-for-ME_part-2.pdf Show less

Humanitarians have often used remote monitoring and evaluation (M&E) approaches when working in complex, fragile and conflict-affected environments. However, COVID-19 has impacted M&E practice in a large and unexpected way, with global travel restrictions and other measures to reduce the spread of the virus leading to a surge in the use of remote digital M&E methods over 2020 and 2021. While some of the remote mechanisms may just be temporary, others may become integrated into existing… Show more

Humanitarians have often used remote monitoring and evaluation (M&E) approaches when working in complex, fragile and conflict-affected environments. However, COVID-19 has impacted M&E practice in a large and unexpected way, with global travel restrictions and other measures to reduce the spread of the virus leading to a surge in the use of remote digital M&E methods over 2020 and 2021. While some of the remote mechanisms may just be temporary, others may become integrated into existing approaches or used as additional ways to gather data for M&E in the long term. This paper explores some of the implications of the rapid shift towards remote M&E, drawing on reflections from a series of M&E Skills-Building Workshops hosted by ALNAP in February 2021 for humanitarian M&E practitioners. It summarises five key areas where there are ethical issues, challenges, gaps and limitations with remote M&E, and highlights emerging good practice. Show less

This Manifesto includes key action points and a call for a governance model purposefully designed to deliver on the needs and rights of children. It is the first step in ensuring that children’s rights are given due weight in data governance legal frameworks and processes as they evolve around the world.

Challenges due to the COVID-19 pandemic have led to all kinds of innovations and adaptations focused on remote targeting, enrollment, verification and delivery. This case study explores digital remote targeting approaches used by GiveDirectly within cash programming, and the ways the organization is addressing data responsibility.

The growing emphasis on linking humanitarian cash and voucher assistance and government-led social protection systems has contributed to an increase in concerns about data sharing with governments. This case study addresses these commonly held concerns, with a particular focus on fragile settings or conflict environments. It looks at different strategies that Cash and Voucher Assistance (CVA) actors could implement to mitigate real and potential harms that sharing CVA beneficiary data with… Show more

The growing emphasis on linking humanitarian cash and voucher assistance and government-led social protection systems has contributed to an increase in concerns about data sharing with governments. This case study addresses these commonly held concerns, with a particular focus on fragile settings or conflict environments. It looks at different strategies that Cash and Voucher Assistance (CVA) actors could implement to mitigate real and potential harms that sharing CVA beneficiary data with governments could cause for crisis-affected populations. Show less

This updated toolkit builds on ELAN’s widely-used 2016 Data Starter Kit, which was one of the first resources providing orientation on data privacy and security in humanitarian work. The toolkit is structured according to the stages of the data lifecycle and split into 7 tipsheets which address issues of Context and Organizational Capacity; Design and Planning; and Implementation. The toolkit, offers a range of ways to work data responsibility into program planning, design, implementation and… Show more

This updated toolkit builds on ELAN’s widely-used 2016 Data Starter Kit, which was one of the first resources providing orientation on data privacy and security in humanitarian work. The toolkit is structured according to the stages of the data lifecycle and split into 7 tipsheets which address issues of Context and Organizational Capacity; Design and Planning; and Implementation. The toolkit, offers a range of ways to work data responsibility into program planning, design, implementation and MEAL (monitoring, evaluation, accountability and learning) activities related to Cash and Voucher Assistance (CVA). Show less

This study found 4 key risks for migrating and displaced children when agencies introduce programming that involves digital innovation and digital technology. These are: exclusion and self exclusion, harm cause by humanitarian innovation, increased exposure to online harms, and data misuse or mishandling. The study also explores how the sector is responding to child safeguarding risks posed by digital technology and offers recommendations for immediate and practical next steps to ensure that… Show more

This study found 4 key risks for migrating and displaced children when agencies introduce programming that involves digital innovation and digital technology. These are: exclusion and self exclusion, harm cause by humanitarian innovation, increased exposure to online harms, and data misuse or mishandling. The study also explores how the sector is responding to child safeguarding risks posed by digital technology and offers recommendations for immediate and practical next steps to ensure that migrant or displaced children can benefit from digital innovation while staying safe. Show less

Even before the COVID-19 pandemic, the world was hurtling down a path of rapid growth in technology. But since the pandemic took hold, technology has assumed an even more prominent role in children’s lives. In this brief, we offer a glimpse into challenges related to the protection of children’s data during the COVID-19 pandemic and highlight some key policy issues that need to be addressed. These challenges will continue into the foreseeable future unless we collectively articulate better that… Show more

Even before the COVID-19 pandemic, the world was hurtling down a path of rapid growth in technology. But since the pandemic took hold, technology has assumed an even more prominent role in children’s lives. In this brief, we offer a glimpse into challenges related to the protection of children’s data during the COVID-19 pandemic and highlight some key policy issues that need to be addressed. These challenges will continue into the foreseeable future unless we collectively articulate better that the world we want for children is one where their rights are respected, and data about them is safeguarded and is not exploited for commercial and other purposes. Show less

This paper explores the peer-reviewed evidence base of “traditional” technology-enabled monitoring, evaluation, research, and learning (MERL Tech) in international development assistance from 2015 to 2019. The authors conducted a scoping review that searched seven databases, screened 3,054 reference titles and abstracts, coded 886 abstracts, and extracted and analyzed conclusions and recommendations from the full texts of 256 studies. The findings reveal the most frequently reported… Show more

This paper explores the peer-reviewed evidence base of “traditional” technology-enabled monitoring, evaluation, research, and learning (MERL Tech) in international development assistance from 2015 to 2019. The authors conducted a scoping review that searched seven databases, screened 3,054 reference titles and abstracts, coded 886 abstracts, and extracted and analyzed conclusions and recommendations from the full texts of 256 studies. The findings reveal the most frequently reported technologies, MERL activities, and the sub-sectors, and the geographies where those tech-enabled activities occur. Gaps in the evidence for specific technologies, MERL activities, and sectors are mapped. The data reveals which technologies are trusted more than others and reported barriers to effective MERL Tech implementation and areas that researchers suggest for further investigation. The results suggest that the evidence from peer-reviewed studies is not proportional to estimated MERL Tech activity, significant publication bias exists, and further knowledge synthesis of unindexed grey literature is needed to provide a more comprehensive and possibly accurate description of MERL Tech practice. Show less

This paper probes trends in the use of big data by a community of early adopters working in monitoring, evaluation, research, and learning (MERL) in the development and humanitarian sectors. Qualitative analysis was conducted on data from MERL Tech conference records and key informant interviews. Findings indicate that MERL practitioners are in a fragmented, experimental phase, with use and application of big data varying widely, accompanied by shifting terminologies. We take an in-depth look… Show more

This paper probes trends in the use of big data by a community of early adopters working in monitoring, evaluation, research, and learning (MERL) in the development and humanitarian sectors. Qualitative analysis was conducted on data from MERL Tech conference records and key informant interviews. Findings indicate that MERL practitioners are in a fragmented, experimental phase, with use and application of big data varying widely, accompanied by shifting terminologies. We take an in-depth look at barriers to and enablers of use of big data within MERL, as well as benefits and drawbacks. Concerns about bias, privacy, and the potential for big data to magnify existing inequalities arose frequently. The research surfaced a need for more systematic and broader sharing of big data use cases and case studies in the development sector. Show less

This is the first paper in the State of the Field series on the role and impact of technology on monitoring, evaluation, research, and learning (MERL). The series provides collective knowledge on trends, issues and challenges in the MERL Tech field from 2014 to 2019, as well as opportunities for the MERL Tech community to responsibly advance the field. This series of four papers, explores three waves of innovation in MERL Tech. Where available, we based our findings on the academic literature… Show more

This is the first paper in the State of the Field series on the role and impact of technology on monitoring, evaluation, research, and learning (MERL). The series provides collective knowledge on trends, issues and challenges in the MERL Tech field from 2014 to 2019, as well as opportunities for the MERL Tech community to responsibly advance the field. This series of four papers, explores three waves of innovation in MERL Tech. Where available, we based our findings on the academic literature. Where the evidence base is weak or nonexistent, we have aimed to identify trends and possibilities, gaps in knowledge, and areas where more systematic analysis would help to define what is needed to achieve inclusive and people-centered MERL that benefits from technology, rather than MERL that is driven by technology and digital data. Show less

This episode of the podcadt tackles the issue of data responsibility in humanitarian cash and voucher assistance (CVA), exploring questions around informed consent, legitimate interest, ethics and regulations including GDPR. Featuring Amos Doornbos, Linda Raftree, James Eaton Lee and Ric Tighe.

A discussion with Kristy Crabtree from International Rescue Committee on digital ethics and what organizations should be considering in their digital humanitarian efforts.

The Responsible Data Maturity Model is aimed a providing organizations with a way to self-assess and plan for improvements in how they manage data responsibly. It was developed for CARE but can be used by any organization.

This article invited two experts to weigh in on the pros and cons of the use of biometrics for aid work.

This is a chapter written for a book titled: Information and Communication Technologies for Development Evaluation, published by the Independent Office of Evaluation of International Fund for Agricultural Development (IFAD). The book gives an insight into the implications of new and emerging technologies in development evaluation.

Description of the book: Growing technologies such as big data analytics, machine learning and remote sensing present new opportunities for development… Show more

This is a chapter written for a book titled: Information and Communication Technologies for Development Evaluation, published by the Independent Office of Evaluation of International Fund for Agricultural Development (IFAD). The book gives an insight into the implications of new and emerging technologies in development evaluation.

Description of the book: Growing technologies such as big data analytics, machine learning and remote sensing present new opportunities for development practitioners and development evaluators, particularly when measuring indicators of the Sustainable Development Goals. The volume provides an overview of information and communication technologies (ICTs) in the context of evaluation, looking at the theory and practice, and discussing how the landscape may unfold. It also considers concerns about privacy, ethics and inclusion, which are crucial issues for development practitioners and evaluators working in the interests of vulnerable populations across the globe. Among the contributions are case studies of seven organizations using various technologies for data collection, analysis, dissemination and learning. Show less

This document aims to provide USAID staff and local partners with a framework for identifying and understanding risks associated with development data. It is meant as a conversation starter—to highlight important concerns and provide actionable advice—to help those who use data in development programs maximize utility while also managing risk. By starting to have conversations around responsible data practices, staff and partners will begin to build competency in this area. USAID’s Journey to… Show more

This document aims to provide USAID staff and local partners with a framework for identifying and understanding risks associated with development data. It is meant as a conversation starter—to highlight important concerns and provide actionable advice—to help those who use data in development programs maximize utility while also managing risk. By starting to have conversations around responsible data practices, staff and partners will begin to build competency in this area. USAID’s Journey to Self Reliance includes supporting countries to build their own technological capacity and readiness by taking ownership of their data and being held accountable that it is kept safe. Show less

This paper explores the ways in which current and emerging digital and social media channels can be used to drive behaviour change in young men and women from lower socioeconomic groups around the world. Author Linda Raftree examined peer-reviewed literature, grey literature, non-governmental organisation (NGO) research and evaluation reports, industry framework and guidance documents, and articles and blog posts in order: to explore relevant social and behaviour change communication (SBCC)… Show more

This paper explores the ways in which current and emerging digital and social media channels can be used to drive behaviour change in young men and women from lower socioeconomic groups around the world. Author Linda Raftree examined peer-reviewed literature, grey literature, non-governmental organisation (NGO) research and evaluation reports, industry framework and guidance documents, and articles and blog posts in order: to explore relevant social and behaviour change communication (SBCC) theories, models, frameworks, and approaches; to uncover learning from current and past digital SBCC initiatives; to identify good practice and common pitfalls in digital SBCC design; to lay out good practices for digital safeguarding; and to provide a few examples of how monitoring and evaluation and learning (MEL) can be designed for SBCC initiatives. Show less

This document offers practical tips and guidelines for safeguarding children online digitally in Girl Effect’s programmes and platforms, including how to safeguard personal and/or sensitive data collected from or about children throughout the data lifecycle. It does not constitute legal advice in any way and you are encouraged to seek your own advice to ensure your own specific circumstances have been taken into consideration in how you approach digital safeguarding. The report is based on Girl… Show more

This document offers practical tips and guidelines for safeguarding children online digitally in Girl Effect’s programmes and platforms, including how to safeguard personal and/or sensitive data collected from or about children throughout the data lifecycle. It does not constitute legal advice in any way and you are encouraged to seek your own advice to ensure your own specific circumstances have been taken into consideration in how you approach digital safeguarding. The report is based on Girl Effect’s learning and experiences over the past 4 years and builds on an earlier Principles and Practices document published by Girl Effect in 2016. Show less

The Global Center for Youth Employment and Banyan Global joined forces to produce a study of microwork-impact sourcing and its potential to help address the global youth unemployment challenge. This study investigated a wide range of actors currently involved in microwork and impact sourcing, including service providers, clients, workers, and the enabling environment surrounding them. The research team analyzed more than 90 relevant documents, conducted 40 stakeholder consultations, visited… Show more

The Global Center for Youth Employment and Banyan Global joined forces to produce a study of microwork-impact sourcing and its potential to help address the global youth unemployment challenge. This study investigated a wide range of actors currently involved in microwork and impact sourcing, including service providers, clients, workers, and the enabling environment surrounding them. The research team analyzed more than 90 relevant documents, conducted 40 stakeholder consultations, visited multiple microwork and impact sourcing centers in Kenya, and sought to establish an in-depth understanding of the microwork-impact sourcing ecosystem. The study maps current trends, opportunities, challenges, and success factors surrounding the implementation of microwork-focused impact sourcing. Show less

This Guide is aimed at providing Girl Effect staff and partners with guidance on protecting girls who are using the digital tools that Girl Effect creates and/or promotes. It offers a framework to better protect girls’ personal information and privacy and to ensure that the content Girl Effect provides or sources does not put girls at risk. It lists the kinds of questions to ask partners, sponsors or vendors before engaging in partnership agreements, and helps Girl Effect consider how to go… Show more

This Guide is aimed at providing Girl Effect staff and partners with guidance on protecting girls who are using the digital tools that Girl Effect creates and/or promotes. It offers a framework to better protect girls’ personal information and privacy and to ensure that the content Girl Effect provides or sources does not put girls at risk. It lists the kinds of questions to ask partners, sponsors or vendors before engaging in partnership agreements, and helps Girl Effect consider how to go about
collecting, using, storing and disposing of data that is gathered as part of the organization's digital tools and platforms, as well as when conducting monitoring, evaluation, learning and research activities. Show less

ICT is being rapidly introduced into development evaluation. Evaluators need to keep abreast
of these important evolutions in the field in order to stay current and continue shaping the
future of development work through their practice. ICT is helping to resolve some age-old issues
that evaluators face, especially in terms of time, resources and data quality constraints. At the
same time, the introduction of ICTs into evaluation brings a series of new ethical, political… Show more

ICT is being rapidly introduced into development evaluation. Evaluators need to keep abreast
of these important evolutions in the field in order to stay current and continue shaping the
future of development work through their practice. ICT is helping to resolve some age-old issues
that evaluators face, especially in terms of time, resources and data quality constraints. At the
same time, the introduction of ICTs into evaluation brings a series of new ethical, political and
methodological challenges that evaluators need to address. The article also weighs the potential
opportunities for ICT to strengthen equity and empowerment-focused evaluation against concerns
that these new technologies could result in more ‘extractive’ approaches whereby it becomes
easier for international agencies and donors to collect information remotely thereby reducing the
ability of local communities and civil society to participate in the development dialogue. Show less

A podcast interview that covers career path; balancing family and marriage when a development career takes priority; disruption, revolutions and breakthroughs in the development field; and maintaining personal values, selectiveness at work and curiosity.

The report describes the impact of the documentary film: "Who is Dayani Cristal" at multiple levels, highlighting the importance of moving beyond views and 'clicks' to ensuring respectful inclusion of community members where the film was set and aiming to assess direct impact on their lives in the short and medium term and in immigration programming and policy at local and wider levels.

Article about the long-term detrimental effects of "poverty porn" on the dignity of communities that development organizations aim to support. The article is a summary of an on-line discussion hosted by Regarding Humanity and Kurante in September 2013. The discussion was summarized by Daniel Ramirez into a blogpost, edited by me and originally published on Wait What.

Emerging Opportunities: Monitoring and Evaluation in a Tech-Enabled World, a discussion paper written by Linda Raftree and Michael Bamberger under a grant from the Rockefeller Foundation to Itad provides an overview of how monitoring and evaluation (M&E) practice (including that of the Rockefeller Foundation’s M&E practice) is learning how to better use information and communication technologies (ICTs) for more timely data and more inclusive voice and feedback from beneficiaries and other… Show more

Emerging Opportunities: Monitoring and Evaluation in a Tech-Enabled World, a discussion paper written by Linda Raftree and Michael Bamberger under a grant from the Rockefeller Foundation to Itad provides an overview of how monitoring and evaluation (M&E) practice (including that of the Rockefeller Foundation’s M&E practice) is learning how to better use information and communication technologies (ICTs) for more timely data and more inclusive voice and feedback from beneficiaries and other stakeholders.

The increasing complexity of development coupled with the widening range of public, not-for-profit and private sector actors and the demand for more timely feedback has challenged the utility of conventional approaches to M&E in many development contexts. Though emerging ICTs offer promise, the methodological rigor of tech-enabled M&E has sometimes been questioned and viewed as unreliable in contemporary evaluation debates.

Despite this broad reluctance, M&E innovators are already experimenting in this new space. By reflecting on ways in which these innovators have begun to navigate new territory, and by exploring the great potential for technology to further transform and advance traditional evaluation methods, this paper aims to highlight the current state of tech-enabled M&E while also maintaining a critical perspective which recognizes the limitations and inherent risks which evaluators should remain mindful of when engaging in this new and exciting space.
Show less

The report looks at how children and youth are using ICTs before, during and after migration to make their journey and integration safer and more productive. It also identifies a number of trends in how civil society organizations are using ICTs in their work with child and youth migrants. Finally, the report offers a number of considerations for organizations who wish to integrate ICTs into their work with children and youth on the move.

Short article for the 2013 International Day of the Girl on making ICTs part of a holistic strategy to empower adolescent girls, which includes tackling gender discrimination, poverty and all the barriers that prevent girls from going to school.

The review maps out who is doing what and where, and to the extent possible, discusses evidence of what is working in the field of mobile technology and youth workforce development, including 5 core areas: education and training; demand-side programs and policies; career counseling; entrepreneurship and enterprise development; and overcoming social constraints.

Primary contributors to Chapter 3.