LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
Authors:
Tongshuang Wu,
Haiyi Zhu,
Maya Albayrak,
Alexis Axon,
Amanda Bertsch,
Wenxing Deng,
Ziqi Ding,
Bill Guo,
Sireesh Gururaja,
Tzu-Sheng Kuo,
Jenny T. Liang,
Ryan Liu,
Ihita Mandal,
Jeremiah Milbauer,
Xiaolin Ni,
Namrata Padmanabhan,
Subhashini Ramkumar,
Alexis Sudjianto,
Jordan Taylor,
Ying-Jui Tseng,
Patricia Vaidos,
Zhijin Wu,
Wei Wu,
Chenyang Yang
Abstract:
LLMs have shown promise in replicating human-like behavior in crowdsourcing tasks that were previously thought to be exclusive to human abilities. However, current efforts focus mainly on simple atomic tasks. We explore whether LLMs can replicate more complex crowdsourcing pipelines. We find that modern LLMs can simulate some of crowdworkers' abilities in these "human computation algorithms," but…
▽ More
LLMs have shown promise in replicating human-like behavior in crowdsourcing tasks that were previously thought to be exclusive to human abilities. However, current efforts focus mainly on simple atomic tasks. We explore whether LLMs can replicate more complex crowdsourcing pipelines. We find that modern LLMs can simulate some of crowdworkers' abilities in these "human computation algorithms," but the level of success is variable and influenced by requesters' understanding of LLM capabilities, the specific skills required for sub-tasks, and the optimal interaction modality for performing these sub-tasks. We reflect on human and LLMs' different sensitivities to instructions, stress the importance of enabling human-facing safeguards for LLMs, and discuss the potential of training humans and LLMs with complementary skill sets. Crucially, we show that replicating crowdsourcing pipelines offers a valuable platform to investigate (1) the relative strengths of LLMs on different tasks (by cross-comparing their performances on sub-tasks) and (2) LLMs' potential in complex tasks, where they can complete part of the tasks while leaving others to humans.
△ Less
Submitted 19 July, 2023; v1 submitted 19 July, 2023;
originally announced July 2023.
Investigating Girls' Perspectives and Knowledge Gaps on Ethics and Fairness in Artificial Intelligence in a Lightweight Workshop
Authors:
Jaemarie Solyst,
Alexis Axon,
Angela E. B. Stewart,
Motahhare Eslami,
Amy Ogan
Abstract:
Artificial intelligence (AI) is everywhere, with many children having increased exposure to AI technologies in daily life. We aimed to understand middle school girls' (a group often excluded group in tech) perceptions and knowledge gaps about AI. We created and explored the feasibility of a lightweight (less than 3 hours) educational workshop in which learners considered challenges in their lives…
▽ More
Artificial intelligence (AI) is everywhere, with many children having increased exposure to AI technologies in daily life. We aimed to understand middle school girls' (a group often excluded group in tech) perceptions and knowledge gaps about AI. We created and explored the feasibility of a lightweight (less than 3 hours) educational workshop in which learners considered challenges in their lives and communities and critically considered how existing and future AI could have an impact. After the workshop, learners had nuanced perceptions of AI, understanding AI can both help and harm. We discuss design implications for creating educational experiences in AI and fairness that embolden learners.
△ Less
Submitted 27 February, 2023;
originally announced February 2023.
