-
TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models
Authors:
Ishika Singh,
David Traum,
Jesse Thomason
Abstract:
Classical planning formulations like the Planning Domain Definition Language (PDDL) admit action sequences guaranteed to achieve a goal state given an initial state if any are possible. However, reasoning problems defined in PDDL do not capture temporal aspects of action taking, for example that two agents in the domain can execute an action simultaneously if postconditions of each do not interfer…
▽ More
Classical planning formulations like the Planning Domain Definition Language (PDDL) admit action sequences guaranteed to achieve a goal state given an initial state if any are possible. However, reasoning problems defined in PDDL do not capture temporal aspects of action taking, for example that two agents in the domain can execute an action simultaneously if postconditions of each do not interfere with preconditions of the other. A human expert can decompose a goal into largely independent constituent parts and assign each agent to one of these subgoals to take advantage of simultaneous actions for faster execution of plan steps, each using only single agent planning. By contrast, large language models (LLMs) used for directly inferring plan steps do not guarantee execution success, but do leverage commonsense reasoning to assemble action sequences. We combine the strengths of classical planning and LLMs by approximating human intuitions for two-agent planning goal decomposition. We demonstrate that LLM-based goal decomposition leads to faster planning times than solving multi-agent PDDL problems directly while simultaneously achieving fewer plan execution steps than a single agent plan alone and preserving execution success. Additionally, we find that LLM-based approximations of subgoals can achieve similar multi-agent execution steps than those specified by human experts. Website and resources at https://glamor-usc.github.io/twostep
△ Less
Submitted 25 March, 2024;
originally announced March 2024.
-
Navigating to Success in Multi-Modal Human-Robot Collaboration: Analysis and Corpus Release
Authors:
Stephanie M. Lukin,
Kimberly A. Pollard,
Claire Bonial,
Taylor Hudson,
Ron Arstein,
Clare Voss,
David Traum
Abstract:
Human-guided robotic exploration is a useful approach to gathering information at remote locations, especially those that might be too risky, inhospitable, or inaccessible for humans. Maintaining common ground between the remotely-located partners is a challenge, one that can be facilitated by multi-modal communication. In this paper, we explore how participants utilized multiple modalities to inv…
▽ More
Human-guided robotic exploration is a useful approach to gathering information at remote locations, especially those that might be too risky, inhospitable, or inaccessible for humans. Maintaining common ground between the remotely-located partners is a challenge, one that can be facilitated by multi-modal communication. In this paper, we explore how participants utilized multiple modalities to investigate a remote location with the help of a robotic partner. Participants issued spoken natural language instructions and received from the robot: text-based feedback, continuous 2D LIDAR mapping, and upon-request static photographs. We noticed that different strategies were adopted in terms of use of the modalities, and hypothesize that these differences may be correlated with success at several exploration sub-tasks. We found that requesting photos may have improved the identification and counting of some key entities (doorways in particular) and that this strategy did not hinder the amount of overall area exploration. Future work with larger samples may reveal the effects of more nuanced photo and dialogue strategies, which can inform the training of robotic agents. Additionally, we announce the release of our unique multi-modal corpus of human-robot communication in an exploration context: SCOUT, the Situated Corpus on Understanding Transactions.
△ Less
Submitted 26 October, 2023;
originally announced October 2023.
-
Interactive Evaluation of Dialog Track at DSTC9
Authors:
Shikib Mehri,
Yulan Feng,
Carla Gordon,
Seyed Hossein Alavi,
David Traum,
Maxine Eskenazi
Abstract:
The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to exte…
▽ More
The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategies that extend them to back-and-forth interactions with real users. The progression from static corpora to interactive evaluation introduces unique challenges and facilitates a more thorough assessment of open-domain dialog systems. This paper provides an overview of the track, including the methodology and results. Furthermore, it provides insights into how to best evaluate open-domain dialog models
△ Less
Submitted 28 July, 2022;
originally announced July 2022.
-
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges
Authors:
Shikib Mehri,
Jinho Choi,
Luis Fernando D'Haro,
Jan Deriu,
Maxine Eskenazi,
Milica Gasic,
Kallirroi Georgila,
Dilek Hakkani-Tur,
Zekang Li,
Verena Rieser,
Samira Shaikh,
David Traum,
Yi-Ting Yeh,
Zhou Yu,
Yizhe Zhang,
Chen Zhang
Abstract:
This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research.
This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research.
△ Less
Submitted 18 March, 2022;
originally announced March 2022.
-
Overview of the Ninth Dialog System Technology Challenge: DSTC9
Authors:
Chulaka Gunasekara,
Seokhwan Kim,
Luis Fernando D'Haro,
Abhinav Rastogi,
Yun-Nung Chen,
Mihail Eric,
Behnam Hedayatnia,
Karthik Gopalakrishnan,
Yang Liu,
Chao-Wei Huang,
Dilek Hakkani-Tür,
Jinchao Li,
Qi Zhu,
Lingxiao Luo,
Lars Liden,
Kaili Huang,
Shahin Shayandeh,
Runze Liang,
Baolin Peng,
Zheng Zhang,
Swadheen Shukla,
Minlie Huang,
Jianfeng Gao,
Shikib Mehri,
Yulan Feng
, et al. (14 additional authors not shown)
Abstract:
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems, namely, 1. Task-oriented dialog Modeling with unstructured knowledge access, 2. Multi-domain task-oriented dialog, 3. Interactive evaluation of dialog, and 4. Situated interactive multi-modal dialog. This…
▽ More
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems, namely, 1. Task-oriented dialog Modeling with unstructured knowledge access, 2. Multi-domain task-oriented dialog, 3. Interactive evaluation of dialog, and 4. Situated interactive multi-modal dialog. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.
△ Less
Submitted 12 November, 2020;
originally announced November 2020.
-
Balancing Efficiency and Coverage in Human-Robot Dialogue Collection
Authors:
Matthew Marge,
Claire Bonial,
Stephanie Lukin,
Cory Hayes,
Ashley Foots,
Ron Artstein,
Cassidy Henry,
Kimberly Pollard,
Carla Gordon,
Felix Gervits,
Anton Leuski,
Susan Hill,
Clare Voss,
David Traum
Abstract:
We describe a multi-phased Wizard-of-Oz approach to collecting human-robot dialogue in a collaborative search and navigation task. The data is being used to train an initial automated robot dialogue system to support collaborative exploration tasks. In the first phase, a wizard freely typed robot utterances to human participants. For the second phase, this data was used to design a GUI that includ…
▽ More
We describe a multi-phased Wizard-of-Oz approach to collecting human-robot dialogue in a collaborative search and navigation task. The data is being used to train an initial automated robot dialogue system to support collaborative exploration tasks. In the first phase, a wizard freely typed robot utterances to human participants. For the second phase, this data was used to design a GUI that includes buttons for the most common communications, and templates for communications with varying parameters. Comparison of the data gathered in these phases show that the GUI enabled a faster pace of dialogue while still maintaining high coverage of suitable responses, enabling more efficient targeted data collection, and improvements in natural language understanding using GUI-collected data. As a promising first step towards interactive learning, this work shows that our approach enables the collection of useful training data for navigation-based HRI tasks.
△ Less
Submitted 7 October, 2018; v1 submitted 3 October, 2018;
originally announced October 2018.
-
Multimodal Dialogue Management for Multiparty Interaction with Infants
Authors:
Setareh Nasihati Gilani,
David Traum,
Arcangelo Merla,
Eugenia Hee,
Zoey Walker,
Barbara Manini,
Grady Gallagher,
Laura-Ann Petitto
Abstract:
We present dialogue management routines for a system to engage in multiparty agent-infant interaction. The ultimate purpose of this research is to help infants learn a visual sign language by engaging them in naturalistic and socially contingent conversations during an early-life critical period for language development (ages 6 to 12 months) as initiated by an artificial agent. As a first step, we…
▽ More
We present dialogue management routines for a system to engage in multiparty agent-infant interaction. The ultimate purpose of this research is to help infants learn a visual sign language by engaging them in naturalistic and socially contingent conversations during an early-life critical period for language development (ages 6 to 12 months) as initiated by an artificial agent. As a first step, we focus on creating and maintaining agent-infant engagement that elicits appropriate and socially contingent responses from the baby. Our system includes two agents, a physical robot and an animated virtual human. The system's multimodal perception includes an eye-tracker (measures attention) and a thermal infrared imaging camera (measures patterns of emotional arousal). A dialogue policy is presented that selects individual actions and planned multiparty sequences based on perceptual inputs about the baby's internal changing states of emotional engagement. The present version of the system was evaluated in interaction with 8 babies. All babies demonstrated spontaneous and sustained engagement with the agents for several minutes, with patterns of conversationally relevant and socially contingent behaviors. We further performed a detailed case-study analysis with annotation of all agent and baby behaviors. Results show that the baby's behaviors were generally relevant to agent conversations and contained direct evidence for socially contingent responses by the baby to specific linguistic samples produced by the avatar. This work demonstrates the potential for language learning from agents in very young babies and has especially broad implications regarding the use of artificial agents with babies who have minimal language exposure in early life.
△ Less
Submitted 5 September, 2018;
originally announced September 2018.
-
Consequences and Factors of Stylistic Differences in Human-Robot Dialogue
Authors:
Stephanie M. Lukin,
Kimberly A. Pollard,
Claire Bonial,
Matthew Marge,
Cassidy Henry,
Ron Arstein,
David Traum,
Clare R. Voss
Abstract:
This paper identifies stylistic differences in instruction-giving observed in a corpus of human-robot dialogue. Differences in verbosity and structure (i.e., single-intent vs. multi-intent instructions) arose naturally without restrictions or prior guidance on how users should speak with the robot. Different styles were found to produce different rates of miscommunication, and correlations were fo…
▽ More
This paper identifies stylistic differences in instruction-giving observed in a corpus of human-robot dialogue. Differences in verbosity and structure (i.e., single-intent vs. multi-intent instructions) arose naturally without restrictions or prior guidance on how users should speak with the robot. Different styles were found to produce different rates of miscommunication, and correlations were found between style differences and individual user variation, trust, and interaction experience with the robot. Understanding potential consequences and factors that influence style can inform design of dialogue systems that are robust to natural variation from human users.
△ Less
Submitted 20 July, 2018;
originally announced July 2018.
-
ScoutBot: A Dialogue System for Collaborative Navigation
Authors:
Stephanie M. Lukin,
Felix Gervits,
Cory J. Hayes,
Anton Leuski,
Pooja Moolchandani,
John G. Rogers III,
Carlos Sanchez Amaro,
Matthew Marge,
Clare R. Voss,
David Traum
Abstract:
ScoutBot is a dialogue interface to physical and simulated robots that supports collaborative exploration of environments. The demonstration will allow users to issue unconstrained spoken language commands to ScoutBot. ScoutBot will prompt for clarification if the user's instruction needs additional input. It is trained on human-robot dialogue collected from Wizard-of-Oz experiments, where robot r…
▽ More
ScoutBot is a dialogue interface to physical and simulated robots that supports collaborative exploration of environments. The demonstration will allow users to issue unconstrained spoken language commands to ScoutBot. ScoutBot will prompt for clarification if the user's instruction needs additional input. It is trained on human-robot dialogue collected from Wizard-of-Oz experiments, where robot responses were initiated by a human wizard in previous interactions. The demonstration will show a simulated ground robot (Clearpath Jackal) in a simulated environment supported by ROS (Robot Operating System).
△ Less
Submitted 20 July, 2018;
originally announced July 2018.
-
Laying Down the Yellow Brick Road: Development of a Wizard-of-Oz Interface for Collecting Human-Robot Dialogue
Authors:
Claire Bonial,
Matthew Marge,
Ron artstein,
Ashley Foots,
Felix Gervits,
Cory J. Hayes,
Cassidy Henry,
Susan G. Hill,
Anton Leuski,
Stephanie M. Lukin,
Pooja Moolchandani,
Kimberly A. Pollard,
David Traum,
Clare R. Voss
Abstract:
We describe the adaptation and refinement of a graphical user interface designed to facilitate a Wizard-of-Oz (WoZ) approach to collecting human-robot dialogue data. The data collected will be used to develop a dialogue system for robot navigation. Building on an interface previously used in the development of dialogue systems for virtual agents and video playback, we add templates with open param…
▽ More
We describe the adaptation and refinement of a graphical user interface designed to facilitate a Wizard-of-Oz (WoZ) approach to collecting human-robot dialogue data. The data collected will be used to develop a dialogue system for robot navigation. Building on an interface previously used in the development of dialogue systems for virtual agents and video playback, we add templates with open parameters which allow the wizard to quickly produce a wide variety of utterances. Our research demonstrates that this approach to data collection is viable as an intermediate step in developing a dialogue system for physical robots in remote locations from their users - a domain in which the human and robot need to regularly verify and update a shared understanding of the physical environment. We show that our WoZ interface and the fixed set of utterances and templates therein provide for a natural pace of dialogue with good coverage of the navigation domain.
△ Less
Submitted 17 October, 2017;
originally announced October 2017.
-
Discourse Obligations in Dialogue Processing
Authors:
David R. Traum,
James F. Allen
Abstract:
We show that in modeling social interaction, particularly dialogue, the attitude of obligation can be a useful adjunct to the popularly considered attitudes of belief, goal, and intention and their mutual and shared counterparts. In particular, we show how discourse obligations can be used to account in a natural manner for the connection between a question and its answer in dialogue and how obl…
▽ More
We show that in modeling social interaction, particularly dialogue, the attitude of obligation can be a useful adjunct to the popularly considered attitudes of belief, goal, and intention and their mutual and shared counterparts. In particular, we show how discourse obligations can be used to account in a natural manner for the connection between a question and its answer in dialogue and how obligations can be used along with other parts of the discourse context to extend the coverage of a dialogue system.
△ Less
Submitted 14 July, 1994;
originally announced July 1994.
