-
Improving alignment of dialogue agents via targeted human judgements
Authors:
Amelia Glaese,
Nat McAleese,
Maja Trębacz,
John Aslanides,
Vlad Firoiu,
Timo Ewalds,
Maribeth Rauh,
Laura Weidinger,
Martin Chadwick,
Phoebe Thacker,
Lucy Campbell-Gillingham,
Jonathan Uesato,
Po-Sen Huang,
Ramona Comanescu,
Fan Yang,
Abigail See,
Sumanth Dathathri,
Rory Greig,
Charlie Chen,
Doug Fritz,
Jaume Sanchez Elias,
Richard Green,
Soňa Mokrá,
Nicholas Fernando,
Boxi Wu
, et al. (9 additional authors not shown)
Abstract:
We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into na…
▽ More
We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.
△ Less
Submitted 28 September, 2022;
originally announced September 2022.
-
NeRF-VAE: A Geometry Aware 3D Scene Generative Model
Authors:
Adam R. Kosiorek,
Heiko Strathmann,
Daniel Zoran,
Pol Moreno,
Rosalia Schneider,
Soňa Mokrá,
Danilo J. Rezende
Abstract:
We propose NeRF-VAE, a 3D scene generative model that incorporates geometric structure via NeRF and differentiable volume rendering. In contrast to NeRF, our model takes into account shared structure across scenes, and is able to infer the structure of a novel scene -- without the need to re-train -- using amortized inference. NeRF-VAE's explicit 3D rendering process further contrasts previous gen…
▽ More
We propose NeRF-VAE, a 3D scene generative model that incorporates geometric structure via NeRF and differentiable volume rendering. In contrast to NeRF, our model takes into account shared structure across scenes, and is able to infer the structure of a novel scene -- without the need to re-train -- using amortized inference. NeRF-VAE's explicit 3D rendering process further contrasts previous generative models with convolution-based rendering which lacks geometric structure. Our model is a VAE that learns a distribution over radiance fields by conditioning them on a latent scene representation. We show that, once trained, NeRF-VAE is able to infer and render geometrically-consistent scenes from previously unseen 3D environments using very few input images. We further demonstrate that NeRF-VAE generalizes well to out-of-distribution cameras, while convolutional models do not. Finally, we introduce and study an attention-based conditioning mechanism of NeRF-VAE's decoder, which improves model performance.
△ Less
Submitted 1 April, 2021;
originally announced April 2021.
-
Imitating Interactive Intelligence
Authors:
Josh Abramson,
Arun Ahuja,
Iain Barr,
Arthur Brussee,
Federico Carnevale,
Mary Cassin,
Rachita Chhaparia,
Stephen Clark,
Bogdan Damoc,
Andrew Dudzik,
Petko Georgiev,
Aurelia Guy,
Tim Harley,
Felix Hill,
Alden Hung,
Zachary Kenton,
Jessica Landon,
Timothy Lillicrap,
Kory Mathewson,
Soňa Mokrá,
Alistair Muldal,
Adam Santoro,
Nikolay Savinov,
Vikrant Varma,
Greg Wayne
, et al. (4 additional authors not shown)
Abstract:
A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central cha…
▽ More
A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount.
△ Less
Submitted 20 January, 2021; v1 submitted 10 December, 2020;
originally announced December 2020.
-
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
Authors:
Felix Hill,
Sona Mokra,
Nathaniel Wong,
Tim Harley
Abstract:
Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following…
▽ More
Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.
△ Less
Submitted 19 May, 2020;
originally announced May 2020.
