Showing 1–50 of 101 results for author: Goldman, D

AquaMILR: Mechanical intelligence simplifies control of undulatory robots in cluttered fluid environments

Authors: Tianyu Wang, Nishanth Mankame, Matthew Fernandez, Velin Kojouharov, Daniel I. Goldman

Abstract: While undulatory swimming of elongate limbless robots has been extensively studied in open hydrodynamic environments, less research has been focused on limbless locomotion in complex, cluttered aquatic environments. Motivated by the concept of mechanical intelligence, where controls for obstacle navigation can be offloaded to passive body mechanics in terrestrial limbless locomotion, we hypothesiz… ▽ More While undulatory swimming of elongate limbless robots has been extensively studied in open hydrodynamic environments, less research has been focused on limbless locomotion in complex, cluttered aquatic environments. Motivated by the concept of mechanical intelligence, where controls for obstacle navigation can be offloaded to passive body mechanics in terrestrial limbless locomotion, we hypothesize that principles of mechanical intelligence can be extended to cluttered hydrodynamic regimes. To test this, we developed an untethered limbless robot capable of undulatory swimming on water surfaces, utilizing a bilateral cable-driven mechanism inspired by organismal muscle actuation morphology to achieve programmable anisotropic body compliance. We demonstrated through robophysical experiments that, similar to terrestrial locomotion, an appropriate level of body compliance can facilitate emergent swim through complex hydrodynamic environments under pure open-loop control. Moreover, we found that swimming performance depends on undulation frequency, with effective locomotion achieved only within a specific frequency range. This contrasts with highly damped terrestrial regimes, where inertial effects can often be neglected. Further, to enhance performance and address the challenges posed by nondeterministic obstacle distributions, we incorporated computational intelligence by developing a real-time body compliance tuning controller based on cable tension feedback. This controller improves the robot's robustness and overall speed in heterogeneous hydrodynamic environments. △ Less

Submitted 1 July, 2024; originally announced July 2024.

Generative AI for 2D Character Animation

Authors: Jaime Guajardo, Ozgun Bursalioglu, Dan B Goldman

Abstract: In this pilot project, we teamed up with artists to develop new workflows for 2D animation while producing a short educational cartoon. We identified several workflows to streamline the animation process, bringing the artists' vision to the screen more effectively. In this pilot project, we teamed up with artists to develop new workflows for 2D animation while producing a short educational cartoon. We identified several workflows to streamline the animation process, bringing the artists' vision to the screen more effectively. △ Less

Submitted 21 May, 2024; v1 submitted 17 May, 2024; originally announced May 2024.

Comments: Project page: https://genai-2d-character-animation.github.io/

BoLD: Fast and Cheap Dispute Resolution

Authors: Mario M. Alvarez, Henry Arneson, Ben Berger, Lee Bousfield, Chris Buckland, Yafah Edelman, Edward W. Felten, Daniel Goldman, Raul Jordan, Mahimna Kelkar, Akaki Mamageishvili, Harry Ng, Aman Sanghi, Victor Shoup, Terence Tsao

Abstract: BoLD is a new dispute resolution protocol that is designed to replace the originally deployed Arbitrum dispute resolution protocol. Unlike that protocol, BoLD is resistant to delay attacks. It achieves this resistance without a significant increase in onchain computation costs and with reduced staking costs. BoLD is a new dispute resolution protocol that is designed to replace the originally deployed Arbitrum dispute resolution protocol. Unlike that protocol, BoLD is resistant to delay attacks. It achieves this resistance without a significant increase in onchain computation costs and with reduced staking costs. △ Less

Submitted 16 April, 2024; originally announced April 2024.

State-dependent stiffness enhances wave propagation along elastic filaments

Authors: Clément Moreau, Benjamin J. Walker, Rebecca N. Poon, Daniel Soto, Daniel I. Goldman, Eamonn A. Gaffney, Kirsty Y. Wan

Abstract: We study an elastic filament beating in a viscous fluid with curvature-dependent bending stiffness. Our numerical and experimental investigations reveal that such differential stiffness can sustain planar bending waves far along flexible filaments. In particular, basal actuation is a viable, parsimonious mechanism for generating high-amplitude planar bending waves, in stark contrast to the uniform… ▽ More We study an elastic filament beating in a viscous fluid with curvature-dependent bending stiffness. Our numerical and experimental investigations reveal that such differential stiffness can sustain planar bending waves far along flexible filaments. In particular, basal actuation is a viable, parsimonious mechanism for generating high-amplitude planar bending waves, in stark contrast to the uniform-stiffness case. Further, the resulting beat patterns closely resemble the power-and-recovery strokes of propulsive biological filaments, suggesting applications in robotic and engineered systems. △ Less

Submitted 21 February, 2024; originally announced February 2024.

Comments: 6 pages, 4 figures

Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion

Authors: Kira Prabhu, Jane Wu, Lynn Tsai, Peter Hedman, Dan B Goldman, Ben Poole, Michael Broxton

Abstract: This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we sh… ▽ More This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Predicted depth is used as additional supervision to encourage accurate geometry. We compare our approach to 3D inpainting methods that focus on object removal. Because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion. △ Less

Submitted 6 December, 2023; originally announced December 2023.

Readout Guidance: Learning Control from Diffusion Features

Authors: Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, Aleksander Holynski

Abstract: We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple… ▽ More We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io. △ Less

Submitted 2 April, 2024; v1 submitted 4 December, 2023; originally announced December 2023.

Comments: CVPR 2024

Learning manipulation of steep granular slopes for fast Mini Rover turning

Authors: Deniz Kerimoglu, Daniel Soto, Malone Lincoln Hemsley, Joseph Brunner, Sehoon Ha, Tingnan Zhang, Daniel I. Goldman

Abstract: Future planetary exploration missions will require reaching challenging regions such as craters and steep slopes. Such regions are ubiquitous and present science-rich targets potentially containing information regarding the planet's internal structure. Steep slopes consisting of low-cohesion regolith are prone to flow downward under small disturbances, making it very challenging for autonomous rov… ▽ More Future planetary exploration missions will require reaching challenging regions such as craters and steep slopes. Such regions are ubiquitous and present science-rich targets potentially containing information regarding the planet's internal structure. Steep slopes consisting of low-cohesion regolith are prone to flow downward under small disturbances, making it very challenging for autonomous rovers to traverse. Moreover, the navigation trajectories of rovers are heavily limited by the terrain topology and future systems will need to maneuver on flowable surfaces without getting trapped, allowing them to further expand their reach and increase mission efficiency. In this work, we used a laboratory-scale rover robot and performed maneuvering experiments on a steep granular slope of poppy seeds to explore the rover's turning capabilities. The rover is capable of lifting, sweeping, and spinning its wheels, allowing it to execute leg-like gait patterns. The high-dimensional actuation capabilities of the rover facilitate effective manipulation of the underlying granular surface. We used Bayesian Optimization (BO) to gain insight into successful turning gaits in high dimensional search space and found strategies such as differential wheel spinning and pivoting around a single sweeping wheel. We then used these insights to further fine-tune the turning gait, enabling the rover to turn 90 degrees at just above 4 seconds with minimal slip. Combining gait optimization and human-tuning approaches, we found that fast turning is empowered by creating anisotropic torques with the sweeping wheel. △ Less

Submitted 2 October, 2023; originally announced October 2023.

Comments: 6 pages, 6 figures, conference paper submission for ICRA2024

Anisotropic body compliance facilitates robotic sidewinding in complex environments

Authors: Velin Kojouharov, Tianyu Wang, Matthew Fernandez, Jiyeon Maeng, Daniel I. Goldman

Abstract: Sidewinding, a locomotion strategy characterized by the coordination of lateral and vertical body undulations, is frequently observed in rattlesnakes and has been successfully reconstructed by limbless robotic systems for effective movement across diverse terrestrial terrains. However, the integration of compliant mechanisms into sidewinding limbless robots remains less explored, posing challenges… ▽ More Sidewinding, a locomotion strategy characterized by the coordination of lateral and vertical body undulations, is frequently observed in rattlesnakes and has been successfully reconstructed by limbless robotic systems for effective movement across diverse terrestrial terrains. However, the integration of compliant mechanisms into sidewinding limbless robots remains less explored, posing challenges for navigation in complex, rheologically diverse environments. Inspired by a notable control simplification via mechanical intelligence in lateral undulation, which offloads feedback control to passive body mechanics and interactions with the environment, we present an innovative design of a mechanically intelligent limbless robot for sidewinding. This robot features a decentralized bilateral cable actuation system that resembles organismal muscle actuation mechanisms. We develop a feedforward controller that incorporates programmable body compliance into the sidewinding gait template. Our experimental results highlight the emergence of mechanical intelligence when the robot is equipped with an appropriate level of body compliance. This allows the robot to 1) locomote more energetically efficiently, as evidenced by a reduced cost of transport, and 2) navigate through terrain heterogeneities, all achieved in an open-loop manner, without the need for environmental awareness. △ Less

Submitted 23 September, 2023; originally announced September 2023.

Robust self-propulsion in sand using simply controlled vibrating cubes

Authors: Bangyuan Liu, Tianyu Wang, Velin Kojouharov, Frank L. Hammond III, Daniel I. Goldman

Abstract: Much of the Earth and many surfaces of extraterrestrial bodies are composed of in-cohesive particle matter. Locomoting on granular terrain is challenging for common robotic devices, either wheeled or legged. In this work, we discover a robust alternative locomotion mechanism on granular media -- generating movement via self-vibration. To demonstrate the effectiveness of this locomotion mechanism,… ▽ More Much of the Earth and many surfaces of extraterrestrial bodies are composed of in-cohesive particle matter. Locomoting on granular terrain is challenging for common robotic devices, either wheeled or legged. In this work, we discover a robust alternative locomotion mechanism on granular media -- generating movement via self-vibration. To demonstrate the effectiveness of this locomotion mechanism, we develop a cube-shaped robot with an embedded vibratory motor and conduct systematic experiments on diverse granular terrains of various particle properties. We investigate how locomotion changes as a function of vibration frequency/intensity on granular terrains. Compared to hard surfaces, we find such a vibratory locomotion mechanism enables the robot to move faster, and more stable on granular surfaces, facilitated by the interaction between the body and surrounding granules. The simplicity in structural design and controls of this robotic system indicates that vibratory locomotion can be a valuable alternative way to produce robust locomotion on granular terrains. We further demonstrate that such cube-shape robots can be used as modular units for morphologically structured vibratory robots with capabilities of maneuverable forward and turning motions, showing potential practical scenarios for robotic systems. △ Less

Submitted 22 September, 2023; originally announced September 2023.

Multi-legged matter transport: a framework for locomotion on noisy landscapes

Authors: Baxi Chong, Juntao He, Daniel Soto, Tianyu Wang, Daniel Irvine, Grigoriy Blekherman, Daniel I. Goldman

Abstract: While the transport of matter by wheeled vehicles or legged robots can be guaranteed in engineered landscapes like roads or rails, locomotion prediction in complex environments like collapsed buildings or crop fields remains challenging. Inspired by principles of information transmission which allow signals to be reliably transmitted over noisy channels, we develop a ``matter transport" framework… ▽ More While the transport of matter by wheeled vehicles or legged robots can be guaranteed in engineered landscapes like roads or rails, locomotion prediction in complex environments like collapsed buildings or crop fields remains challenging. Inspired by principles of information transmission which allow signals to be reliably transmitted over noisy channels, we develop a ``matter transport" framework demonstrating that non-inertial locomotion can be provably generated over ``noisy" rugose landscapes (heterogeneities on the scale of locomotor dimensions). Experiments confirm that sufficient spatial redundancy in the form of serially-connected legged robots leads to reliable transport on such terrain without requiring sensing and control. Further analogies from communication theory coupled to advances in gaits (coding) and sensor-based feedback control (error detection/correction) can lead to agile locomotion in complex terradynamic regimes. △ Less

Submitted 8 May, 2023; originally announced May 2023.

Probing hydrodynamic fluctuation-induced forces with an oscillating robot

Authors: Steven W. Tarr, Joseph S. Brunner, Daniel Soto, Daniel I. Goldman

Abstract: We study the dynamics of an oscillating, free-floating robot that generates radially expanding gravity capillary waves at a fluid surface. In open water, the device does not self-propel; near a rigid boundary, it can be attracted or repelled. Visualization of the wave field dynamics reveals that when near a boundary, a complex interference of generated and reflected waves induces a wave amplitude… ▽ More We study the dynamics of an oscillating, free-floating robot that generates radially expanding gravity capillary waves at a fluid surface. In open water, the device does not self-propel; near a rigid boundary, it can be attracted or repelled. Visualization of the wave field dynamics reveals that when near a boundary, a complex interference of generated and reflected waves induces a wave amplitude fluctuation asymmetry. Attraction increases as wave frequency increases or robot-boundary separation decreases. Theory on confined gravity-capillary wave radiation dynamics developed by Hocking in the 1980s captures the observed parameter dependence due to these "Hocking fields." The flexibility of the robophysical system allows detailed characterization and analysis of locally generated nonequilibrium fluctuation-induced forces [M. Kardar and R. Golestanian, Rev. Mod. Phys. 71, 1233 (1999)]. △ Less

Submitted 20 February, 2024; v1 submitted 7 May, 2023; originally announced May 2023.

Comments: 11 pages, 5 figures, 8 supplementary figures

Journal ref: Phys. Rev. Lett. 132, 084001 (2024)

Mechanical Intelligence Simplifies Control in Terrestrial Limbless Locomotion

Authors: Tianyu Wang, Christopher Pierce, Velin Kojouharov, Baxi Chong, Kelimar Diaz, Hang Lu, Daniel I. Goldman

Abstract: Limbless locomotors, from microscopic worms to macroscopic snakes, traverse complex, heterogeneous natural environments typically using undulatory body wave propagation. Theoretical and robophysical models typically emphasize body kinematics and active neural/electronic control. However, we contend that because such approaches often neglect the role of passive, mechanically controlled processes (t… ▽ More Limbless locomotors, from microscopic worms to macroscopic snakes, traverse complex, heterogeneous natural environments typically using undulatory body wave propagation. Theoretical and robophysical models typically emphasize body kinematics and active neural/electronic control. However, we contend that because such approaches often neglect the role of passive, mechanically controlled processes (those involving "mechanical intelligence"), they fail to reproduce the performance of even the simplest organisms. To uncover principles of how mechanical intelligence aids limbless locomotion in heterogeneous terradynamic regimes, here we conduct a comparative study of locomotion in a model of heterogeneous terrain (lattices of rigid posts). We used a model biological system, the highly studied nematode worm Caenorhabditis elegans, and a robophysical device whose bilateral actuator morphology models that of limbless organisms across scales. The robot's kinematics quantitatively reproduced the performance of the nematodes with purely open-loop control; mechanical intelligence simplified control of obstacle navigation and exploitation by reducing the need for active sensing and feedback. An active behavior observed in C. elegans, undulatory wave reversal upon head collisions, robustified locomotion via exploitation of the systems' mechanical intelligence. Our study provides insights into how neurally simple limbless organisms like nematodes can leverage mechanical intelligence via appropriately tuned bilateral actuation to locomote in complex environments. These principles likely apply to neurally more sophisticated organisms and also provide a design and control paradigm for limbless robots for applications like search and rescue and planetary exploration. △ Less

Submitted 1 February, 2024; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: Published in Science Robotics

Journal ref: Sci. Robot. 8, eadi2243 (2023)

Gait design for limbless obstacle aided locomotion using geometric mechanics

Authors: Baxi Chong, Tianyu Wang, Daniel Irvine, Velin Kojouharov, Bo Lin, Howie Choset, Daniel I. Goldman, Grigoriy Blekherman

Abstract: Limbless robots have the potential to maneuver through cluttered environments that conventional robots cannot traverse. As illustrated in their biological counterparts such as snakes and nematodes, limbless locomotors can benefit from interactions with obstacles, yet such obstacle-aided locomotion (OAL) requires properly coordinated high-level self-deformation patterns (gait templates) as well as… ▽ More Limbless robots have the potential to maneuver through cluttered environments that conventional robots cannot traverse. As illustrated in their biological counterparts such as snakes and nematodes, limbless locomotors can benefit from interactions with obstacles, yet such obstacle-aided locomotion (OAL) requires properly coordinated high-level self-deformation patterns (gait templates) as well as low-level body adaptation to environments. Most prior work on OAL utilized stereotyped traveling-wave gait templates and relied on local body deformations (e.g., passive body mechanics or decentralized controller parameter adaptation based on force feedback) for obstacle navigation, while gait template design for OAL remains less studied. In this paper, we explore novel gait templates for OAL based on tools derived from geometric mechanics (GM), which thus far has been limited to homogeneous environments. Here, we expand the scope of GM to obstacle-rich environments. Specifically, we establish a model that maps the presence of an obstacle to directional constraints in optimization. In doing so, we identify novel gait templates suitable for sparsely and densely distributed obstacle-rich environments respectively. Open-loop robophysical experiments verify the effectiveness of our identified OAL gaits in obstacle-rich environments. We posit that when such OAL gait templates are augmented with appropriate sensing and feedback controls, limbless locomotors will gain robust function in obstacle rich environments. △ Less

Submitted 13 February, 2023; originally announced February 2023.

Geometry of contact: contact planning for multi-legged robots via spin models duality

Authors: Baxi Chong, Di Luo, Tianyu Wang, Gabriel Margolis, Juntao He, Pulkit Agrawal, Marin Soljačić, Daniel I. Goldman

Abstract: Contact planning is crucial in locomoting systems.Specifically, appropriate contact planning can enable versatile behaviors (e.g., sidewinding in limbless locomotors) and facilitate speed-dependent gait transitions (e.g., walk-trot-gallop in quadrupedal locomotors). The challenges of contact planning include determining not only the sequence by which contact is made and broken between the locomoto… ▽ More Contact planning is crucial in locomoting systems.Specifically, appropriate contact planning can enable versatile behaviors (e.g., sidewinding in limbless locomotors) and facilitate speed-dependent gait transitions (e.g., walk-trot-gallop in quadrupedal locomotors). The challenges of contact planning include determining not only the sequence by which contact is made and broken between the locomotor and the environments, but also the sequence of internal shape changes (e.g., body bending and limb shoulder joint oscillation). Most state-of-art contact planning algorithms focused on conventional robots (e.g.biped and quadruped) and conventional tasks (e.g. forward locomotion), and there is a lack of study on general contact planning in multi-legged robots. In this paper, we show that using geometric mechanics framework, we can obtain the global optimal contact sequence given the internal shape changes sequence. Therefore, we simplify the contact planning problem to a graph optimization problem to identify the internal shape changes. Taking advantages of the spatio-temporal symmetry in locomotion, we map the graph optimization problem to special cases of spin models, which allows us to obtain the global optima in polynomial time. We apply our approach to develop new forward and sidewinding behaviors in a hexapod and a 12-legged centipede. We verify our predictions using numerical and robophysical models, and obtain novel and effective locomotion behaviors. △ Less

Submitted 7 February, 2023; v1 submitted 6 February, 2023; originally announced February 2023.

Comments: SI video: https://doi.org/10.5281/zenodo.7608693

Report number: MIT-CTP/5526

Physics of Smart Matter: integrating active matter and control to gain insights into living systems

Authors: Herbert Levine, Daniel I. Goldman

Abstract: We offer our opinion on the benefits of integration of insights from active matter physics with principles of regulatory interactions and control to develop a field we term ``smart matter". This field can provide insight into important principles in living systems as well as aid engineering of responsive, robust and functional collectives. We offer our opinion on the benefits of integration of insights from active matter physics with principles of regulatory interactions and control to develop a field we term ``smart matter". This field can provide insight into important principles in living systems as well as aid engineering of responsive, robust and functional collectives. △ Less

Submitted 31 January, 2023; originally announced February 2023.

Comments: opinion piece

Water surface swimming dynamics in lightweight centipedes

Authors: Kelimar Diaz, Baxi Chong, Steven Tarr, Eva Erickson, Daniel I. Goldman

Abstract: Study of the locomotion of a centipede (L. forficatus) at the air-water interface reveals that it does not predominantly use its 14 leg pairs to locomote; unlike most swimmers which propagate head-to-tail body bending waves, this species propels via tail-to-head waves. Its low mass and body-fluid contact yield locomotion dynamics in which fluid wave drag forces dominate inertia. Microorganism-insp… ▽ More Study of the locomotion of a centipede (L. forficatus) at the air-water interface reveals that it does not predominantly use its 14 leg pairs to locomote; unlike most swimmers which propagate head-to-tail body bending waves, this species propels via tail-to-head waves. Its low mass and body-fluid contact yield locomotion dynamics in which fluid wave drag forces dominate inertia. Microorganism-inspired wave drag resistive force theory captures swimming performance of the macroscale animal, motivating a control hypothesis for the animals self-propulsion body and limb kinematics. △ Less

Submitted 18 October, 2022; originally announced October 2022.

Self-propulsion via slipping: frictional swimming in multi-legged locomotors

Authors: Baxi Chong, Juntao He, Shengkai Li, Eva Erickson, Kelimar Diaz, Tianyu Wang, Daniel Soto, Daniel I. Goldman

Abstract: Locomotion is typically studied either in continuous media where bodies and legs experience forces generated by the flowing medium, or on solid substrates dominated by friction. In the former, centralized coordination is believed to facilitate appropriate slipping through the medium for propulsion. In the latter, slip is often assumed minimal and thus avoided via decentralized controls. We discove… ▽ More Locomotion is typically studied either in continuous media where bodies and legs experience forces generated by the flowing medium, or on solid substrates dominated by friction. In the former, centralized coordination is believed to facilitate appropriate slipping through the medium for propulsion. In the latter, slip is often assumed minimal and thus avoided via decentralized controls. We discover in laboratory experiments that terrestrial locomotion of a meter scale multi-segmented/legged robophysical model resembles undulatory fluid swimming. Experiments varying waves of limb stepping and body bending reveal how these parameters result in effective terrestrial locomotion despite seemingly ineffective isotropic frictional contacts. Dissipation dominates over inertial effects in this macroscopic-scaled regime, resulting in essentially geometric locomotion akin to microscopic-scale swimming. Theoretical analysis demonstrates that the high-dimensional multi-segmented/legged dynamics can be simplified to a centralized low-dimensional model, which reveals an effective Resistive Force Theory with an acquired viscous drag anisotropy. We extend our low-dimensional, geometric analysis to illustrate how body undulation can aid performance in non-flat obstacle-rich terrains and also use the scheme to quantitatively model how body undulation affects performance of biological centipede locomotion (the desert centipede S. polymorpha) moving at relatively high speeds (~0.5 body lengths/sec). Our results could facilitate control of multilegged robots in complex terradynamic scenarios. △ Less

Submitted 9 August, 2022; v1 submitted 21 July, 2022; originally announced July 2022.

Amorphous Entangled Active Matter

Authors: William Savoie, Harry Tuazon, M. Saad Bhamla, Daniel I. Goldman

Abstract: The design of amorphous entangled systems, specifically from soft and active materials, has the potential to open exciting new classes of active, shape-shifting, and task-capable 'smart' materials. However, the global emergent mechanics that arises from the local interactions of individual particles are not well understood. In this study, we examine the emergent properties of amorphous entangled s… ▽ More The design of amorphous entangled systems, specifically from soft and active materials, has the potential to open exciting new classes of active, shape-shifting, and task-capable 'smart' materials. However, the global emergent mechanics that arises from the local interactions of individual particles are not well understood. In this study, we examine the emergent properties of amorphous entangled systems in three different examples: an in-silico "smarticle" collection, its robophysical chain, and living entangled aggregate of worm blobs (L. variegatus). In simulations, we examine how material properties change for a collective composed of dynamic three-link robots. We compare three methods of controlling entanglement in a collective: externally oscillations, shape-changes, and internal oscillations. We find that large-amplitude changes of the particle's shape using the shape-change procedure produced the highest average number of entanglements, with respect to the aspect ratio (l/w), improving the tensile strength of the collective. We demonstrate application of these simulations in two experimental systems: robotic chains and entangled worm blobs. In the robophysical models, we find emergent auxeticity behavior upon straining the confined collective. And finally, we show how the individual worm activity in a blob can be controlled through the ambient dissolved oxygen in water, leading to complex emergent properties of the living entangled collective, such as solid-like entanglement and tumbling. Taken together, our work reveals principles by which future shape-modulating, potentially soft robotic systems may dynamically alter their material properties, advancing our understanding of living entangled materials, while inspiring new classes of synthetic emergent super-materials. △ Less

Submitted 7 July, 2022; originally announced July 2022.

Comments: 15 pages, 19 figures, W.S. and H.T. contributed equally to this work

Gliders in shape-changing active matter

Authors: Akash Vardhan, Ram Avinery, Hridesh Kedia, Shengkai Li, Kurt Wiesenfeld, Daniel I. Goldman

Abstract: We report in experiment and simulation the spontaneous formation of dynamically bound pairs of shape changing smarticle robots undergoing locally repulsive collisions. Borrowing terminology from Conway's simulated Game of Life, these physical `gliders' robustly emerge from an ensemble of individually undulating three-link two-motor smarticles and can remain bound for hundreds of undulations and tr… ▽ More We report in experiment and simulation the spontaneous formation of dynamically bound pairs of shape changing smarticle robots undergoing locally repulsive collisions. Borrowing terminology from Conway's simulated Game of Life, these physical `gliders' robustly emerge from an ensemble of individually undulating three-link two-motor smarticles and can remain bound for hundreds of undulations and travel for multiple robot dimensions. Gliders occur in two distinct binding symmetries and form over a wide range of angular flapping extent. This parameter sets the maximal concavity which influences formation probability and translation characteristics. Analysis of dynamics in simulation reveals the mechanism of effective dynamical attraction -- a result of the emergent interplay of appropriately oriented and timed repulsive interactions. △ Less

Submitted 9 June, 2022; originally announced June 2022.

Oxygenation-Controlled Collective Dynamics in Aquatic Worm Blobs

Authors: Harry Tuazon, Emily Kaufman, Daniel I. Goldman, M. Saad Bhamla

Abstract: Many types of organisms utilize group aggregation as a method for survival. The freshwater oligochaete, California blackworms Lumbriculus variegatus form tightly entangled structures, or worm "blobs", that have adapted to survive in extremely low levels of dissolved oxygen (DO). Individual blackworms adapt to hypoxic environments through respiration via their mucous body wall and posterior ciliate… ▽ More Many types of organisms utilize group aggregation as a method for survival. The freshwater oligochaete, California blackworms Lumbriculus variegatus form tightly entangled structures, or worm "blobs", that have adapted to survive in extremely low levels of dissolved oxygen (DO). Individual blackworms adapt to hypoxic environments through respiration via their mucous body wall and posterior ciliated hindgut, which they wave above them. However, the change in collective behavior at different levels of DO is not known. Using a closed-loop respirometer with flow, we discover that the relative tail reaching activity flux in low DO is $\sim$75x higher than in the high DO condition. Additionally, when flow rate is increased to suspend the worm blobs upward, we find that the average exposed surface area of a blob in low DO is $\sim$1.4x higher than in high DO. Furthermore, we observe emergent properties that arise when a worm blob is exposed to extreme DO levels. Here we show that internal stress is generated when worm blobs are exposed to high DO levels, allowing them to be physically lifted off from the bottom of a conical container using a serrated endpiece. Our results demonstrate how both collective behavior and the emergent generation of internal stress in worm blobs change to accommodate differing levels of oxygen, which could be used to model and simulate swarm robots, self-assembly structures, or soft material entanglements. △ Less

Submitted 8 June, 2022; originally announced June 2022.

Mechanistic framework for reduced-order models in soft materials: Application to three-dimensional granular intrusion

Authors: Shashank Agarwal, Daniel I Goldman, Ken Kamrin

Abstract: Soft materials often display complex behaviors that transition through apparent solid- and fluid-like regimes. While a growing number of microscale simulation methods exist for these materials, reduced-order models that encapsulate the global-scale physics are often desired to predict how external bodies interact with soft media, as occurs in diverse situations from impact and penetration problems… ▽ More Soft materials often display complex behaviors that transition through apparent solid- and fluid-like regimes. While a growing number of microscale simulation methods exist for these materials, reduced-order models that encapsulate the global-scale physics are often desired to predict how external bodies interact with soft media, as occurs in diverse situations from impact and penetration problems to locomotion over natural terrains. This work proposes a systematic program to develop three-dimensional reduced-order models for soft materials from a fundamental basis using continuum symmetries and rheological principles. In particular, we derive a reduced-order technique for modeling intrusion in granular media which we term three-dimensional Resistive Force Theory (3D-RFT), which is capable of accurately and quickly predicting the resistive stress distribution on arbitrary-shaped intruding bodies. Aided by a continuum description of the granular medium, a comprehensive set of spatial symmetry constraints, and a limited amount of reference data, we develop a self-consistent and accurate 3D-RFT. We verify the model capabilities in a wide range of cases and show it can be quickly recalibrated to different media and intruder surface types. The premises leading to 3D-RFT anticipate application to other soft materials with strongly hyperlocalized intrusion behavior. △ Less

Submitted 10 December, 2022; v1 submitted 30 May, 2022; originally announced May 2022.

Comments: 12 pages, 7 figures, 1 SI document (12 pages, 8 figures, and 4 tables)

Asymptotic Inference for Infinitely Imbalanced Logistic Regression

Authors: Dorian Goldman, Bo Zhang

Abstract: In this paper we extend the work of Owen (2007) by deriving a second order expansion for the slope parameter in logistic regression, when the size of the majority class is unbounded and the minority class is finite. More precisely, we demonstrate that the second order term converges to a normal distribution and explicitly compute its variance, which surprisingly once again depends only on the mean… ▽ More In this paper we extend the work of Owen (2007) by deriving a second order expansion for the slope parameter in logistic regression, when the size of the majority class is unbounded and the minority class is finite. More precisely, we demonstrate that the second order term converges to a normal distribution and explicitly compute its variance, which surprisingly once again depends only on the mean of the minority class points and not their arrangement under mild regularity assumptions. In the case that the majority class is normally distributed, we illustrate that the variance of the the limiting slope depends exponentially on the z-score of the average of the minority class's points with respect to the majority class's distribution. We confirm our results by Monte Carlo simulations. △ Less

Submitted 27 April, 2022; originally announced April 2022.

A robophysical model of spacetime dynamics

Authors: Shengkai Li, Hussain N. Gynai, Steven Tarr, Emily Alicea-Muñoz, Pablo Laguna, Gongjie Li, Daniel I. Goldman

Abstract: Systems consisting of spheres rolling on elastic membranes have been used to introduce a core conceptual idea of General Relativity (GR): how curvature guides the movement of matter. However, such schemes cannot accurately represent relativistic dynamics in the laboratory because of the dominance of dissipation and external gravitational fields. Here we demonstrate that an ``active" object (a whee… ▽ More Systems consisting of spheres rolling on elastic membranes have been used to introduce a core conceptual idea of General Relativity (GR): how curvature guides the movement of matter. However, such schemes cannot accurately represent relativistic dynamics in the laboratory because of the dominance of dissipation and external gravitational fields. Here we demonstrate that an ``active" object (a wheeled robot), which moves in a straight line on level ground and can alter its speed depending on the curvature of the deformable terrain it moves on, can exactly capture dynamics in curved relativistic spacetimes. Via the systematic study of the robot's dynamics in the radial and orbital directions, we develop a mapping of the emergent trajectories of a wheeled vehicle on a spandex membrane to the motion in a curved spacetime. Our mapping demonstrates how the driven robot's dynamics mix space and time in a metric, and shows how active particles do not necessarily follow geodesics in the real space but instead follow geodesics in a fiducial spacetime. The mapping further reveals how parameters such as the membrane elasticity and instantaneous speed allow the programming of a desired spacetime, such as the Schwarzschild metric near a non-rotating blackhole. Our mapping and framework facilitate creation of a robophysical analog to a general relativistic system in the laboratory at low cost that can provide insights into active matter in deformable environments and robot exploration in complex landscapes. △ Less

Submitted 6 October, 2023; v1 submitted 10 February, 2022; originally announced February 2022.

Journal ref: Scientific Reports 13, 21589 (2023)

Generalized Omega Turn Gait Enables Agile Limbless Robot Turning in Complex Environments

Authors: Tianyu Wang, Baxi Chong, Yuelin Deng, Ruijie Fu, Howie Choset, Daniel I. Goldman

Abstract: Reorientation (turning in plane) plays a critical role for all robots in any field application, especially those that in confined spaces. While important, reorientation remains a relatively unstudied problem for robots, including limbless mechanisms, often called snake robots. Instead of looking at snakes, we take inspiration from observations of the turning behavior of tiny nematode worms C. eleg… ▽ More Reorientation (turning in plane) plays a critical role for all robots in any field application, especially those that in confined spaces. While important, reorientation remains a relatively unstudied problem for robots, including limbless mechanisms, often called snake robots. Instead of looking at snakes, we take inspiration from observations of the turning behavior of tiny nematode worms C. elegans. Our previous work presented an in-place and in-plane turning gait for limbless robots, called an omega turn, and prescribed it using a novel two-wave template. In this work, we advance omega turn-inspired controllers in three aspects: 1) we use geometric methods to vary joint angle amplitudes and forward wave spatial frequency in our turning equation to establish a wide and precise amplitude modulation and frequency modulation on omega turn; 2) we use this new relationship to enable robots with fewer internal degrees of freedom (i.e., fewer joints in the body) to achieve desirable performance, and 3) we apply compliant control methods to this relationship to handle unmodelled effects in the environment. We experimentally validate our approach on a limbless robot that the omega turn can produce effective and robust turning motion in various types of environments, such as granular media and rock pile. △ Less

Submitted 3 March, 2022; v1 submitted 3 February, 2022; originally announced February 2022.

Comments: Accepted to ICRA 2022

Coordinating tiny limbs and long bodies: geometric mechanics of diverse undulatory lizard locomotion

Authors: Baxi Chong, Tianyu Wang, Eva Erickson, Philip J Bergmann, Daniel I. Goldman

Abstract: Although typically possessing four limbs and short bodies, lizards have evolved a diversity of body plans, from short-bodied and fully-limbed to elongate and nearly limbless. Such diversity in body morphology is hypothesized as adaptations to locomotion cluttered terrestrial environments, but the mode of propulsion -- e.g., the use of body and/or limbs to interact with the substrate -- and potenti… ▽ More Although typically possessing four limbs and short bodies, lizards have evolved a diversity of body plans, from short-bodied and fully-limbed to elongate and nearly limbless. Such diversity in body morphology is hypothesized as adaptations to locomotion cluttered terrestrial environments, but the mode of propulsion -- e.g., the use of body and/or limbs to interact with the substrate -- and potential body/limb coordination remain unstudied. Here, we use biological experiments, a geometric theory of locomotion, and robophysical experiments to comparatively and systematically investigate such dynamics in a diverse sample of lizard morphologies. Locomotor field studies in short-limb, elongated lizards (Brachymeles) and laboratory studies of full-limbed lizards (Uma scoparia and Sceloporus olivaceus) and a limbless laterally undulating organism (Chionactis occipitalis) reveal that the body wave dynamics can be described by a combination of traveling and standing waves; the ratio of the amplitudes of these components is inversely related to limb length. We use geometric theory to analyze and explain the wave dynamics and body-leg coordination observations; the theory predicts that leg thrust modulates the body weight distribution and self-propulsion generation mechanism, which in turn facilitates the choice of body waves. We test our hypothesis in biological experiments by inducing the use of traveling wave in stereotyped lizards by modulating the ground penetration resistance, as well as in controlled non-biological experiments involving an undulating limbed robophysical model. Our models could be valuable in understanding functional constraints on the evolutionary process of elongation and limb reduction in lizards, as well as advancing robot designs. △ Less

Submitted 23 January, 2022; originally announced January 2022.

Locomotion without force, and impulse via dissipation: Robotic swimming in curved space via geometric phase

Authors: Shengkai Li, Tianyu Wang, Velin H. Kojouharov, James McInerney, Yasemin O. Aydin, Enes Aydin, Daniel I. Goldman, D. Zeb Rocklin

Abstract: Locomotion by shape changes (spermatozoon swimming, snake slithering, bird flapping) or gas expulsion (rocket firing) is assumed to require environmental interaction, due to conservation of momentum. As first noted in (Wisdom, 2003) and later in (Guéron, 2009) and (Avron et al, 2006), in curved space or spacetime the non-commutativity of translations permits translation without momentum exchange,… ▽ More Locomotion by shape changes (spermatozoon swimming, snake slithering, bird flapping) or gas expulsion (rocket firing) is assumed to require environmental interaction, due to conservation of momentum. As first noted in (Wisdom, 2003) and later in (Guéron, 2009) and (Avron et al, 2006), in curved space or spacetime the non-commutativity of translations permits translation without momentum exchange, just as falling cats and lizards can self-deform to reorient in flat space without environmental interaction. Translation in curved space can occur not only in gravitationally induced curved spacetime (where translation is predicted to be on the order of $10^{-23}$ m per gait cycle) but also in the curved surfaces encountered by locomotors in real-world environments. Here we show that a precision robophysical apparatus consisting of motors driven on curved tracks (and thereby confined to a spherical surface without a solid substrate) can self-propel without environmental momentum exchange (impulse) via shape changes that can generate gauge potentials that manifest as translations. Our system produces shape changes comparable to the environment's inverse curvatures and generates from zero momentum forward movement of $10^{-1}$ cm per gait cycle even while resisted by weak gravitational and frictional forces. Dissipation via friction eventually arrests the robot but also imbues it with momentum which can be released upon a cessation of shape changes. This work demonstrates how the interaction between environmental curvature, active driving and geometric phases yields rich, exotic phenomena. △ Less

Submitted 19 January, 2022; v1 submitted 17 December, 2021; originally announced December 2021.

Journal ref: Proceedings of the National Academy of Sciences 119 (31), e2200924119, 2022

A general locomotion control framework for multi-legged locomotors

Authors: Baxi Chong, Yasemin O. Aydin, Jennifer M. Rieser, Guillaume Sartoretti, Tianyu Wang, Julian Whitman, Abdul Kaba, Enes Aydin, Ciera McFarland, Kelimar Diaz Cruz, Jeffery W. Rankin, Krijn B Michel, Alfredo Nicieza, John R Hutchinson, Howie Choset, Daniel I. Goldman

Abstract: Serially connected robots are promising candidates for performing tasks in confined spaces such as search-and-rescue in large-scale disasters. Such robots are typically limbless, and we hypothesize that the addition of limbs could improve mobility. However, a challenge in designing and controlling such devices lies in the coordination of high-dimensional redundant modules in a way that improves mo… ▽ More Serially connected robots are promising candidates for performing tasks in confined spaces such as search-and-rescue in large-scale disasters. Such robots are typically limbless, and we hypothesize that the addition of limbs could improve mobility. However, a challenge in designing and controlling such devices lies in the coordination of high-dimensional redundant modules in a way that improves mobility. Here we develop a general framework to control serially connected multi-legged robots. Specifically, we combine two approaches to build a general shape control scheme which can provide baseline patterns of self-deformation ("gaits") for effective locomotion in diverse robot morphologies. First, we take inspiration from a dimensionality reduction and a biological gait classification scheme to generate cyclic patterns of body deformation and foot lifting/lowering, which facilitate generation of arbitrary substrate contact patterns. Second, we use geometric mechanics methods to facilitates identification of optimal phasing of these undulations to maximize speed and/or stability. Our scheme allows the development of effective gaits in multi-legged robots locomoting on flat frictional terrain with diverse number of limbs (4, 6, 16, and even 0 limbs) and body actuation capabilities (including sidewinding gaits on limbless devices). By properly coordinating the body undulation and the leg placement, our framework combines the advantages of both limbless robots (modularity) and legged robots (mobility). We expect that our framework can provide general control schemes for the rapid deployment of general multi-legged robots, paving the ways toward machines that can traverse complex environments under real-life conditions. △ Less

Submitted 3 February, 2022; v1 submitted 1 December, 2021; originally announced December 2021.

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Authors: Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, Steven M. Seitz

Abstract: Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches st… ▽ More Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at https://hypernerf.github.io. △ Less

Submitted 10 September, 2021; v1 submitted 24 June, 2021; originally announced June 2021.

Comments: SIGGRAPH Asia 2021, Project page: https://hypernerf.github.io/

Neural RGB-D Surface Reconstruction

Authors: Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies

Abstract: Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appeara… ▽ More Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera refinement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions. △ Less

Submitted 14 March, 2022; v1 submitted 9 April, 2021; originally announced April 2021.

Comments: CVPR'22; Project page: https://dazinovic.github.io/neural-rgbd-surface-reconstruction/ Video: https://youtu.be/iWuSowPsC3g

Efficacy of simple continuum models for diverse granular intrusions

Authors: Shashank Agarwal, Andras Karsai, Daniel I Goldman, Ken Kamrin

Abstract: Granular intrusion is commonly observed in natural and human-made settings. Unlike typical solids and fluids, granular media can simultaneously display fluid-like and solid-like characteristics in a variety of intrusion scenarios. This multi-phase behavior increases the difficulty of accurately modeling these and other yielding (or flowable) materials. Micro-scale modeling methods, such as DEM (Di… ▽ More Granular intrusion is commonly observed in natural and human-made settings. Unlike typical solids and fluids, granular media can simultaneously display fluid-like and solid-like characteristics in a variety of intrusion scenarios. This multi-phase behavior increases the difficulty of accurately modeling these and other yielding (or flowable) materials. Micro-scale modeling methods, such as DEM (Discrete Element Method), capture this behavior by modeling the media at the grain scale, but there is often interest in the macro-scale characterizations of such systems. We examine the efficacy of a macro-scale continuum approach in modeling and understanding the physics of various macroscopic phenomena in a variety of granular intrusion cases using two basic frictional yielding constitutive models. We compare predicted granular force response and material flow to experimental data in four quasi-2D intrusion cases: (1) depth-dependent force response in horizontal submerged-intruder motion; (2) separation dependent drag variation in parallel-plate vertical-intrusion; (3) initial-density-dependent drag fluctuations in free surface plowing, and (4) flow zone development during vertical plate intrusions in under-compacted granular media. Our continuum modeling approach captures the flow process and drag forces while providing key meso- and macro-scopic insights. The modeling results are then compared to experimental data. Our study highlights how continuum modeling approaches provide an alternative for efficient modeling as well as a conceptual understanding of various granular intrusion phenomena. △ Less

Submitted 15 July, 2021; v1 submitted 25 January, 2021; originally announced January 2021.

Comments: 14 pages, 8 figures, and 4 movies

Low rattling: A predictive principle for self-organization in active collectives

Authors: Pavel Chvykov, Thomas A. Berrueta, Akash Vardhan, William Savoie, Alexander Samland, Todd D. Murphey, Kurt Wiesenfeld, Daniel I. Goldman, Jeremy L. England

Abstract: Self-organization is frequently observed in active collectives, from ant rafts to molecular motor assemblies. General principles describing self-organization away from equilibrium have been challenging to identify. We offer a unifying framework that models the behavior of complex systems as largely random, while capturing their configuration-dependent response to external forcing. This allows deri… ▽ More Self-organization is frequently observed in active collectives, from ant rafts to molecular motor assemblies. General principles describing self-organization away from equilibrium have been challenging to identify. We offer a unifying framework that models the behavior of complex systems as largely random, while capturing their configuration-dependent response to external forcing. This allows derivation of a Boltzmann-like principle for understanding and manipulating driven self-organization. We validate our predictions experimentally in shape-changing robotic active matter, and outline a methodology for controlling collective behavior. Our findings highlight how emergent order depends sensitively on the matching between external patterns of forcing and internal dynamical response properties, pointing towards future approaches for design and control of active particle mixtures and metamaterials. △ Less

Submitted 3 January, 2021; originally announced January 2021.

Journal ref: Science, Vol. 371, Issue 6524, pp. 90-95 (2021)

Reconstruction of Backbone Curves for Snake Robots

Authors: Tianyu Wang, Bo Lin, Baxi Chong, Julian Whitman, Matthew Travers, Daniel I. Goldman, Greg Blekherman, Howie Choset

Abstract: Snake robots composed of alternating single-axis pitch and yaw joints have many internal degrees of freedom, which make them capable of versatile three-dimensional locomotion. In motion planning process, snake robot motions are often designed kinematically by a chronological sequence of continuous backbone curves that capture desired macroscopic shapes of the robot. However, as the geometric arran… ▽ More Snake robots composed of alternating single-axis pitch and yaw joints have many internal degrees of freedom, which make them capable of versatile three-dimensional locomotion. In motion planning process, snake robot motions are often designed kinematically by a chronological sequence of continuous backbone curves that capture desired macroscopic shapes of the robot. However, as the geometric arrangement of single-axis rotary joints creates constraints on the rotations in the robot, it is challenging for the robot to reconstruct an arbitrary 3D curve. When the robot configuration does not accurately achieve the desired shapes defined by these backbone curves, the robot can have unexpected contacts with the environment, such that the robot does not achieve the desired motion. In this work, we propose a method for snake robots to reconstruct desired backbone curves by posing an optimization problem that exploits the robot's geometric structure. We verified that our method enables fast and accurate curve-configuration conversions through its applications to commonly used 3D gaits. We also demonstrated via robot experiments that 1) our method results in smooth locomotion on the robot; 2) our method allows the robot to approach the numerically predicted locomotive performance of a sequence of continuous backbone curve. △ Less

Submitted 17 February, 2021; v1 submitted 8 December, 2020; originally announced December 2020.

Comments: Accepted for publication in IEEE RA-L

Nerfies: Deformable Neural Radiance Fields

Authors: Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla

Abstract: We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local mini… ▽ More We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity. △ Less

Submitted 9 September, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

Comments: ICCV 2021, Project page with videos: https://nerfies.github.io/

Effect of two parallel intruders on net work during granular penetrations

Authors: Swapnil Pravin, Brian Chang, Endao Han, Lionel London, Daniel I. Goldman, Heinrich M. Jaeger, S. Tonia Hsieh

Abstract: The impact of single passive intruders into granular particles has been studied in detail. However, the impact force produced by multiple intruders separated at a distance from one another, and hence the effect of their presence in close proximity to one another, is largely unexplored. Here, we use numerical simulations and laboratory experiments to study the force response of two parallel rods in… ▽ More The impact of single passive intruders into granular particles has been studied in detail. However, the impact force produced by multiple intruders separated at a distance from one another, and hence the effect of their presence in close proximity to one another, is largely unexplored. Here, we use numerical simulations and laboratory experiments to study the force response of two parallel rods intruding vertically into granular media while varying the gap spacing between them. We also explored the effect of variations in friction, intruder size, and particle size on the force response. The net work ($W$) of the two rods over the depth of intrusion was measured, and the instantaneous velocities of particles over the duration of intrusion were calculated by simulations. We found that the work done by the intruders changes with distance between them. We observed a peak in $W$ at a gap spacing of $\sim$3 particle diameters, which was up to 25% greater than $W$ at large separation (\textgreater 11 particle diameters), beyond which the net work plateaued. This peak was likely due to less particle flow between intruders as we found a larger number of strong forces---identified as force chains---in the particle domain at gaps surrounding the peak force. Although higher friction caused greater force generation during intrusion, the gap spacing between the intruders at which the peak work was generated remained unchanged. Larger intruder sizes resulted in greater net work with the peak in $W$ occurring at slightly larger intruder separations. Taken together, our results show that peak work done by two parallel intruders remained within a narrow range, remaining robust to most other tested parameters. △ Less

Submitted 28 October, 2020; originally announced October 2020.

Comments: 12 pages, 8 figures, 6 supplemental figures, and appendix

Journal ref: Phys. Rev. E 104, 024902 (2021)

Programming Active Cohesive Granular Matter with Mechanically Induced Phase Changes

Authors: Shengkai Li, Bahnisikha Dutta, Sarah Cannon, Joshua J. Daymude, Ram Avinery, Enes Aydin, Andréa W. Richa, Daniel I. Goldman, Dana Randall

Abstract: Active matter physics and swarm robotics have provided powerful tools for the study and control of ensembles driven by internal sources. At the macroscale, controlling swarms typically utilizes significant memory, processing power, and coordination unavailable at the microscale, e.g., for colloidal robots, which could be useful for fighting disease, fabricating intelligent textiles, and designing… ▽ More Active matter physics and swarm robotics have provided powerful tools for the study and control of ensembles driven by internal sources. At the macroscale, controlling swarms typically utilizes significant memory, processing power, and coordination unavailable at the microscale, e.g., for colloidal robots, which could be useful for fighting disease, fabricating intelligent textiles, and designing nanocomputers. To develop principles that that can leverage physics of interactions and thus can be utilized across scales, we take a two-pronged approach: a theoretical abstraction of self-organizing particle systems and an experimental robot system of active cohesive granular matter that intentionally lacks digital electronic computation and communication, using minimal (or no) sensing and control, to test theoretical predictions. We consider the problems of aggregation, dispersion, and collective transport. As predicted by the theory, as a parameter representing interparticle attraction increases, the robots transition from a dispersed phase to an aggregated one, forming a dense, compact collective. When aggregated, the collective can transport non-robot "impurities" in their environment, thus performing an emergent task driven by the physics underlying the transition. These results point to a fruitful interplay between algorithm design and active matter robophysics that can result in new nonequilibrium physics and principles for programming collectives without the need for complex algorithms or capabilities. △ Less

Submitted 2 February, 2021; v1 submitted 11 September, 2020; originally announced September 2020.

Journal ref: Science Advances 7(17), eabe8494, 2021

Measurements of ${}^{nat}Pb(p,xn){}^{201-207}Bi$, ${}^{204}Pb(p,1-4n){}^{201-204}Bi$ and ${}^{206}Pb(p,3n){}^{204}Bi$ cross-sections at astrophysical energies ($E_p \leq 30 \, MeV$)

Authors: P V Guillaumon, I D Goldman, V R Vanin, H Barcellos de Oliveira

Abstract: Cross-sections for ${}^{nat}Pb(p,xn){}^{201-207}Bi$, ${}^{204}Pb(p,1-4n){}^{201-204}Bi$ and ${}^{206}Pb(p,3n){}^{204}Bi$ reactions have been determined in the astrophysical energy range $20-30 \, MeV$. The analysis were performed by $γ$-spectroscopy associated with half-lives measurements. The results were compared with previously experimental data, when available, and with theoretical calculation… ▽ More Cross-sections for ${}^{nat}Pb(p,xn){}^{201-207}Bi$, ${}^{204}Pb(p,1-4n){}^{201-204}Bi$ and ${}^{206}Pb(p,3n){}^{204}Bi$ reactions have been determined in the astrophysical energy range $20-30 \, MeV$. The analysis were performed by $γ$-spectroscopy associated with half-lives measurements. The results were compared with previously experimental data, when available, and with theoretical calculations performed using TALYS code. We report a possible new $γ$ transition from ${}^{204m}{Pb}$ and other theoretical discrepancies, probably due overestimation of the Coulomb barrier and neutron binding energy. Also reported are new determinations of the energies and intensities of the $γ$'s emitted in the decays of ${}^{205,206}Bi$. We discuss possible implications for the r/rp-processes in neutrino-driven wind in supernovae and neutron stars mergers. △ Less

Submitted 6 September, 2020; originally announced September 2020.

The importance of charged particle reactions in the r-process on supernovae and neutron stars

Authors: Pedro V. Guillaumon, Iuda D. Goldman

Abstract: We propose a $(p,xn)$ mechanism with dynamic production as a new set of nuclear reactions that could produce high density neutrons and explain the r- and rp-elements. We calculate the rate of thorium and uranium produced by our proposed mechanism and show that it is compatible with different stellar conditions found in explosive events at an initial temperature of $T \geq 3\times 10^{9} K$ with a… ▽ More We propose a $(p,xn)$ mechanism with dynamic production as a new set of nuclear reactions that could produce high density neutrons and explain the r- and rp-elements. We calculate the rate of thorium and uranium produced by our proposed mechanism and show that it is compatible with different stellar conditions found in explosive events at an initial temperature of $T \geq 3\times 10^{9} K$ with a "freeze-out" by a neutrino-driven wind. We show that charged particle reactions could explain the discrepancies in the abundances of ${}^{232}Th$ and ${}^{235,238}U$ nucleochronometers. We extend the endpoint of the rapid proton (rp) process far beyond the previous work by showing that $(p,xn)$ reactions could contribute to the nucleosynthesis of heavy stable neutron deficient nuclides, like ${}^{190}Pt$, ${}^{184}Os$, ${}^{180}W$ and ${}^{174}$Hf. This implies in a broader definition of the rp-process and has important consequences for the nucleosynthesis of heavy elements. We show that we did not need to assume an extreme condition for the drip line of super neutron-rich nuclei. △ Less

Submitted 3 September, 2020; originally announced September 2020.

GeLaTO: Generative Latent Textured Objects

Authors: Ricardo Martin-Brualla, Rohit Pandey, Sofien Bouaziz, Matthew Brown, Dan B Goldman

Abstract: Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to enc… ▽ More Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to encode both medium and fine scale geometry as well as view-dependent appearance. To generate the proxies' textures, we learn a joint latent space allowing category-level appearance and geometry interpolation. The proxies are independently rasterized with their corresponding neural texture and composited using a U-Net, which generates an output photorealistic image including an alpha map. We demonstrate the effectiveness of our approach by reconstructing complex objects from a sparse set of views. We show results on a dataset of real images of eyeglasses frames, which are particularly challenging to reconstruct using classical methods. We also demonstrate that these coarse proxies can be handcrafted when the underlying object geometry is easy to model, like eyeglasses, or generated using a neural network for more complex categories, such as cars. △ Less

Submitted 11 August, 2020; originally announced August 2020.

Comments: ECCV 2020 Spotlight. Project website: https://gelato-paper.github.io

Journal ref: European Conference on Computer Vision 2020

Surprising simplicity in the modeling of dynamic granular intrusion

Authors: Shashank Agarwal, Andras Karsai, Daniel I Goldman, Ken Kamrin

Abstract: Granular intrusions, such as dynamic impact or wheel locomotion, are complex multiphase phenomena where the grains exhibit solid-like and fluid-like characteristics together with an ejected gas-like phase. Despite decades of modeling efforts, a unified description of the physics in such intrusions is as yet unknown. Here we show that a continuum model based on the simple notions of frictional flow… ▽ More Granular intrusions, such as dynamic impact or wheel locomotion, are complex multiphase phenomena where the grains exhibit solid-like and fluid-like characteristics together with an ejected gas-like phase. Despite decades of modeling efforts, a unified description of the physics in such intrusions is as yet unknown. Here we show that a continuum model based on the simple notions of frictional flow and tension-free separation describes complex granular intrusions near free surfaces. This model captures dynamics in a variety of experiments including wheel locomotion, plate intrusions, and running legged robots. The model reveals that three effects (a static contribution and two dynamic ones) primarily give rise to intrusion forces in such scenarios. Identification of these effects enables the development of a further reduced-order technique (Dynamic Resistive Force Theory) for rapid modeling of granular locomotion of arbitrarily shaped intruders. The continuum-motivated strategy we propose for identifying physical mechanisms and corresponding reduced-order relations has potential use for a variety of other materials. △ Less

Submitted 21 January, 2021; v1 submitted 21 May, 2020; originally announced May 2020.

Comments: 41 pages including supplementary document, 10 figures, and 8 video

State of the Art on Neural Rendering

Authors: Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer

Abstract: Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer… ▽ More Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state-of-the-art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. This state-of-the-art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems. △ Less

Submitted 8 April, 2020; originally announced April 2020.

Comments: Eurographics 2020 survey paper

Field-mediated locomotor dynamics on highly deformable surfaces

Authors: Shengkai Li, Yasemin Ozkan Aydin, Charles Xiao, Gabriella Small, Hussain N. Gynai, Gongjie Li, Jennifer M. Rieser, Pablo Laguna, Daniel I. Goldman

Abstract: In many systems motion occurs on deformed and deformable surfaces, setting up the possibility for dynamical interactions solely mediated by the coupling of the entities with their environment. Here we study the "two-body" dynamics of robot locomotion on a highly deformable spandex membrane in two scenarios: one in which a robot orbits a large central depression and the other where the two robots a… ▽ More In many systems motion occurs on deformed and deformable surfaces, setting up the possibility for dynamical interactions solely mediated by the coupling of the entities with their environment. Here we study the "two-body" dynamics of robot locomotion on a highly deformable spandex membrane in two scenarios: one in which a robot orbits a large central depression and the other where the two robots affect each other's motion solely through mutual environmental deformations. Inspired by the resemblance of the orbits of the single robot with those of general relativistic orbits around black holes, we recast the vehicle plus membrane dynamics in physical space into the geodesic motion of a "test particle" in a fiducial curved space-time and demonstrate how this framework facilitates understanding the observed dynamics. The two-robot problem also exhibits a resemblance with Einstein's general relativistic view of gravity, which in the words of Wheeler: "spacetime tells matter how to move; matter tells spacetime how to curve." We generalize this case the mapping to include a reciprocal coupling that translates into robotic curvature-based control schemes which modify interaction (promoting avoidance or aggregation) without long-range sensing. Our work provides a starting point for developing a mechanical analog gravity system as well as develops a framework that can provide insights into active matter in deformable environments and robot exploration in complex landscapes. △ Less

Submitted 3 August, 2021; v1 submitted 7 April, 2020; originally announced April 2020.

Journal ref: Proceedings of the National Academy of Sciences 119 (30), e2113912119, 2022

TRAPPIST-1 Habitable Atmosphere Intercomparison (THAI). Motivations and protocol version 1.0

Authors: Thomas J. Fauchez, Martin Turbet, Eric T. Wolf, Ian Boutle, Michael J. Way, Anthony D. Del Genio, Nathan J. Mayne, Konstantinos Tsigaridis, Ravi K. Kopparapu, Jun Yang, Francois Forget, Avi Mandell, Shawn D. Domagal Goldman

Abstract: Upcoming telescopes such as the James Webb Space Telescope (JWST), or the Extremely Large Telescope (ELTs), may soon be able to characterize, through transmission, emission or reflection spectroscopy, the atmospheres of rocky exoplanets orbiting nearby M dwarfs. One of the most promising candidates is the late M dwarf system TRAPPIST-1 which has seven known transiting planets for which Transit Tim… ▽ More Upcoming telescopes such as the James Webb Space Telescope (JWST), or the Extremely Large Telescope (ELTs), may soon be able to characterize, through transmission, emission or reflection spectroscopy, the atmospheres of rocky exoplanets orbiting nearby M dwarfs. One of the most promising candidates is the late M dwarf system TRAPPIST-1 which has seven known transiting planets for which Transit Timing Variation (TTV) measurements suggest that they are terrestrial in nature, with a possible enrichment in volatiles. Among these seven planets, TRAPPIST-1e seems to be the most promising candidate to have habitable surface conditions, receiving ~66 % of the Earth's incident radiation, and thus needing only modest greenhouse gas inventories to raise surface temperatures to allow surface liquid water to exist. TRAPPIST-1e is therefore one of the prime targets for JWST atmospheric characterization. In this context, the modeling of its potential atmosphere is an essential step prior to observation. Global Climate Models (GCMs) offer the most detailed way to simulate planetary atmospheres. However, intrinsic differences exist between GCMs which can lead to different climate prediction and thus observability of gas and/or cloud features in transmission and thermal emission spectra. Such differences should preferably be known prior to observations. In this paper we present a protocol to inter-compare planetary GCMs. Four testing cases are considered for TRAPPIST-1e but the methodology is applicable to other rocky exoplanets in the Habitable Zone. The four test cases included two land planets composed with a modern Earth and pure CO2 atmospheres, respectively, and two aqua planets with the same atmospheric compositions. Currently, there are four participating models (LMDG, ROCKE-3D, ExoCAM, UM), however this protocol is intended to let other teams participate as well. △ Less

Submitted 25 February, 2020; originally announced February 2020.

Comments: Accepted and published in Geoscientific Model Development (GMD)

Journal ref: Geosci. Model Dev., 13, 707-716, 2020

A resistive force model of legged locomotion on granular media

Authors: Chen Li, Tingnan Zhang, Daniel I. Goldman

Abstract: Compared to agile legged animals, wheeled and tracked vehicles often suffer large performance loss on granular surfaces like sand and gravel. Understanding the mechanics of legged locomotion on granular media can aid the development of legged robots with improved mobility on granular surfaces; however, no general force model yet exists for granular media to predict ground reaction forces during co… ▽ More Compared to agile legged animals, wheeled and tracked vehicles often suffer large performance loss on granular surfaces like sand and gravel. Understanding the mechanics of legged locomotion on granular media can aid the development of legged robots with improved mobility on granular surfaces; however, no general force model yet exists for granular media to predict ground reaction forces during complex limb intrusions. Inspired by a recent study of sand-swimming, we develop a resistive force model in the vertical plane for legged locomotion on granular media. We divide an intruder of complex morphology and kinematics, e.g., a bio-inspired robot L-leg rotated through uniform granular media (loosely packed ~ 1 mm diameter poppy seeds), into small segments, and measure stresses as a function of depth, orientation, and direction of motion using a model leg segment. Summation of segmental forces over the intruder predicts the net forces on both an L-leg and a reversed L-leg rotated through granular media with better accuracy than using simple one-dimensional penetration and drag force models. A multi-body dynamic simulation using the resistive force model predicts the speeds of a small legged robot (15 cm, 150 g) moving on granular media using both L-legs and reversed L-legs. △ Less

Submitted 31 October, 2019; originally announced November 2019.

Journal ref: Adaptive Mobile Robotics, pp. 433-440 (2012)

Towards a terramechanics for bio-inspired locomotion in granular environments

Authors: Chen Li, Yang Ding, Nick Gravish, Ryan D. Maladen, Andrew Masse, Paul B. Umbanhowar, Haldun Komsuoglu, Daniel E. Koditschek, Daniel I. Goldman

Abstract: Granular media (GM) present locomotor challenges for terrestrial and extraterrestrial devices because they can flow and solidify in response to localized intrusion of wheels, limbs, and bodies. While the development of airplanes and submarines is aided by understanding of hydrodynamics, fundamental theory does not yet exist to describe the complex interactions of locomotors with GM. In this paper,… ▽ More Granular media (GM) present locomotor challenges for terrestrial and extraterrestrial devices because they can flow and solidify in response to localized intrusion of wheels, limbs, and bodies. While the development of airplanes and submarines is aided by understanding of hydrodynamics, fundamental theory does not yet exist to describe the complex interactions of locomotors with GM. In this paper, we use experimental, computational, and theoretical approaches to develop a terramechanics for bio-inspired locomotion in granular environments. We use a fluidized bed to prepare GM with a desired global packing fraction, and use empirical force measurements and the Discrete Element Method (DEM) to elucidate interaction mechanics during locomotion-relevant intrusions in GM such as vertical penetration and horizontal drag. We develop a resistive force theory (RFT) to account for more complex intrusions. We use these force models to understand the locomotor performance of two bio-inspired robots moving on and within GM. △ Less

Submitted 31 October, 2019; originally announced November 2019.

Journal ref: ASCE Earth & Space Conference (2012), 264-273

Geometric phase predicts locomotion performance in undulating living systems across scales

Authors: Jennifer M Rieser, Baxi Chong, Chaohui Gong, Henry C Astley, Perrin E Schiebel, Kelimar Diaz, Christopher Pierce, Hang Lu, Ross L Hatton, Howie Choset, Daniel I Goldman

Abstract: Self-propelling organisms locomote via generation of patterns of self-deformation. Despite the diversity of body plans, internal actuation schemes and environments in limbless vertebrates and invertebrates, such organisms often use similar travelling waves of axial body bending for movement. Delineating how parameters (wave amplitudes, frequencies) lead to locomotor performance (e.g. speed, energy… ▽ More Self-propelling organisms locomote via generation of patterns of self-deformation. Despite the diversity of body plans, internal actuation schemes and environments in limbless vertebrates and invertebrates, such organisms often use similar travelling waves of axial body bending for movement. Delineating how parameters (wave amplitudes, frequencies) lead to locomotor performance (e.g. speed, energy, turning capabilities) remains challenging. Here we show that a geometric framework that replaces laborious calculation with a diagrammatic scheme is well suited to discovery and comparison of optimal patterns of wave dynamics in diverse living systems. We focus on a regime of undulatory locomotion, that of highly damped environments, which is applicable not only to small organism movement in viscous fluids, but also larger animals moving in frictional fluids (sand) and on frictional ground. We find that the travelling wave dynamics used by mm-scale nematode worms and cm-scale desert dwelling snakes and lizards can be described by time series of the weights associated with two principal modes. The approximately circular closed path trajectories of mode weights in a self-deformation space enclose near-maximal surface integral (geometric phase) for organisms spanning two decades in body length. We hypothesize that such trajectories are targets of control (which we refer to as ``serpenoid templates"). Further, the geometric approach reveals how seemingly complex behaviors such as turning in worms and sidewinding snakes can be described as modulations of templates. Thus, the use of differential geometry in living systems can assist in the growth of comparative neuromechanics, allowing a common description of locomotion across taxa and providing hypotheses for function at lower levels of organization. △ Less

Submitted 2 January, 2024; v1 submitted 26 June, 2019; originally announced June 2019.

Text-based Editing of Talking-head Video

Authors: Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala

Abstract: Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video wi… ▽ More Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis. △ Less

Submitted 4 June, 2019; originally announced June 2019.

Comments: A version with higher resolution images can be downloaded from the authors' website

Neural Rerendering in the Wild

Authors: Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-Brualla

Abstract: We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn t… ▽ More We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos. △ Less

Submitted 8 April, 2019; originally announced April 2019.

Comments: To be presented at CVPR 2019 (oral). Supplementary video available at http://youtu.be/E1crWQn_kmY

Optical trapping below the diffraction limit with a tunable beam waist using super-oscillating beams

Authors: Harel Nagar, Tamir Admon, Doron Goldman, Amir Eyal, Yael Roichman

Abstract: Super-oscillating beams can be used to create light spots whose size is below the diffraction limit with a side ring of high intensity adjacent to them. Optical traps made of the super-oscillating part of such beams exhibit superior localization of submicron beads compared to regular optical traps. Here we focus on the effect of the ratio of particle size to trap size on the localization and stiff… ▽ More Super-oscillating beams can be used to create light spots whose size is below the diffraction limit with a side ring of high intensity adjacent to them. Optical traps made of the super-oscillating part of such beams exhibit superior localization of submicron beads compared to regular optical traps. Here we focus on the effect of the ratio of particle size to trap size on the localization and stiffness of optical traps made of super-oscillating beams. We find a non-monotonic dependence of trapping stiffness on the ratio of particle size to beam size. Optimal trapping is achieved when the particle is larger than the beam waist of the superoscillating feature but small enough not to overlap with the side ring. △ Less

Submitted 13 March, 2019; originally announced March 2019.

Comments: 5 pages, 5 Figs

Modeling of the interaction of rigid wheels with dry granular media

Authors: Shashank Agarwal, Carmine Senatore, Tingnan Zhang, Mark Kingsbury, Karl Iagnemma, Daniel I. Goldman, Ken Kamrin

Abstract: We analyze the capabilities of various recently developed techniques, namely Resistive Force Theory (RFT) and continuum plasticity implemented with the Material Point Method (MPM), in capturing dynamics of wheel--dry granular media interactions. We compare results to more conventionally accepted methods of modeling wheel locomotion. While RFT is an empirical force model for arbitrarily-shaped bodi… ▽ More We analyze the capabilities of various recently developed techniques, namely Resistive Force Theory (RFT) and continuum plasticity implemented with the Material Point Method (MPM), in capturing dynamics of wheel--dry granular media interactions. We compare results to more conventionally accepted methods of modeling wheel locomotion. While RFT is an empirical force model for arbitrarily-shaped bodies moving through granular media, MPM-based continuum modeling allows the simulation of full granular flow and stress fields. RFT allows for rapid evaluation of interaction forces on arbitrary shaped intruders based on a local surface stress formulation depending on depth, orientation, and movement of surface elements. We perform forced-slip experiments for three different wheel types and three different granular materials, and results are compared with RFT, continuum modeling, and a traditional terramechanics semi-empirical method. Results show that for the range of inputs considered, RFT can be reliably used to predict rigid wheel granular media interactions with accuracy exceeding that of traditional terramechanics methodology in several circumstances. Results also indicate that plasticity-based continuum modeling provides an accurate tool for wheel-soil interaction while providing more information to study the physical processes giving rise to resistive stresses in granular media. △ Less

Submitted 29 January, 2019; originally announced January 2019.

Explicit cloud representation in the Atmos 1D climate model for Earth and rocky planet applications

Authors: Thomas Fauchez, Giada Arney, Ravi Kumar Kopparapu, Shawn Domagal Goldman

Abstract: 1D climate models are less sophisticated than 3D global circulation models (GCMs), however their computational time is much less expensive, allowing a large number of runs in a short period of time to explore a wide parameter space. Exploring parameter space is particularly important for predicting the observable properties of exoplanets, for which few parameters are known with certainty. Therefor… ▽ More 1D climate models are less sophisticated than 3D global circulation models (GCMs), however their computational time is much less expensive, allowing a large number of runs in a short period of time to explore a wide parameter space. Exploring parameter space is particularly important for predicting the observable properties of exoplanets, for which few parameters are known with certainty. Therefore, 1D climate models are still very useful tools for planetary studies. In most of these 1D models, clouds are not physically represented in the atmosphere, despite having a well-known, significant impact on a planetary radiative budget. This impact is simulated by artificially raising surface albedo, in order to reproduce the observed-averaged surface temperature (i.e. 288 K for modern Earth) and a radiative balance at the top of the atmosphere. This non-physical representation of clouds, causes atmospheric longwave and shortwaves fluxes to not match observational data. Additionally, this technique represents a parameter that is highly-tuned to modern Earth's climate, and may not be appropriate for planets that deviate from modern Earth's climate conditions. In this paper, we present an update to the climate model within the Atmos 1D atmospheric modeling package with a physical representation of clouds. We show that this physical representation of clouds in the atmosphere allows both longwave and shortwave fluxes to match observational data. This improvement will allow us to study the energy fluxes for a variety of cloudy rocky planets, and increase our confidence in future simulations of temperature profile and net energy balance. △ Less

Submitted 26 November, 2018; originally announced November 2018.

Comments: Accepted in AIMS Geosciences