-
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination
Authors:
Jianing Yang,
Xuweiyi Chen,
Nikhil Madaan,
Madhavan Iyengar,
Shengyi Qian,
David F. Fouhey,
Joyce Chai
Abstract:
The integration of language and 3D perception is crucial for developing embodied agents and robots that comprehend and interact with the physical world. While large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, their adaptation to 3D environments (3D-LLMs) remains in its early stages. A primary challenge is the absence of large-scale datase…
▽ More
The integration of language and 3D perception is crucial for developing embodied agents and robots that comprehend and interact with the physical world. While large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, their adaptation to 3D environments (3D-LLMs) remains in its early stages. A primary challenge is the absence of large-scale datasets that provide dense grounding between language and 3D scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset comprising 40,087 household scenes paired with 6.2 million densely-grounded scene-language instructions. Our results show that instruction tuning with 3D-GRAND significantly enhances grounding capabilities and reduces hallucinations in 3D-LLMs. As part of our contributions, we propose a comprehensive benchmark 3D-POPE to systematically evaluate hallucination in 3D-LLMs, enabling fair comparisons among future models. Our experiments highlight a scaling effect between dataset size and 3D-LLM performance, emphasizing the critical role of large-scale 3D-text datasets in advancing embodied AI research. Notably, our results demonstrate early signals for effective sim-to-real transfer, indicating that models trained on large synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and 3D-POPE, we aim to equip the embodied AI community with essential resources and insights, setting the stage for more reliable and better-grounded 3D-LLMs. Project website: https://3d-grand.github.io
△ Less
Submitted 12 June, 2024; v1 submitted 7 June, 2024;
originally announced June 2024.
-
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
Authors:
Jianing Yang,
Xuweiyi Chen,
Shengyi Qian,
Nikhil Madaan,
Madhavan Iyengar,
David F. Fouhey,
Joyce Chai
Abstract:
3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipe…
▽ More
3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website https://chat-with-nerf.github.io/ .
△ Less
Submitted 21 September, 2023;
originally announced September 2023.
-
The Medical Algorithms Project
Authors:
M. Sriram Iyengar,
John R. Svirbely
Abstract:
The Medical Algorithms Project, a web-based resource located at www.medal.org, is the world's largest collection of medical-related spreadsheets, consisting of over 13,500 Excel spreadsheets each encoding a medical algorithm from 45 different areas of medical practice. This free resource is in use worldwide with over 106,000 registered users as of March 1, 2009.
The Medical Algorithms Project, a web-based resource located at www.medal.org, is the world's largest collection of medical-related spreadsheets, consisting of over 13,500 Excel spreadsheets each encoding a medical algorithm from 45 different areas of medical practice. This free resource is in use worldwide with over 106,000 registered users as of March 1, 2009.
△ Less
Submitted 6 August, 2009;
originally announced August 2009.
