skip to main content
research-article

Towards an Understanding of Augmented Reality Extensions for Existing 3D Data Analysis Tools

Published: 23 April 2020 Publication History
  • Get Citation Alerts
  • Abstract

    We present an observational study with domain experts to understand how augmented reality (AR) extensions to traditional PC-based data analysis tools can help particle physicists to explore and understand 3D data. Our goal is to allow researchers to integrate stereoscopic AR-based visual representations and interaction techniques into their tools, and thus ultimately to increase the adoption of modern immersive analytics techniques in existing data analysis workflows. We use Microsoft's HoloLens as a lightweight and easily maintainable AR headset and replicate existing visualization and interaction capabilities on both the PC and the AR view. We treat the AR headset as a second yet stereoscopic screen, allowing researchers to study their data in a connected multi-view manner. Our results indicate that our collaborating physicists appreciate a hybrid data exploration setup with an interactive AR extension to improve their understanding of particle collision events.

    Supplementary Material

    MP4 File (paper528vf.mp4)
    Supplemental video
    MP4 File (paper528pv.mp4)
    Preview video
    MP4 File (a528-wang-presentation.mp4)

    References

    [1]
    George Alverson, Giulio Eulisse, Thomas McCauley, and Lucas Taylor. 2012. iSpy: A powerful and lightweight event display. Journal of Physics: Conference Series 396, 2, Article 022002 (Dec. 2012), 6 pages.
    [2]
    Sabrina Amrouche, Laurent Basara, Paolo Calafiura, Victor Estrade, Steven Farrell, Diogo R Ferreira, Liam Finnie, Nicole Finnie, Cécile Germain, Vladimir Vava Gligorov, and others. 2019. The Tracking Machine Learning challenge: Accuracy phase. arXiv preprint 1904.06778. https://arxiv.org/abs/1904.06778
    [3]
    Benjamin Bach, Ronell Sicat, Johanna Beyer, Maxime Cordeil, and Hanspeter Pfister. 2018. The hologram in my hand: How effective is interactive exploration of 3D visualizations in immersive tangible augmented reality? IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 457--467.
    [4]
    Matthew Bellis, Riccardo Maria Bianchi, Sebastien Binet, Ciril Bohak, Benjamin Couturier, Hadrien Grasland, Oliver Gutsche, Sergey Linev, Alex Martyniuk, Thomas McCauley, Edward Moyse, Alja Mrak Tadel, Mark Neubauer, Jeremi Niedziela, Leo Piilonen, Jim Pivarski, Martin Ritter, Tai Sakuma, Matevz Tadel, and Ben Waugh. 2018. HEP Software Foundation Community White Paper Working Group - Visualization. arXiv preprint 1811.10309. https://arxiv.org/abs/1811.10309
    [5]
    Hrvoje Benko, Edward W. Ishak, and Steven Feiner. 2004. Collaborative mixed reality visualization of an archaeological excavation. In Proc. ISMAR. IEEE Computer Society, Los Alamitos, 132--140.
    [6]
    Lonni Besançon, Paul Issartel, Mehdi Ammi, and Tobias Isenberg. 2017. Hybrid tactile/tangible interaction for 3D data exploration. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 881--890.
    [7]
    Lonni Besançon, Paul Issartel, Mehdi Ammi, and Tobias Isenberg. 2017. Mouse, tactile, and tangible input for 3D manipulation. In Proc. CHI. ACM, New York, 4727--4740.
    [8]
    Lonni Besançon, Amir Semmo, David J. Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic. 2018. Reducing affective responses to surgical images through color manipulation and stylization. In Proc. Expressive. ACM, New York, Article 4, 13 pages.
    [9]
    Lonni Besançon, Mickael Sereno, Mehdi Ammi, Lingyun Yu, and Tobias Isenberg. 2019. Hybrid touch/tangible spatial 3D data selection. Computer Graphics Forum 38, 3 (June 2019), 553--567.
    [10]
    Richard Brath. 2014. 3D InfoVis is here to stay: Deal with it. In Proc. 3DVis. IEEE Computer Society, Los Alamitos, 25--31.
    [11]
    Frederik Brudy, Christian Holz, Roman Rädle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices. In Proc. CHI. ACM, New York, Article 562, 28 pages.
    [12]
    Sheelagh Carpendale. 2008. Evaluating information visualizations. In Information Visualization: Human-Centered Issues and Perspectives. LNCS, Vol. 4950. Springer, Berlin/Heidelberg, 19--45.
    [13]
    Citi. 2016. Citi HoloLens holographic workstation. Online video: https://www.youtube.com/watch?v=0NogltmewmQ. (2016). Accessed: August 10, 2019.
    [14]
    Carolina Cruz-Neira, Daniel J. Sandin, Thomas A. DeFanti, Robert V. Kenyon, and John C. Hart. 1992. The CAVE: Audio visual experience automatic virtual environment. Communications of the ACM 35, 6 (June 1992), 64--72.
    [15]
    Gerwin de Haan, Michal Koutek, and Frits H. Post. 2002. Towards intuitive exploration tools for data visualization in VR. In Proc. VRST. ACM, New York, 105--112.
    [16]
    Zach Duer, Leo Piilonen, and George Glasson. 2018. Belle2VR: A virtual-reality visualization of subatomic particle physics in the Belle II experiment. IEEE Computer Graphics and Applications 38 (May/June 2018), 33--43.
    [17]
    Alessandro Febretti, Arthur Nishimoto, Terrance Thigpen, Jonas Talandis, Lance Long, JD Pirtle, Tom Peterka, Alan Verlo, Maxine Brown, Dana Plepys, and others. 2013. CAVE2: A hybrid reality environment for immersive simulation and information analysis. In The Engineering Reality of Virtual Reality (SPIE Proceedings Series), Vol. 8649. SPIE, Bellingham, Washington, Article 864903, 12 pages.
    [18]
    Jorge A. Wagner Filho, Carla M. D. S. Freitas, and Luciana Nedel. 2019. Comfortable immersive analytics with the virtualdesk metaphor. IEEE Computer Graphics and Applications 39, 3 (May 2019), 41--53.
    [19]
    Andrew Forsberg, Jian Chen, and David Laidlaw. 2009. Comparing 3D vector field visualization methods: A user study. IEEE Transactions on Visualization and Computer Graphics 15, 6 (Nov. 2009), 1219--1226.
    [20]
    Bernd Fröhlich, Stephen Barrass, Björn Zehner, John Plate, and Martin Göbel. 1999. Exploring geo-scientific data in virtual environments. In Proc. Visualization. IEEE Computer Society, Los Alamitos, 169--173.
    [21]
    Chi-Wing Fu, Wooi-Boon Goh, and Junxiang Allen Ng. 2010. Multi-touch techniques for exploring large-scale 3D astrophysical simulations. In Proc. CHI. ACM, New York, 2213--2222.
    [22]
    Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Text Entry in Immersive Head-Mounted Display-based Virtual Reality using Standard Keyboards. arXiv preprint 1802.00626. http://arxiv.org/abs/1802.00626
    [23]
    Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed endurance: A metric to quantify arm fatigue of mid-air interactions. In Proc. CHI. ACM, New York, 1063--1072.
    [24]
    Microsoft HoloLens. 2015. Microsoft hololens: Partner demo with Maya by Autodesk. Online video: https://www.youtube.com/watch?v=q0K3n0Gf8mA. (2015). Accessed: August 10, 2019.
    [25]
    Christophe Hurter, Nathalie Henry Riche, Steven M. Drucker, Maxime Cordeil, Richard Alligier, and Romain Vuillemot. 2019. FiberClay: Sculpting three dimensional trajectories to reveal structural insights. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan. 2019), 704--714.
    [26]
    Daniel F. Keefe. 2008. Free-form VR interactions in scientific visualization. In Proc. IEEE Visualization Workshop on Scientific Workflow with Immersive Interfaces for Visualization. http://ivlab.cs.umn.edu/ generated/pub-Keefe-2008-FreeFormSciVis.php
    [27]
    Thomas Kittelmann, Vakhtang Tsulaia, Joseph Boudreau, and Edward Moyse. 2010. The Virtual Point 1 event display for the ATLAS experiment. Journal of Physics: Conference Series 219, part 3, Article 032012 (May 2010), 10 pages.
    [28]
    Tijmen Klein, Florimond Guéniat, Luc Pastur, Frédéric Vernier, and Tobias Isenberg. 2012. A design study of direct-touch interaction for exploratory 3D scientific visualization. Computer Graphics Forum 31, 3 (June 2012), 1225--1234.
    [29]
    Wolfgang Krueger and Bernd Froehlich. 1994. The responsive workbench. IEEE Computer Graphics and Applications 14, 3 (May 1994), 12--15.
    [30]
    David López, Lora Oehlberg, Candemir Doger, and Tobias Isenberg. 2016. Towards an understanding of mobile touch navigation in a stereoscopic viewing environment for 3D data exploration. IEEE Transactions on Visualization and Computer Graphics 22, 5 (May 2016), 1616--1629.
    [31]
    Claes Lundström, Thomas Rydell, Camilla Forsell, Anders Persson, and Anders Ynnerman. 2011. Multi-touch table system for medical visualization: Application to orthopedic surgery planning. IEEE Transactions on Visualization and Computer Graphics 17, 12 (Dec. 2011), 1775--1784.
    [32]
    Tahir Mahmood, Erik Butler, Nicholas Davis, Jian Huang, and Aidong Lu. 2018. Building multiple coordinated spaces for effective immersive analytics through distributed cognition. In Proc. BDVA. IEEE Computer Society, Los Alamitos, 119--129.
    [33]
    Veera Bhadra Harish Mandalika, Alexander I. Chernoglazov, Mark Billinghurst, Christoph Bartneck, Michael A. Hurrell, Niels de Ruiter, Anthony P. H. Butler, and Philip H. Butler. 2018. A hybrid 2D/3D user interface for radiological diagnosis. Journal of Digital Imaging 31, 1 (Feb. 2018), 56--73.
    [34]
    G. Elisabeta Marai, Angus G. Forbes, and Andrew Johnson. 2016. Interdisciplinary immersive analytics at the electronic visualization laboratory: Lessons learned and upcoming challenges. In Proc. IA. IEEE Computer Society, Los Alamitos, 54--59.
    [35]
    Stefan Marks, Javier E. Estevez, and Andy M. Connor. 2014. Towards the Holodeck: Fully immersive virtual reality visualisation of scientific and engineering data. In Proc. IVCNZ. ACM, New York, 42--47.
    [36]
    Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H. Thomas (Eds.). 2018. Immersive Analytics. Springer, Berlin/Heidelberg.
    [37]
    John P. McIntire, Paul R. Havig, and Eric E. Geiselman. 2014. Stereoscopic 3D displays and human performance: A comprehensive review. Displays 35, 1 (Jan. 2014), 18--26.
    [38]
    Microsoft. 2019. Microsoft HoloLens: A new way to see your world. Web site: https://www.microsoft.com/en-us/hololens/hardware. (2019). Accessed: Jan. 2019.
    [39]
    Alexandre Millette and Michael J. McGuffin. 2016. DualCAD: Integrating augmented reality with a desktop gui and smartphone interaction. In Proc. ISMAR-Adjunct. IEEE, Piscataway, NJ, USA, 21--26.
    [40]
    Ken Nagao, Yucong Ye, Chuan Wang, Issei Fujishiro, and Kwan-Liu Ma. 2016. Enabling interactive scientific data visualization and analysis with see-through HMDs and a large tiled display. In Proc. IA. IEEE Computer Society, Los Alamitos, 1--6.
    [41]
    Jeremi Niedziela and Barthélémy von Haller. 2017. Event visualisation in ALICE -- Current status and strategy for Run 3. Journal of Physics: Conference Series 898, 7, Article 072008 (2017), 8 pages.
    [42]
    Oculus. 2019. Oculus Rift: VR headset for VR ready PCs. Web site: https://www.oculus.com/rift/. (2019). Accessed: Mar. 2019.
    [43]
    Prabhat, Andrew Forsberg, Michael Katzourin, Kristi Wharton, and Mel Slater. 2008. A comparative study of desktop, fishtank, and Cave systems for the exploration of volume rendered confocal data sets. IEEE Transactions on Visualization and Computer Graphics 14, 3 (May 2008), 551--563.
    [44]
    Mickael Sereno, Lonni Besançon, and Tobias Isenberg. 2019. Supporting volumetric data visualization and analysis by combining augmented reality visuals with multi-touch input. In Posters EuroVis. 21--23.
    [45]
    Marcos Serrano, Barrett Ens, Xing-Dong Yang, and Pourang Irani. 2015. Gluey: Developing a head-worn display interface to unify the interaction experience in distributed display environments. In Proc. MobileHCI. ACM, New York, 161--171.
    [46]
    Takashi Shibata. 2002. Head mounted display. Displays 23, 1 (April 2002), 57--64.
    [47]
    Nicole Sultanum, Sowmya Somanath, Ehud Sharlin, and Mario Costa Sousa. 2011. "Point it, split it, peel it, view it": Techniques for interactive reservoir visualization on tabletops. In Proc. ITS. ACM, New York, 192--201.
    [48]
    Erik Sundén, Ingemar Lundgren, and Anders Ynnerman. 2017. Hybrid virtual reality touch table -- an immersive collaborative platform for public explanatory use of cultural objects and sites. In Proc. Eurographics Workshop on Graphics and Cultural Heritage. Eurographics Association, Goslar, Germany, 109--113.
    [49]
    Ivan E. Sutherland. 1968. A head-mounted three dimensional display. In Proceedings of the Fall Joint Computer Conference, Part I. ACM, New York, 757--764.
    [50]
    Zsolt Szalavári, Dieter Schmalsteig, Anton Fuhrmann, and Michael Gervautz. 1998. "Studierstube": An environment for collaboration in augmented reality. Virtual Reality 3, 1 (March 1998), 37--48.
    [51]
    Rensu P. Theart, Ben Loos, and Thomas R. Niesler. 2017. Virtual reality assisted microscopy data visualization and colocalization analysis. BMC Bioinformatics 18, 2, Article 64 (2017), 16 pages.
    [52]
    VIVE. 2019. VIVE: Discover virtual reality beyond imagination. Web site: https://www.vive.com. (2019). Accessed: Mar. 2019.
    [53]
    Ilija Vukotic, Edward Moyse, and Riccardo Maria Bianchi. 2015. ATLASrift -- A Virtual Reality application. arXiv preprint 1511.00047. https://arxiv.org/abs/1511.00047
    [54]
    Xiyao Wang, Lonni Besançon, Florimond Guéniat, Mickael Sereno, Mehdi Ammi, and Tobias Isenberg. 2019. A vision of bringing immersive visualization to scientific workflows. In Proc. CHI Workshop on Immersive Analytics. https://hal.archives-ouvertes.fr/hal-02053969
    [55]
    Colin Ware and Glenn Franck. 1996. Evaluating stereo and motion cues for visualizing information nets in three dimensions. ACM Transactions on Graphics 15, 2 (April 1996), 121--140.
    [56]
    Usher Will, Klacansky Pavol, Federer Frederick, Bremer Peer-Timo, Knoll Aaron, Yarch Jeff, Angelucci Alessandra, and Pascucci Valerio. 2018. A virtual reality visualization tool for neuron tracing. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan 2018), 994--1003.
    [57]
    Graham Wills. 2008. Linked data views. In Handbook of Data Visualization, Chun-houh Chen, Wolfgang Härdle, and Antony Unwin (Eds.). Springer, Berlin/Heidelberg, Chapter II.9, 217--241.
    [58]
    Bob G. Witmer and Michael J. Singer. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence 7, 3 (June 1998), 225--240.
    [59]
    Lingyun Yu, Konstantinos Efstathiou, Petra Isenberg, and Tobias Isenberg. 2016. CAST: Effective and efficient user interaction for context-aware selection in 3D particle clouds. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 886--895.

    Cited By

    View all
    • (2024)Exploring Spatial Organization Strategies for Virtual Content in Mixed Reality EnvironmentsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638181(1-6)Online publication date: 11-May-2024
    • (2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
    • (2024)A Survey of Designs for Combined 2D+3D Visual RepresentationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338851630:6(2888-2902)Online publication date: Jun-2024
    • Show More Cited By

    Index Terms

    1. Towards an Understanding of Augmented Reality Extensions for Existing 3D Data Analysis Tools

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
        April 2020
        10688 pages
        ISBN:9781450367080
        DOI:10.1145/3313831
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 23 April 2020

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. 3D visualization
        2. hybrid visualization system
        3. immersive analytics
        4. user interface

        Qualifiers

        • Research-article

        Conference

        CHI '20
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI PLAY '24
        The Annual Symposium on Computer-Human Interaction in Play
        October 14 - 17, 2024
        Tampere , Finland

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)203
        • Downloads (Last 6 weeks)19

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Exploring Spatial Organization Strategies for Virtual Content in Mixed Reality EnvironmentsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638181(1-6)Online publication date: 11-May-2024
        • (2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
        • (2024)A Survey of Designs for Combined 2D+3D Visual RepresentationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338851630:6(2888-2902)Online publication date: Jun-2024
        • (2024)Analyzing User Behaviour Patterns in a Cross-Virtuality Immersive Analytics SystemIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212930:5(2613-2623)Online publication date: May-2024
        • (2024)A Study on Collaborative Visual Data Analysis in Augmented Reality with Asymmetric Display TypesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210330:5(2633-2643)Online publication date: May-2024
        • (2024)This is the Table I Want! Interactive Data Transformation on Desktop and in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329960230:8(5635-5650)Online publication date: Aug-2024
        • (2024)chARpack: The Chemistry Augmented Reality PackageJournal of Chemical Information and Modeling10.1021/acs.jcim.4c0046264:12(4700-4708)Online publication date: 30-May-2024
        • (2023)DeAR: Combining Desktop and Augmented Reality for Visual Data AnalysisProceedings of the 25th Symposium on Virtual and Augmented Reality10.1145/3625008.3625021(233-237)Online publication date: 6-Nov-2023
        • (2023)Drawing Connections: Designing Situated Links for Immersive MapsProceedings of the ACM on Human-Computer Interaction10.1145/36042527:MHCI(1-26)Online publication date: 13-Sep-2023
        • (2023)Exploring Gaze-assisted and Hand-based Region Selection in Augmented RealityProceedings of the ACM on Human-Computer Interaction10.1145/35911297:ETRA(1-19)Online publication date: 18-May-2023
        • Show More Cited By

        View Options

        Get Access

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media