Skip to main content
SpringerLink
Log in

Importance Performance Matrix Analysis for Assessing User Experience with Intelligent Voice Assistants: A Strategic Evaluation

  • Original Research
  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

The digital transformation, in which we have actively participated over the last decades, involves integrating new technology into every aspect of the business and necessitates a significant overhaul of traditional business structures. Recently there has been an exponential increase in the presence of Artificial Intelligence (AI) in people’s daily lives, and many new AI-infused products have been developed. This technology is relatively young and has the potential to significantly affect both industry and society. The paper focuses on the Intelligent Voice Assistants (IVAs) and the User eXperience (UX) evaluation. IVAs are a relatively new phenomenon that has generated much academic and industrial research interest. Starting from the contribution to systematization provided by the Artificial Intelligence User Experience (AIXE®) scale, the idea is to develop an easy UX evaluation tool for IVAs that decision-makers can adopt. The work proposes the Partial Least Squares-Path Modeling (PLS-PM) to investigate different dimensions that affect the UX, and to verify if it becomes possible to quantify the impact and performance of each dimension on the general latent dimension of UX. The Importance Performance Matrix Analysis (IPMA) is utilised to evaluate and identify the primary factors that significantly influence the adoption of IVAs. IVA developers should examine the main aspects as a guide to enhancing the UX for individuals utilising IVAs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Latent dimensions, or also called latent variables, are aspects of a phenomenon that can only be inferred indirectly through a mathematical model from other observable variables that can be directly observed or measured.

  2. AIXE®is a measurement scale protected as unpublished work deposited at the Politecnico di Milano University.

References

  • Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P.N., Inkpen, K., et al. (2019). Guidelines for human-ai interaction. In: Proceedings of the 2019 Chi conference on human factors in computing systems, pp. 1–13.

  • Anderson, J. C., & Gerbing, D. W. (1982). Some methods for respecifying measurement models to obtain unidimensional construct measurement. Journal of Marketing Research, 19(4), 453–460.

    Article  Google Scholar 

  • Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411.

    Article  Google Scholar 

  • Arts, J. W., Frambach, R. T., & Bijmolt, T. H. (2011). Generalizations on consumer innovation adoption: A meta-analysis on drivers of intention and behavior. International Journal of Research in Marketing, 28(2), 134–144.

    Article  Google Scholar 

  • Balasudarsun, N., Sathish, M., & Gowtham, K. (2018). Optimal ways for companies to use facebook messenger chatbot as a marketing communication channel. Asian Journal of Business Research, 8(2), 1.

    Article  Google Scholar 

  • Bargas-Avila, J.A., & Hornbæk, K. (2011). Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 2689–2698.

  • Berdasco, A., López, G., Diaz, I., Quesada, L., & Guerrero, L. A. (2019). User experience comparison of intelligent personal assistants: Alexa, google assistant, siri and cortana. Multidisciplinary Digital Publishing Institute Proceedings, 31(1), 51.

    Google Scholar 

  • Bui, T. H., & Nguyen, V. P. (2023). The impact of artificial intelligence and digital economy on vietnam’s legal system. International Journal for the Semiotics of Law-Revue internationale de Sémiotique juridique, 36(2), 969–989.

    Article  Google Scholar 

  • Čaić, M., Odekerken-Schröder, G., & Mahr, D. (2018). Service robots: Value co-creation and co-destruction in elderly care networks. Journal of Service Management, 29(2), 178–205.

    Article  Google Scholar 

  • Castelo, N., Bos, M., & Lehman, D. (2018). Consumer adoption of algorithms that blur the line between human and machine. Graduate School of Business: Columbia University Working Paper, 23–37.

  • Cataldo, R., Grassia, M. G., Lauro, N. C., & Marino, M. (2017). Developments in higher-order pls-pm for the building of a system of composite indicators. Quality & Quantity, 51, 657–674.

    Article  Google Scholar 

  • Cataldo, R., Crocetta, C., Grassia, M. G., Lauro, N. C., Marino, M., & Voytsekhovska, V. (2021). Methodological pls-pm framework for sdgs system. Social Indicators Research, 156, 701–723.

    Article  Google Scholar 

  • Cataldo, R., Grassia, M. G., & Lauro, C. N. (2023). Alternative approaches to higher order pls path modeling: A discussion on methodological issues and applications. Partial Least Squares Path Modeling: Basic Concepts, Methodological Issues and Applications (pp. 229–266). Cham: Springer.

    Chapter  Google Scholar 

  • Chang, W., Franke, G. R., & Lee, N. (2016). Comparing reflective and formative measures: New insights from relevant simulations. Journal of Business Research, 69(8), 3177–3185.

    Article  Google Scholar 

  • Chin, W. W., et al. (1998). The partial least squares approach to structural equation modeling. Modern Methods for Business Research, 295(2), 295–336.

    Google Scholar 

  • Chi, O. H., Jia, S., Li, Y., & Gursoy, D. (2021). Developing a formative scale to measure consumers’ trust toward interaction with artificially intelligent (ai) social robots in service delivery. Computers in Human Behavior, 118, 106700.

    Article  Google Scholar 

  • Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60, 102383.

    Article  Google Scholar 

  • Cooper, P.W. (2018). Consumer intelligence series: Prepare for the voice revolution.

  • Davenport, T., Guha, A., Grewal, D., & Bressgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48, 24–42.

    Article  Google Scholar 

  • Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017). Ux design innovation: Challenges for working with machine learning as a design material. In: Proceedings of the 2017 Chi conference on human factors in computing systems, pp. 278–288.

  • García-Fernández, J., Fernández-Gavira, J., Sánchez-Oliver, A. J., Gálvez-Ruíz, P., Grimaldi-Puyana, M., & Cepeda-Carrión, G. (2020). Importance-performance matrix analysis (ipma) to evaluate servicescape fitness consumer by gender and age. International Journal of Environmental Research and Public Health, 17(18), 6562.

    Article  Google Scholar 

  • Gartner. (2019). Digital Disruption. Information Technology, Glossary Available at: https://www.gartner.com/it-glossary/digital-disruption

  • Google PAIR - People + AI Guidebook. (2019). Available at: https://design.google/ai-guidebook

  • Guenther, P., Guenther, M., Ringle, C. M., Zaefarian, G., & Cartwright, S. (2023). Improving pls-sem use for business marketing research. Industrial Marketing Management, 111, 127–142.

    Article  Google Scholar 

  • Hair, J. F., Ringle, C. M., & Sarstedt, M. (2011). Pls-sem: Indeed a silver bullet. Journal of Marketing theory and Practice, 19(2), 139–152.

    Article  Google Scholar 

  • Hair, J. F., Jr., Matthews, L. M., Matthews, R. L., & Sarstedt, M. (2017). Pls-sem or cb-sem: Updated guidelines on which method to use. International Journal of Multivariate Data Analysis, 1(2), 107–123.

    Article  Google Scholar 

  • Hair, J. F., Jr., Babin, B. J., & Krey, N. (2017). Covariance-based structural equation modeling in the journal of advertising: Review and recommendations. Journal of Advertising, 46(1), 163–177.

    Article  Google Scholar 

  • Hair, J., Hollingsworth, C. L., Randolph, A. B., & Chong, A. Y. L. (2017). An updated and expanded assessment of pls-sem in information systems research. Industrial Management & Data Systems, 117(3), 442–458.

    Article  Google Scholar 

  • Hair, J. F., Hult, G. T. M., Ringle, C. M., Sarstedt, M., & Thiele, K. O. (2017). Mirror, mirror on the wall: A comparative evaluation of composite-based structural equation modeling methods. Journal of the Academy of Marketing Science, 45, 616–632.

    Article  Google Scholar 

  • Hair Jr, J.F., Sarstedt, M., Ringle, C.M., Gudergan, S.P. (2017). Advanced Issues in Partial Least Squares Structural Equation Modeling. saGe publications.

  • Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of pls-sem. European Business Review, 31(1), 2–24.

    Article  Google Scholar 

  • Hair Jr, J.F., Hult, G.T.M., Ringle, C.M., & Sarstedt, M. (2021). A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). Sage publications.

  • Han, S., & Yang, H. (2018). Understanding adoption of intelligent personal assistants: A parasocial relationship perspective. Industrial Management & Data Systems, 118(3), 618–636.

    Article  Google Scholar 

  • Hasan, R., Shams, R., & Rahman, M. (2021). Consumer trust and perceived risk for voice-controlled artificial intelligence: The case of siri. Journal of Business Research, 131, 591–597.

    Article  Google Scholar 

  • Henseler, J. (2017). Bridging design and behavioral research with variance-based structural equation modeling. Journal of Advertising, 46(1), 178–192.

    Article  Google Scholar 

  • Hootsuite. (2022). Hootsuite Digital, Available at https://datareportal.com/

  • Hoy, M. B. (2018). Alexa, siri, cortana, and more: An introduction to voice assistants. Medical Reference Services Quarterly, 37(1), 81–88.

    Article  Google Scholar 

  • Ivanov, S., & Webster, C. (2019). Conceptual framework of the use of robots, artificial intelligence and service automation in travel, tourism, and hospitality companies. Robots, artificial intelligence, and service automation in travel, tourism and hospitality.

  • Jain, S., Basu, S., Dwivedi, Y. K., & Kaur, S. (2022). Interactive voice assistants-does brand credibility assuage privacy risks? Journal of Business Research, 139, 701–717.

    Article  Google Scholar 

  • Jiménez-Barreto, J., Rubio, N., & Molinillo, S. (2021). Find a flight for me, oscar! motivational customer experiences with chatbots. International Journal of Contemporary Hospitality Management, 33(11), 3860–3882.

    Article  Google Scholar 

  • Jones, V. K. (2018). Voice-activated change: Marketing in the age of artificial intelligence and virtual assistants. Journal of Brand Strategy, 7(3), 233–245.

    Google Scholar 

  • Jöreskog, K. G. (1970). A general method for analysis of covariance structures. Biometrika, 57(2), 239–251.

    Article  Google Scholar 

  • Kasilingam, D. L. (2020). Understanding the attitude and intention to use smartphone chatbots for shopping. Technology in Society, 62, 101280.

    Article  Google Scholar 

  • Kaur, D., Sahdev, S. L., Sharma, D., Siddiqui, L., et al. (2020). Banking 4.0:the influence of artificial intelligence on the banking industry & how ai is changing the face of modern day banks. International Journal of Management, 11(6), 577.

    Article  Google Scholar 

  • Latan, H. (2018). Pls path modeling in hospitality and tourism research: The golden age and days of future past. Applying partial least squares in tourism and hospitality research, 53–83.

  • Lauro, N. C., Grassia, M. G., & Cataldo, R. (2018). Model based composite indicators: New developments in partial least squares-path modeling for the building of different types of composite indicators. Social Indicators Research, 135, 421–455.

    Article  Google Scholar 

  • Law, E.L.-C., Roto, V., Hassenzahl, M., Vermeeren, A.P., & Kort, J. (2009). Understanding, scoping and defining user experience: a survey approach. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 719–728.

  • Lim, W. M., Kumar, S., Verma, S., & Chaturvedi, R. (2022). Alexa, what do we know about conversational commerce? Insights from a systematic literature review. Psychology & Marketing, 39(6), 1129–1155.

    Article  Google Scholar 

  • Lohmöller, J.-B. (2013). Latent Variable Path Modeling with Partial Least Squares. Cham: Springer.

    Google Scholar 

  • Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019). Resistance to medical artificial intelligence. Journal of Consumer Research, 46(4), 629–650.

    Article  Google Scholar 

  • Loureiro, S. M. C., Japutra, A., Molinillo, S., & Bilro, R. G. (2021). Stand by me: Analyzing the tourist-intelligent voice assistant relationship quality. International Journal of Contemporary Hospitality Management, 33(11), 3840–3859.

    Article  Google Scholar 

  • Malodia, S., Islam, N., Kaur, P., & Dhir, A. (2021). Why do people use artificial intelligence (ai)-enabled voice assistants? IEEE Transactions on Engineering Management., 71, 491–505.

    Article  Google Scholar 

  • Mariani, M. M., Perez-Vega, R., & Wirtz, J. (2022). Ai in marketing, consumer research and psychology: A systematic literature review and research agenda. Psychology & Marketing, 39(4), 755–776.

    Article  Google Scholar 

  • Martilla, J. A., & James, J. C. (1977). Importance-performance analysis. Journal of Marketing, 41(1), 77–79.

    Article  Google Scholar 

  • McLean, G., & Osei-Frimpong, K. (2019). Hey alexa\(\ldots\) examine the variables influencing the use of artificial intelligent in-home voice assistants. Computers in Human Behavior, 99, 28–37.

    Article  Google Scholar 

  • Pitardi, V., & Marriott, H. R. (2021). Alexa, she’s not human but\(\ldots\) unveiling the drivers of consumers’ trust in voice-based artificial intelligence. Psychology & Marketing, 38(4), 626–642.

    Article  Google Scholar 

  • Pradhan, A., Findlater, L., & Lazar, A. (2019). " phantom friend" or" just a box with information" personification and ontological categorization of smart speaker-based voice assistants by older adults. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–21.

  • Radziwill, N.M., & Benton, M.C. (2017). Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579

  • Rana, J., Gaur, L., Singh, G., Awan, U., & Rasheed, M. I. (2021). Reinforcing customer journey through artificial intelligence: A review and research agenda. International Journal of Emerging Markets, 17(7), 1738–1758.

    Article  Google Scholar 

  • Ringle, C. M., & Sarstedt, M. (2016). Gain more insight from your pls-sem results: The importance-performance map analysis. Industrial Management & Data Systems, 116(9), 1865–1886.

    Article  Google Scholar 

  • Russolillo, G. (2012). Non-metric partial least squares.

  • Sanchez, G., Trinchera, L., Sanchez, M. G., & FactoMineR, S. (2013). Package “plspm.” Citeseer: State College, PA, USA.

  • Sarstedt, M., Hair, J. F., Pick, M., Liengaard, B. D., Radomir, L., & Ringle, C. M. (2022). Progress in partial least squares structural equation modeling use in marketing research in the last decade. Psychology & Marketing, 39(5), 1035–1064.

    Article  Google Scholar 

  • Sciannamè, M., Zavarrone, E., et al. (2022). The qualities of ai-infused products. reflections on emerging ux dimensions. In: Embedding Intelligence. Designerly Reflections on AI-infused Products, pp. 48–76. Franco Angeli Design International.

  • Spallazzo, D., & Sciannamè, M. (2022). Embedding intelligence: designerly reflections on ai-infused products. Embedding intelligence, 1–127.

  • Tenenhaus, M., Vinzi, V. E., Chatelin, Y.-M., & Lauro, C. (2005). Pls path modeling. Computational Statistics & Data Analysis, 48(1), 159–205.

    Article  Google Scholar 

  • Yang, Q. (2020). Profiling artificial intelligence as a material for user experience design. PhD thesis, Carnegie Mellon University.

  • Yang, H., & Lee, H. (2019). Understanding user behavior of virtual personal assistant devices. Information Systems and e-Business Management, 17, 65–87.

    Article  Google Scholar 

  • Zarour, M., & Alharbi, M. (2017). User experience framework that combines aspects, dimensions, and measurement methods. Cogent Engineering, 4(1), 1421006.

    Article  Google Scholar 

  • Zavarrone, E., & Cataldo, R. (2022). Cfa & pls-pm for ux-ai product infused. In: Proceedings Of The 2022 SIS Conference.

Download references

Funding

This research received no external funding.

Author information

Authors and Affiliations

  1. Department of Social Sciences, University of Naples Federico II, Naples, Italy

    Rosanna Cataldo, Maria Gabriella Grassia & Marina Marino

  2. Department of Humanities, IULM, Milan, Italy

    Martha Friel & Emma Zavarrone

Authors
  1. Rosanna Cataldo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martha Friel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Gabriella Grassia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marina Marino
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emma Zavarrone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rosanna Cataldo.

Ethics declarations

Conflict of interest

The authors declare no Conflict of interest.

Ethics approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent for publication

All authors have read and agreed to the published version of the manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cataldo, R., Friel, M., Grassia, M.G. et al. Importance Performance Matrix Analysis for Assessing User Experience with Intelligent Voice Assistants: A Strategic Evaluation. Soc Indic Res (2024). https://doi.org/10.1007/s11205-024-03362-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11205-024-03362-3

Keywords

Search

Navigation