Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system design

2.1 The central role of cognition research in HCI’s past (and future)

Human cognition has always been at the core of HCI research. Long before computers and trying to understand human information processing, ¹⁰ the origins of cognitive psychology, looking at memory and attention, are typically attributed to the late 1800s. ¹¹ It is important to note that many assumptions in HCI are grounded in the findings of cognitive psychology, including e.g. divided attention, spatial and episodic memory, and dual processing. Indeed, many HCI researchers, and much of an HCI curriculum, come from cognitive psychology backgrounds. A key challenge for research now, as discussed later, is to properly understand different types of cognitive activity (e.g. ¹² ) and how to identify, track, and measure them. Due to this very close relationship between HCI and human cognition, we expect this history to have renewed importance in the developments of cognitively focused wearables.

With a key focus on health and safety, many findings from cognitive psychology were adopted in Human Factors and Cognitive Ergonomics. ^{13 , 14} Traditionally, major concerns have been, for example in understanding the capacity of people in safety-critical job roles like air traffic control ^{15 , 16} and designing shift work based on people’s ability to maintain focus and to recover effectively from fatigue. ¹⁷ It is perhaps not surprising that these use cases have inspired early research into studying air traffic controllers with neurotechnology ^{18 , 19} and that industry has seen initial examples of applying neurotechnology in safety helmets,^[1] to increase profits. The basis for many subjective and behavioural measures of cognitive activity in HCI research comes from human factors and cognitive psychology research. Similarly, HCI is not alone in its drive to build on top of cognitive psychology. Educational psychology, for example, has seen similar models develop for attention and cognitive load ²⁰ to inform the design of instructional material and understand how people best study and learn.

2.2 Inferring cognition from observational and self-reported data

As it emerged as a distinct research field in the 1980s, HCI research has long sought to understand the cognitive aspects of people’s experiences with technology. Early analytical inspection methods in HCI were based on cognitive psychology models, including GOMS ²¹ and the Cognitive Walkthrough. ²² Those methods allowed researchers to evaluate interaction with an interface in reference to a cognitive model. When working with users, research relied on four main techniques: (1) performance measures (e.g. task completion time, correctness, errors), (2) behavioural measures (linguistic features such as sentence length or word complexity), (3) subjective self-reports (e.g. questionnaires), and (4) psychophysiological measures (e.g. heart rate, pupil dilation, blink rate, electrodermal activity, EEG). ^{23 , 24}

Performance and behavioural measures are a valuable way of assessing people’s capacity to complete tasks, often analysed through observing performance drops or mistakes. Where performance might be measured in time to complete tasks, it is typically measured in terms of negative outcomes, such as reduced performance and error rate. Consequently, much research turned to inferring aspects of cognition from people’s ability to complete secondary tasks parallel to their work. Open protocols such as the think-aloud method ²⁵ encourage participants to express their experiences and perceptions in the interaction with a system or technology. These insights, in turn, can be applied to gain insights into cognitive processes. Speech analysis can help estimate, e.g. mental workload through sentence length or complexity, tempo, ²⁶ or pauses. ²⁷ Other behavioural approaches rely on, e.g. body posture analysis for fatigue ^{28 , 29} or on analysis of people’s mouse usage patterns. ³⁰

Performance and behavioural measures largely rely on experimenters to interpret observations. Complementary subjective self-reports can help researchers assess cognition from the user’s individual perspective, focusing on subjective perception and experiences. There are many subjective scales available to estimate different forms of cognitive activity and suit different purposes. These measures can be of various levels of complexity and generality, from one-item scales designed for participants to state at intervals during tasks (e.g. ISA ³¹ ) to multi-dimensional questionnaires. One of the most commonly used multi-dimensional scales in HCI comes, again, from safety-critical human factors research that was concerned with astronauts breaching their ability to handle workload. ³² The NASA Task Load Index (TLX) was produced as a detailed and retrospectively applied multi-dimensional subjective rating scale. ^{32 , 33} By being multi-dimensional, NASA TLX recognised that there were facets of both physical and cognitive activity that were important to try and measure, including frustration, mental workload, and temporal workload on the cognitive side.

2.3 Neurotechnology and psychophysiological measures

However, prior work shows that it can be challenging for people to correctly assess their cognitive functioning, especially concerning neurological deficits (e.g. ^{34 , 35} ). Consequently, much attention has been given to psychophysiological measures of cognitive activity, such as EEG, fNIRS, gaze features (specifically for attention), heart rate, and temperature. Such research started in the early 70s with EEG, ³⁶ primarily in medical contexts. Similarly, eye tracking was used in medical circumstances as an input mechanism for disabled users, e.g. for text entry and device control. Many research fields now seek to classify cognitive activity with psychophysiological data, including HCI. Alsurakh et al. highlight that for every physiological signal used to estimate mental workload, the same signals have been used to estimate stress levels. ³⁷ Where research has demonstrated that advanced neurotechnology can be used to identify changes in cognitive activity, it is now largely established that, e.g. heart rate variability reflects stress levels and current smartwatches are beginning to track stress levels in the wild.

Wilson et al. argue that cognitive activity tracking is at a turning point, similar to the move from scientific stages of activity tracking in lab conditions to its widespread proliferation within wearable technology. ⁴ Wearable neurotechnology devices have become increasingly small and powerful and have even been released as consumer devices. Similarly, where machine learning research has made significant advances in improving the accuracy of classifying electro- and neurophysiological signals, recent advancements in AI mean we are likely to see reliable classification of cognitive activity, in consumer devices, in the near future. Notably, we increasingly see the application of consumer-grade EEG technologies in HCI research, such as the Emotiv Epoc ^{38 – 41} or the Muse EEG. ^{42 , 43} These research settings include improving focus at work, use in the creative arts, and integration into environments like virtual or augmented reality. ^{44 , 45} However, the study of how people track their activity over time (Personal Informatics, PI ⁴⁶ ) will need to study how people manage their Cognitive Personal Informatics. This implies that HCI will need to study how PI processes ⁴⁷ are altered by the inclusion of cognition data, particularly in key PI aspects such as goal setting ⁴⁸ and lapsing. ⁴⁹

In many respects, the development of reliable neurotechnology suddenly makes cognitive activity tracking more an HCI research problem, rather than a psychology, human factors, medical engineering, or machine learning problem. Consequently, more researchers will soon have access to devices that can take such measures about people, and people will have unprecedented access to new forms of cognitive data about themselves. Furthermore, society will have to navigate social, privacy, ethics, and trust issues as consumer neurotechnology becomes involved in domains such as medicine and care, driving and insurance, and indeed its use in the workplace. Below, we discuss how maturing neurotechnology will change HCI.

3.1 Implications for methodology – an unstable period of method change

The rigour behind current research using advanced neurotechnology has developed standards over time that speak to the validity of research results. These standards were developed to build confidence and trust in research that is developing a technology and “proving” that it can accurately measure cognitive activity or detect changes. Once the accuracy of such devices is assumed, we expect there will be a period where new community norms will develop for how such tools can be reliably integrated into user studies alongside other methods. For example, Pike et al. studied whether fNIRS can be used alongside the think-aloud protocol as a popular qualitative approach to gaining insight into what people are thinking. ⁵⁰ As with any new method, therefore, we expect that studies will be published that use cognitive measurements before we, as a community, have developed expectations for research quality.

Perhaps more importantly, in the case of measuring cognitive activity, we expect that a large amount of research will begin to publish findings before we really understand what they mean. We can ask questions now, for example, about whether cognitive activity should be “high” or “low” if people are good at doing their job? While many aspects of cognitive activity have been studied extensively in psychology in controlled conditions, we do not yet know what data will be captured from participants in less controlled scenarios and what we should expect to see in all forms of interaction.

We also expect to see a shift in what we learn as we move from subjective measures that cause people to reflect on their experiences, to what might be considered unbiased objective data from people during experiences. In many respects, maturing neurotechnology will allow us to evaluate the difference between what is observed about cognitive activity during experiences, and what people say about them afterwards. For example, the think-aloud method has been discussed for whether it adds cognitive burden to users as a secondary task. ⁵¹ Thus, it has been debated whether it accurately reflects the difficulty of doing a task, or instead reflects the difficulty of discussing what people are doing during a task. Another common concern for assessing cognitive overload is when participants are no longer able to think aloud or forget to provide mid-task subjective ratings (e.g. ISA ³¹ ). In these cases, it is typically considered that at these moments, participants have a very high mental workload and thus are unable to carry out these secondary tasks. ⁵² In this case, performance in a secondary task is often used to measure spare cognitive capacity. ⁵³ Overall, we expect research to be in flux while it decides whether it can accurately detect the same moments, whether these subjective and qualitative methods are still needed, and whether important qualitative context is lost from no longer asking participants to reflect on experiences.

3.2 Implications for theory – learning from long-term real-world data

Given that devices and systems to measure cognitive activity will become widely available and affordable research tools, we expect these measures to become mainstream for HCI studies (alongside other methods). Through increased research, we will gain a more in-depth understanding of many theoretical concepts related to cognition, such as stress, attention, fatigue, or memory.

Unobtrusive consumer neurotechnology will also allow researchers to gather long-term data and data from a huge variety of situations with reduced need for supervising participants. Whether users are already wearing devices that will start to collect data about cognitive activity, such as smart watches, or whether researchers provide participants with devices to wear, we can expect more studies to ask people to collect cognitive data in different parts of our everyday lives in in-the-wild methodologies. Data collected in natural situations will enrich our understanding of theoretical concepts often biased by laboratory conditions, such as attention and interruption management, productivity research, or stress management.

With users becoming more aware of their cognition and systems that can react to cognition, there is a growing need to rethink assumptions about theory in HCI. This evolution may redefine how people perceive and interact with everyday technology. As our understanding of cognitive processes deepens and integrates with technological systems, the traditional frameworks and theories in HCI are likely to undergo significant transformations. This change is not just a matter of technological advancement, but also a shift in the user’s self-awareness and the system’s responsiveness to cognitive states. Thus, it will potentially lead to questioning of how we design, use, and think about technology in our daily lives.

3.3 Implications for design – towards real-world cognition-aware systems

With an increased understanding of cognitive processes during the interaction with intelligent systems, we expect to see a rise in designing adaptive technologies based on cognitive performance or states. As discussed in Section 3.1, having real-time insights into users’ cognition during their interaction with technology can offer new insights into user-centred design processes. Neurotechnology will enable more evidence-based design decision through effectively informing design processes. ⁵⁴

Cognition-aware systems represent a sophisticated evolution in context-aware computing, ⁵⁵ extending beyond traditional input dimensions like location and physical activity to encompass the cognitive context of the user. This cognitive context includes mental information processing aspects such as attention allocation, ⁵⁶ perception, ⁵⁷ memory encoding, storage and retrieval, ^{58 , 59} and learning. ⁶⁰

Today, knowledge workers process a wealth of information while engaging in continuous learning. Here, the first challenge involves technologies that rarely consider users’ fluctuating attention levels, receptiveness, or cognitive capacities throughout the day. Recent years have brought significant advancements in context-aware systems like Cybre-Minder, ⁶¹ which supports users in managing reminders by utilising rich contexts such as time and location. Nonetheless, there remains a gap in technology’s ability to adapt to users’ cognitive states and physiological correlates, e.g. circadian rhythms.

Secondly, the prevalent issue in mobile computing is the distracting nature of reminders and alerts, which often ignore the user’s current context. Despite research on delaying notifications, little has been implemented in consumer products. Context-aware reminding systems like the Jogger prototype ⁶² and smart home applications ⁶³ show promise in utilizing probabilistic models and context information to predict opportune moments for reminder delivery, thereby reducing disruptions and improving task flow.

Third, another opportunity for design with neurotechnologies lies in adjusting information complexity to align with users’ cognitive capacities. Current user interfaces seldom adapt to varying states of sustained attention. Research on adaptive UIs that respond to attentional states is essential for improving task efficiency. The need for UIs that can dynamically adjust to attention fluctuations is echoed in studies exploring sensor-driven adaptive reminder systems, ⁶⁴ context-aware approaches for behaviour analysis ⁶⁵ and adjustments of surrounding task-irrelevant information across the Mixed Reality continuum. ^{56 , 66 , 67}

Lastly, the availability of cognition-aware systems further provides opportunities for enhanced inclusivity. With easier access to data on users’ current cognitive capacities and concurrent visualisation, information, or interaction adaptations ⁶⁸ it will be easier to include and design for individuals with diverse abilities and needs. Prior research emphasised the need to include the neurodivergent population in the research discussions. ^{69 , 70} Yet, to date, there is little research on the potential of consumer neurotechnology for these groups and the associated expectations, needs, and design requirements.

Yet, all opportunities also present significant challenges as the intersection of various fields introduces a complex learning curve for researchers and designers aiming to integrate physiological signals into their studies and interactive systems. This multidisciplinary challenge necessitates a broad understanding and mastery of concepts spanning from biology to computer science, complicating the development process of cognition-aware systems. ⁷¹

Firstly, seamlessly incorporating basic psychophysiological theory into system design is essential for creating meaningful user interactions that accurately reflect psychological states. This requires a large interdisciplinary effort as mapping a physiological signal to a specific user state requires an understanding of psychophysiological inference and how physiological signals might differently react over time and show different responsitivity to various stimuli. ⁷²

Secondly, the identification of signal processing requirements is critical for the accuracy and reliability of cognitive state assessments, demanding the design and implementation of device-independent software tools. These challenges highlight the need for sophisticated methodologies to accurately interpret and respond to the complex spectrum of human cognition. Here, data protection rights should be considered, allowing users to retain formal and legal control over their psychophysiological data. This stipulates that any third party should receive access to such information solely with the user’s explicit approval. ⁷³ Here, privacy-by-design approaches should be considered ⁷⁴ and privacy-preserving mechanisms should be ensured. Here, federated learning presents an innovative solution by allowing for the training of models without the need to centralize sensitive physiological data, thus ensuring privacy and regulatory compliance. ⁷⁵

Third, we need to consider challenges from a user interaction level, such as user acceptance and trust. Users may have concerns about the implications of systems that monitor and respond to their cognitive states if the data presented is incorrect or inaccurate. Moreover, the integration of neurotechnology at such a scale into people’s everyday lives could lead to an overreliance on technology. As an effect, overreliance could potentially diminish users’ perceived agency and subjective intuition. Designers must be cautious not to let technology override human-centred design principles. Balancing the assessments of neurotechnologies with human introspection, self-awareness, and decision-making is crucial to prevent over-dependence and ensure that technology serves as a tool to enhance, rather than dictate, user experience.

3.4 Implications for practice – responsible integration into application contexts

The increasing use of neurotechnology has the potential to influence various fields of practice in significant ways (e.g. in the workplace). Here, HCI researchers play a crucial role in navigating the introduction of neurotechnology into daily practices, ensuring that these developments are approached thoughtfully. It is important that the potential repercussions of introducing neurotechnology in new contexts are studied in-depth and that the findings are communicated effectively, reaching beyond the academic realm to inform real-world applications. HCI practitioners must actively translate insights from other disciplines, such as neuroscience and psychology, into usable and ethical designs that leverage neurotechnology. This interdisciplinary approach is essential for creating neurotechnology applications that are not only technically sound but also socially responsible and beneficial. Moreover, it is crucial for HCI experts to bridge the gap between theoretical research and practical implementation. Researchers have to ensure that the integration of neurotechnology across application domains is seamless, user-friendly, and enhances rather than detracts from human experience and productivity.

For instance, we hypothesise that for knowledge workers, the introduction of neurotechnology may bring changes in monitoring cognitive processes and emotional states of employees. In the educational sector, neurotechnology can support the learning process, as research has already shown the application of cognitive data in teaching and learning analytics. ^{76 , 77} Pedagogical experts, teachers, and HCI scholars need to work together to understand the experiential aspects of these technologies. They should identify practical applications while carefully considering any negative aspects, especially for vulnerable groups like children and neurodivergent individuals. Similarly, in the healthcare sector, neurotechnology offers promising avenues for enhancing patient care and treatment methods. This advancement could potentially provide a more nuanced understanding of neurological and psychological conditions, leading to more tailored treatment approaches. However, healthcare professionals, along with HCI researchers, must thoughtfully assess how to implement such technologies in clinical settings, taking into account patient care, privacy, and treatment efficacy. The implications for practice are considerable, requiring professionals to stay updated and adapt to the changing intersection of technology and human health.

The introduction of neurotechnology into various sectors is a complex scenario marked by both potential benefits and challenges. While these technologies can bring about substantial advancements in new application contexts, they also present opportunities for researchers to gain a deeper understanding of the technologies themselves. Utilising neurotechnology as a research tool, HCI researchers can explore their full spectrum of capabilities and limitations. However, this exploration should encompass more than the perspectives of HCI researchers alone. A comprehensive contextual understanding is crucial, highlighting the need for involvement from the target population in the research process. Involving end-users ensures that the research is grounded in practical realities and addresses the actual needs and concerns of those interacting with these technologies. This inclusive approach is essential. It ensures a balanced evaluation of the potential benefits and risks of neurotechnology, guiding them towards their responsible and ethical application. We note that the risk of exploitation, particularly through constant performance monitoring, is a significant concern as neurotechnology becomes more integrated into everyday life. These tools may infringe on personal privacy or exert undue pressure on individuals, especially in workplace settings where performance is closely scrutinized. Thus, it is imperative to establish strong ethical guidelines and regulatory frameworks for interaction design practice to protect individual rights. It is crucial that we ensure that the advancement of neurotechnology aligns with societal values and respects the dignity of all users.

3.5 Implications for policy – ethics and mental privacy

Unlike our body, face, fingerprints, and speech, which have been subject to tracking for a certain time by now, our mental processes have remained private. At the current stage of research, it is unclear what information could be derived from large real-world data sets of cognitive data. The discussion of NeuroEthics is slowly finding its way into HCI (cf. ^{78 – 80} ) to prepare us for the same future we predict here. In lieu of wide-scale access to neurotechnology, discussions of NeuroEthics have had to remain largely speculative about how cognitive personal informatics data might be used and misused. ^{79 , 81} The news has reported a small number of specific cases where e.g. employers have incorporated neurotechnology into work-wear and profited from managing their workforce using this data. This ethical discussion has already been taken up by researchers (cf. ⁸² ). We expect that we will rapidly encounter many ethical, privacy, and trust issues as such examples become more widespread, and employers choose to try and increase business profits in this way. Other potentially critical cases include the detection of medical conditions in large data sets that have been found in laboratory conditions using medical grade EEG, such as epilepsy, ⁸³ cognitive impairment, ⁸⁴ developmental disorders in children, ⁸⁵ and substance abuse disorders. ⁸⁶ It is unclear if these signal differences could be reliably identified using consumer devices in real-world settings.

One challenge research we will face also relates to how such data is treated in frameworks such as the General Data Protection Regulation (GDPR):^[2] whether data from neurotechnology is medical data, personal data, or protected data. Furthermore, certain types of cognitive data, such as brainwaves collected by EEG devices, can be used to uniquely identify individuals ⁸⁷ and have been shown to function as an authentication method with an accuracy of more than 99 %. ⁸⁸ We consider it an important starting point for HCI to help users understand what cognitive data is being collected, what information could be drawn from it, and for what purpose it is used. In that regard, it is critical to also discuss the conflict between the commercial application of neurotechnologies and its use in research. The objectives of neurotechnology companies can conflict with the ethical standards of academic research, particularly in terms of user consent, data usage, privacy, and transparency. Policies should enforce transparency requirements for neurotechnology of both academic and commercial origin. We need to ensure that consent is informed and freely given as well as clear communication about data practices and privacy.

Regulatory bodies have already become active in the topic (e.g. by the UNESCO^[3]). Yet, the integration of ethical concerns surrounding neurotechnology into existing digital ethics frameworks (e.g. data ethics, AI ethics) remains an open issue. Parallels can be drawn to considerations from other emerging technologies, such as AI (cf. ⁷⁸ ). Similarly to how challenges are addressed in this complex field, the governance of neurotechnology should involve multidisciplinary collaboration, including experts from technology, ethics, law, and social sciences. This collaborative approach can ensure that technological advancements are aligned with societal values and ethical principles.

3.6 Implications for society – balancing performance quantification and mental health

The advent of emerging neurotechnology, particularly in the realm of cognitive personal informatics, poses profound societal implications that extend well beyond the field of HCI. Currently, it is already common to track and share physical activity data (e.g. from running, cycling, or hiking) as a form of motivation or social sharing, even if not for the explicit purpose of showcasing achievements. However, the landscape of cognitive personal informatics is filled with uncertainties, especially regarding the social and privacy norms that might develop once tracking and sharing cognitive activity becomes as accessible as physical activity tracking.

Key questions arise: Will the norms established for sharing physical activity data translate seamlessly to cognitive activity data? How comfortable will individuals be in sharing information about such intimate and internal activities? The potential for shared accountability in cognitive data could offer benefits, but it also raises questions about what kind of cognitive ‘achievements’ people might share or broadcast.

As research in this field progresses, we anticipate a shift from hypothetical inquiries about how people might feel sharing hypothetical cognitive data, to empirical studies examining the actual behaviours and attitudes surrounding the sharing of real neurotechnology data. This shift will undoubtedly include exploring individuals’ reactions to requests for sharing their cognitive data with others. The outcomes of such research will not only inform the development of the technologies themselves. They will also shape our understanding of their broader societal impact, particularly in terms of privacy, social interaction, and our conceptualisation of cognitive health and achievement.

In light of these technological advancements, another critical concern is how society might re-conceptualise the notion of ‘achievement’. The prospect of tracking cognition on a larger scale prompts us to rethink what achievement means beyond the physical realm. In return, we expect it to lead to a more nuanced understanding that recognizes the concept’s complexities and subjective meaning. Such measurements might illuminate traditionally overlooked aspects of cognitive prowess and mental well-being. However, alongside these potential insights, there are significant risks of negative repercussions. For instance, the ability to quantify ‘achievement’ in cognitive terms could lead to misuse, especially in contexts like the workplace, where cognitive metrics might be inappropriately used to assess performance. This raises concerns about a societal shift towards valuing only what can be numerically measured, sidelining important qualitative aspects of human experience. These considerations underscore the need for a careful, ethical approach to developing and applying neurotechnology to ensure they enhance understanding and well-being without becoming tools for narrow, potentially harmful quantification of human capabilities.

Looking ahead, it is crucial to understand the impact of emerging neurotechnologies on the daily routines of everyday users becomes a key concern. While topics such as tracking obsession or digital addiction have already been researched in HCI and related fields (cf. ^{89 , 90} ), this topic could also become relevant in the field of neurotechnologies. This is particularly the case when these technologies are integrated with gamified mobile applications that promote the gaining of points, streaks, and awards. While potentially beneficial in specific contexts, such engaging features could lead to overuse and an unhealthy preoccupation with tracking and quantifying mental states. ⁹¹ Designing these technologies with a balanced approach is essential to positively contribute to users’ mental health and well-being without encouraging detrimental usage habits.

Another relevant aspect concerning users’ well-being focuses on specific well-being support such technologies could provide. Currently, commercial neurotechnologies provide features such as tracking brain activity, facilitating meditation training, managing stress, providing feedback through visualisations, and offering descriptive metrics like “calm,” “neutral,” and “active” (see Muse ³ ). However, the societal impact (and the direction of such an impact) of these technologies is partly dependent on the clarity and significance of these metrics, which often remain ambiguous. For instance, while a “calm” state might positively correlate with a normal heart rate in physiological terms, its implications when associated with brain activity are not as straightforward. This lack of clarity in interpreting cognitive states could potentially lead to widespread unhealthy usage patterns or negative thought cycles across society. Therefore, these technologies need to provide clear, contextually relevant information to support informed decisions about mental wellness, ensuring they enhance rather than compromise the well-being of users on a societal scale. To summarise, how these technologies address and communicate complex cognitive states will significantly influence societal perceptions of mental health and well-being, potentially reshaping our collective understanding of what it means to be mentally healthy or stressed.