Skip to main content
SpringerLink
Log in

Models of Human Emotion and Artificial Emotional Intelligence

  • Chapter
  • First Online:
Modeling Visual Aesthetics, Emotion, and Artistic Style

  • 134 Accesses

Abstract

This chapter bridges the gap between emotion models popular in psychology and their use in affective computing tasks. Emotion modeling has a long and varied history with several competing schools of thought. Here, through a survey of existing literature, we cover existing emotion models popular in psychology, highlighting the strengths and weaknesses of these different approaches in regard to computational tasks involving human emotion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
eBook
USD 139.00
Price excludes VAT (USA)
Hardcover Book
USD 179.99
Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Over 14,000 emotion recognition papers since 2010 according to IEEE Xplore.

  2. 2.

    These include SEEKING (expectancy), FEAR (anxiety), RAGE (anger), LUST (sexual excitement), CARE (nurturing), PANIC/GRIEF (sadness), and PLAY (social joy).

  3. 3.

    Valence and pleasure are often used interchangeably so this is also sometimes referred to as VAD for valence, arousal, and dominance.

  4. 4.

    For example in Cowen et al.’s study on self-reported emotions from video, the videos were gathered from prompts based on 34 predetermined emotion categories.

References

  1. Abelson, R.P., Sermat, V.: Multidimensional scaling of facial expressions. J. Exp. Psychol. 63(6), 546–554 (1962)

    Article  Google Scholar 

  2. Ahuja, C., Lee, D.W., Ishii, R., Morency, L.P.: No gestures left behind: learning relationships between spoken language and freeform gestures. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 1884–1895 (2020)

    Google Scholar 

  3. Ahuja, C., Lee, D.W., Nakano, Y.I., Morency, L.P.: Style transfer for co-speech gesture animation: a multi-speaker conditional-mixture approach (2020). CoRR abs/2007.12553 ArXiv: 2007.12553

    Google Scholar 

  4. Anderson, A., Christoff, K., Stappen, I., Panitz, D., Ghahremani, D.G., Glover, G., Gabrieli, J., Sobel, N.: Dissociated neural representations of intensity and valence in human olfaction. Nat. Neurosci. 6(2), 196–202 (2003)

    Article  Google Scholar 

  5. Arnold, M.B.: Emotion and Personality Psychological Aspects, vol. 1. Columbia University Press, New York (1960)

    Google Scholar 

  6. Bagher, Z.A., Liang, P.P., Poria, S., Cambria, E., Morency, L.P.: Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Long Papers, vol. 1, pp. 2236–2246. Association for Computational Linguistics, Melbourne (2018)

    Google Scholar 

  7. Bann, E.Y., Bryson, J.J.: Measuring cultural relativity of emotional valence and arousal using semantic clustering and twitter (2013). CoRR abs/1304.7507. _eprint: 1304.7507

    Google Scholar 

  8. Barrett, L.F., Lindquist, K.A., Gendron, M.: Language as context for the perception of emotion. Trends Cognitive Sci. 11(8), 327–332 (2007)

    Article  Google Scholar 

  9. Barrett, L.F., Mesquita, B., Gendron, M.: Context in emotion perception. Curr. Dir. Psychol. Sci. 20(5), 286–290 (2011)

    Article  Google Scholar 

  10. Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., Wermter, S.: The OMG-emotion behavior dataset. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE, Rio de Janeiro (2018)

    Google Scholar 

  11. Breckler, S.J.: Empirical validation of affect, behavior, and cognition as distinct components of attitude. J. Pers. Soc. Psychol. 47(6), 1191–1205 (1984)

    Article  Google Scholar 

  12. Bubeck, S.A., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M.T., Zhang, Y.: Sparks of artificial general intelligence: early experiments with GPT-4 (2023). ArXiv:2303.12712 [cs]

    Google Scholar 

  13. Burton, S.J., Samadani, A.A., Gorbet, R., Kulić, D.: Laban movement analysis and affective movement generation for robots and other near-living creatures. In: Dance Notations and Robot Motion, pp. 25–48. Springer, Berlin (2016)

    Google Scholar 

  14. Burunat, E.: Love is a physiological motivation (like hunger, thirst, sleep or sex). Med. Hypotheses 129, 109225 (2019)

    Article  Google Scholar 

  15. Carroll, J.B., Osgood, C.E., May, W.H., Miron, M.S.: Cross-cultural universals of affective meaning. Am. J. Psychol. 89(1), 172 (1976)

    Article  Google Scholar 

  16. Cordaro, D.T., Sun, R., Kamble, S., Hodder, N., Monroy, M., Cowen, A., Bai, Y., Keltner, D.: The recognition of 18 facial-bodily expressions across nine cultures. Emotion 20(7), 1292–1300 (2020)

    Article  Google Scholar 

  17. Cowen, A.S., Keltner, D.: Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proc. Natl. Acad. Sci. 114(38), E7900–E7909 (2017)

    Article  Google Scholar 

  18. Cowen, A.S., Keltner, D.: What the face displays: mapping 28 emotions conveyed by naturalistic expression. Am. Psychol. 75(3), 349–364 (2020)

    Article  Google Scholar 

  19. Cowen, A.S., Elfenbein, H.A., Laukka, P., Keltner, D.: Mapping 24 emotions conveyed by brief human vocalization. Am. Psychol. 74(6), 698–712 (2019)

    Article  Google Scholar 

  20. Darwin, C.: The Expression of the Emotions in Man and Animals. University of Chicago Press, Chicago (2015)

    Google Scholar 

  21. Davies, K.J.: Adaptive homeostasis. Mol. Aspects Med. 49, 1–7 (2016)

    Article  Google Scholar 

  22. Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: GoEmotions: a dataset of fine-grained emotions. In: 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4040–4054 (2020)

    Google Scholar 

  23. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384–392 (1993)

    Article  Google Scholar 

  24. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124. American Psychological Association, Washington (1971)

    Google Scholar 

  25. Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, 2. [pr.] edn. Prentice-Hall, Englewood Cliffs (1975). OCLC: 247971765

    Google Scholar 

  26. Fontaine, J.R., Scherer, K.R., Roesch, E.B., Ellsworth, P.C.: The world of emotions is not two-dimensional. Psychol. Sci. 18(12), 1050–1057 (2007)

    Article  Google Scholar 

  27. Friesen, W., Ekman, P.: EMFACS-7: Emotional Facial Action Coding System. Unpublished manuscript, vol. 2, p. 1. University of California at San Francisco (1983)

    Google Scholar 

  28. Grandjean, D., Sander, D., Scherer, K.R.: Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Conscious. Cogn. 17(2), 484–495 (2008)

    Article  Google Scholar 

  29. Gwet, K.L.: Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters, 4th edn. Advances Analytics, LLC, Gaithersburg (2014)

    Google Scholar 

  30. Izard, C.E.: Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspect. Psychol. Sci. 2(3), 260–280 (2007)

    Article  Google Scholar 

  31. Jack, R.E., Caldara, R., Schyns, P.G.: Internal representations reveal cultural diversity in expectations of facial expressions of emotion. J. Exp. Psychol. General 141(1), 19–25 (2012)

    Article  Google Scholar 

  32. Jackson, J.C., Watts, J., Henry, T.R., List, J.M., Forkel, R., Mucha, P.J., Greenhill, S.J., Gray, R.D., Lindquist, K.A.: Emotion semantics show both cultural variation and universal structure. Science 366(6472), 1517–1522 (2019)

    Article  Google Scholar 

  33. Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., Liu, J.: DFEW: A large-scale database for recognizing dynamic facial expressions in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2881–2889 (2020)

    Google Scholar 

  34. John, O.P., Robins, R.W., Pervin, L.A.: Handbook of Personality: Theory and Research. Guilford Press, New York (2010)

    Google Scholar 

  35. Jones, B.E.: Arousal systems. Front. Biosci. 8(6), s438–s451 (2003)

    Article  Google Scholar 

  36. Keltner, D., Sauter, D., Tracy, J., Cowen, A.: Emotional expression: advances in basic emotion theory. J. Nonverbal Behav. 43(2), 133–160 (2019)

    Article  Google Scholar 

  37. Kollias, D., Tzirakis, P., Nicolaou, M.A., Papaioannou, A., Zhao, G., Schuller, B.A.W., Kotsia, I., Zafeiriou, S.: Deep affect prediction in-the-wild: Aff-Wild database and challenge, deep architectures, and beyond (2018). CoRR abs/1804.10938. ArXiv: 1804.10938

    Google Scholar 

  38. Kort, B., Reilly, R., Picard, R.: An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. In: Proceedings IEEE International Conference on Advanced Learning Technologies, pp. 43–46 (2001)

    Google Scholar 

  39. Kosinski, M.: Theory of Mind May Have Spontaneously Emerged in Large Language Models (2023). ArXiv:2302.02083 [cs]

    Google Scholar 

  40. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: EMOTIC: emotions in context dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2309–2317. IEEE, Honolulu (2017)

    Google Scholar 

  41. Krys, K., Melanie Vauclair, C., Capaldi, C.A., Lun, V.M.C., Bond, M.H., Domínguez-Espinosa, A., Torres, C., Lipp, O.V., Manickam, L.S.S., Xing, C., et al.: Be careful where you smile: culture shapes judgments of intelligence and honesty of smiling individuals. J. Nonverbal Behav. 40, 101–116 (2016)

    Article  Google Scholar 

  42. Lazarus, R.S.: Cognition and motivation in emotion. Am. Psychol. 46(4), 352. American Psychological Association, Washington (1991)

    Google Scholar 

  43. Lazarus, R.S.: Psychological Stress and the Coping Process. McGraw-Hill, New York (1966)

    Google Scholar 

  44. Lazzeri, N., Mazzei, D., Cominelli, L., Cisternino, A., De Rossi, D.: Designing the mind of a social robot. Appl. Sci. 8(2), 302 (2018)

    Article  Google Scholar 

  45. Le Mau, T., Hoemann, K., Lyons, S.H., Fugate, J.M.B., Brown, E.N., Gendron, M., Barrett, L.F.: Professional actors demonstrate variability, not stereotypical expressions, when portraying emotional states in photographs. Nat. Commun. 12(1), 5037 (2021)

    Article  Google Scholar 

  46. LeDoux, J.E.: Emotion circuits in the brain. Ann. Rev. Neurosci. 23(1), 155–184 (2000)

    Article  Google Scholar 

  47. LeDoux, J.E.: Chapter 21 - evolution of human emotion: a view through fear. In: M.A. Hofman, D. Falk (eds.) Progress in Brain Research, Evolution of the Primate Brain, vol. 195, pp. 431–442. Elsevier, Amsterdam (2012)

    Chapter  Google Scholar 

  48. Lindquist, K.A., Gendron, M.: What’s in a word? language constructs emotion perception. Emot. Rev. 5(1), 66–71 (2013)

    Article  Google Scholar 

  49. Livingstone, S.R., Russo, F.A.: The ryerson audio-visual database of emotional speech and song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in north american english. PLoS One 13(5), e0196391 (2018)

    Article  Google Scholar 

  50. Lomas, T.: Towards a cross-cultural lexical map of wellbeing. J. Posit. Psychol. 1–18 (2020)

    Google Scholar 

  51. Luo, Y., Ye, J., Adams, R.B., Jr., Li, J., Newman, M.G., Wang, J.Z.: ARBEE: towards automated recognition of bodily expression of emotion in the wild. Int. J. Comput. Vision 128(1), 1–25 (2020)

    Article  Google Scholar 

  52. MacCormack, J.K., Lindquist, K.A.: Feeling hangry? When hunger is conceptualized as emotion. Emotion 19(2), 301–319 (2019)

    Google Scholar 

  53. MacDonald, K., Patch, E.A., Figueredo, A.J.A.: Love, trust, and evolution: nurturance/love and trust as two independent attachment systems underlying intimate relationships. Psychology 07(02), 238–253 (2016)

    Article  Google Scholar 

  54. Malatesta, C.Z., Haviland, J.M.: Learning display rules: the socialization of emotion expression in infancy. Child Dev. 53(4), 991 (1982)

    Article  Google Scholar 

  55. Marinier, R.P., Laird, J.E., Lewis, R.L.: A computational unification of cognitive behavior and emotion. Cogn. Syst. Res. 10(1), 48–69 (2009)

    Article  Google Scholar 

  56. Maslow, A.H.: A Dynamic Theory of Human Motivation, pp. 26–47. Howard Allen Publishers, Cleveland (1958)

    Google Scholar 

  57. McGlinchey, E.L., Talbot, L.S., Chang, K.h., Kaplan, K.A., Dahl, R.E., Harvey, A.G.: The effect of sleep deprivation on vocal expression of emotion in adolescents and adults. Sleep 34(9), 1233–1241 (2011)

    Google Scholar 

  58. McKeown, G., Valstar, M.F., Cowie, R., Pantic, M.: The SEMAINE corpus of emotionally coloured character interactions. In: 2010 IEEE International Conference on Multimedia and Expo, pp. 1079–1084 (2010)

    Google Scholar 

  59. Mehrabian, A.: Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in Temperament. Curr. Psychol. 14(4), 261–292 (1996)

    Article  MathSciNet  Google Scholar 

  60. Menninghaus, W., Wagner, V., Wassiliwizky, E., Schindler, I., Hanich, J., Jacobsen, T., Koelsch, S.: What are aesthetic emotions? Psychol. Rev. 126(2), 171–195 (2019)

    Article  Google Scholar 

  61. Mesquita, B.: Emotions in collectivist and individualist contexts. J. Pers. Soc. Psychol. 80(1), 68–74 (2001)

    Article  Google Scholar 

  62. Meuleman, B., Rudrauf, D.: Induction and profiling of strong multi-componential emotions in virtual reality. IEEE Trans. Affect. Comput. 12(1), 189–202 (2021)

    Article  Google Scholar 

  63. Mohammadi, G., Vuilleumier, P.: A multi-componential approach to emotion recognition and the effect of personality. IEEE Trans. Affect. Comput. 1–1 (2020)

    Google Scholar 

  64. Moors, A., Ellsworth, P.C., Scherer, K.R., Frijda, N.H.: Appraisal theories of emotion: state of the art and future development. Emot. Rev. 5, 119–124 (2013)

    Article  Google Scholar 

  65. Nojavanasghari, B., Baltrušaitis, T., Hughes, C.E., Morency, L.P.: EmoReact: a multimodal approach and dataset for recognizing emotional responses in children. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 137–144 (2016)

    Google Scholar 

  66. Panksepp, J.: Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford University Press, Oxford (2004)

    Google Scholar 

  67. Panksepp, J., Biven, L.: The Archaeology of Mind: Neuroevolutionary Origins of Human Emotions. A Norton Professional Book, 1st edn. W. W Norton, New York (2012)

    Google Scholar 

  68. Parkinson, C., Walker, T.T., Memmi, S., Wheatley, T.: Emotions are understood from biological motion across remote cultures. Emotion 17(3), 459–477 (2017)

    Article  Google Scholar 

  69. Paul, E., Wallace, F.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  70. Phan, K., Wager, T., Taylor, S.F., Liberzon, I.: Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. NeuroImage 16(2), 331–348 (2002)

    Article  Google Scholar 

  71. Plutchik, R.: The Emotions, revised edn. University Press of America, Lanham (1991)

    Google Scholar 

  72. Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(03) (2005)

    Google Scholar 

  73. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)

    Google Scholar 

  74. Reis, H.T., Wilson, I.M., Monestere, C., Bernstein, S., Clark, K., Seidl, E., Franco, M., Gioioso, E., Freeman, L., Radoane, K.: What is smiling is beautiful and good. Eur. J. Soc. Psychol. 20(3), 259–267 (1990)

    Article  Google Scholar 

  75. Reisenzein, R., Hildebrandt, A., Weber, H.: Personality and emotion. In: G. Matthews, P.J. Corr (eds.) The Cambridge Handbook of Personality Psychology, Cambridge Handbooks in Psychology, 2 edn., pp. 81–100. Cambridge University Press, Cambridge (2020)

    Google Scholar 

  76. Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)

    Article  Google Scholar 

  77. Russell, J.A., Bullock, M.: Multidimensional scaling of emotional facial expressions: similarity from preschoolers to adults. J. Pers. Soc. Psychol. 48(5), 1290–1298 (1985)

    Article  Google Scholar 

  78. Sackeim, H.A.: Hemispheric asymmetry in the expression of positive and negative emotions: neurologic evidence. Arch. Neurol. 39(4), 210 (1982)

    Article  Google Scholar 

  79. Saldien, J., Goris, K., Vanderborght, B., Vanderfaeillie, J., Lefeber, D.: Expressing emotions with the social robot probo. Int. J. Soc. Robot. 2(4), 377–389 (2010)

    Article  Google Scholar 

  80. Sander, D., Grandjean, D., Scherer, K.R.: A systems approach to appraisal mechanisms in emotion. Neural Netw. 18(4), 317–352 (2005)

    Article  Google Scholar 

  81. Scherer, K.R.: Emotions as episodes of subsystem synchronization driven by nonlinear appraisal processes. In: M.D. Lewis, I. Granic (eds.) Emotion, Development, and Self-Organization, 1st edn., pp. 70–99. Cambridge University Press, Cambridge (2000)

    Chapter  Google Scholar 

  82. Scherer, K.R.: What are emotions? And how can they be measured? Soc. Sci. Inf. 44(4), 695–729 (2005)

    Article  Google Scholar 

  83. Scherer, K.R., Fontaine, J.R.J.: The semantic structure of emotion words across languages is consistent with componential appraisal models of emotion. Cogn. Emot. 33(4), 673–682 (2019)

    Article  Google Scholar 

  84. Scherer, K., Zentner, M.: Music evoked emotions are different–more often aesthetic than utilitarian. Behav. Brain Sci. 31(5), 595–596 (2008)

    Article  Google Scholar 

  85. Scherer, K.R., Schorr, A., Johnstone, T. (eds.): Appraisal Processes in Emotion: Theory, Methods, Research. Series in Affective Science. Oxford University Press, Oxford/New York (2001)

    Google Scholar 

  86. Smith, H., Schneider, A.: Critiquing models of emotions. Sociol. Methods Res. 37(4), 560–589 (2009)

    Article  MathSciNet  Google Scholar 

  87. Stock-Homburg, R.: Survey of emotions in human-robot interactions: perspectives from robotic psychology on 20 years of research. Int. J. Soc. Robot. 14(2), 389–411 (2022)

    Article  Google Scholar 

  88. Sznycer, D., Tooby, J., Cosmides, L., Porat, R., Shalvi, S., Halperin, E.: Shame closely tracks the threat of devaluation by others, even across cultures. Proc. Natl. Acad. Sci. 113(10), 2625–2630 (2016)

    Article  Google Scholar 

  89. Verma, D., Wood, J., Lach, G., Herzog, H., Sperk, G., Tasan, R.: Hunger promotes fear extinction by activation of an amygdala microcircuit. Neuropsychopharmacology 41(2), 431–439 (2016)

    Article  Google Scholar 

  90. Watson, D., Wiese, D., Vaidya, J., Tellegen, A.: The two general activation systems of affect: structural findings, evolutionary considerations, and psychobiological evidence. J. Pers. Soc. Psychol. 820–838 (1999)

    Google Scholar 

  91. Wortman, B., Wang, J.Z.: HICEM: a high-coverage emotion model for artificial emotional intelligence. In: IEEE Transactions on Affective Computing (2022). https://doi.org/10.1109/TAFFC.2023.3324902

Download references

Acknowledgements

The work was funded in part by a generous gift from Amazon to the author’s dissertation advisor Professor James Z. Wang. The author also acknowledges the advice and constructive comments from James Wang, Reginald Adams, Jr., and Tal Shafir.

Author information

Authors and Affiliations

  1. College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA, USA

    Benjamin Wortman

Authors
  1. Benjamin Wortman
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Information Sciences and Technology, The Pennsylvania State University, University Park, PA, USA

    James Z. Wang

  2. Department of Psychology, The Pennsylvania State University, University Park, PA, USA

    Reginald B. Adams, Jr.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Wortman, B. (2024). Models of Human Emotion and Artificial Emotional Intelligence. In: Wang, J.Z., Adams, Jr., R.B. (eds) Modeling Visual Aesthetics, Emotion, and Artistic Style. Springer, Cham. https://doi.org/10.1007/978-3-031-50269-9_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50269-9_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50268-2

  • Online ISBN: 978-3-031-50269-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation