-
Exploring Real-Time Music-to-Image Systems for Creative Inspiration in Music Creation
Authors:
Meng Yang,
Maria Teresa Llano,
Jon McCormack
Abstract:
This paper presents a study on the use of a real-time music-to-image system as a mechanism to support and inspire musicians during their creative process. The system takes MIDI messages from a keyboard as input which are then interpreted and analysed using state-of-the-art generative AI models. Based on the perceived emotion and music structure, the system's interpretation is converted into visual…
▽ More
This paper presents a study on the use of a real-time music-to-image system as a mechanism to support and inspire musicians during their creative process. The system takes MIDI messages from a keyboard as input which are then interpreted and analysed using state-of-the-art generative AI models. Based on the perceived emotion and music structure, the system's interpretation is converted into visual imagery that is presented in real-time to musicians. We conducted a user study in which musicians improvised and composed using the system. Our findings show that most musicians found the generated images were a novel mechanism when playing, evidencing the potential of music-to-image systems to inspire and enhance their creative process.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
Visual instrument co-design embracing the unique movement capabilities of a dancer with physical disability
Authors:
Sam Trolland,
Melinda Smith,
Alon Ilsar,
Jon McCormack
Abstract:
This paper explores the design of an expressive visual instrument that embraces the unique movement style of a dancer living with physical disability. Through a collaboration between the dancer and an interaction designer/visual artist, the creative qualities of wearable devices for motion tracking are investigated, with emphasis on integrating the dancer's specific movement capabilities with thei…
▽ More
This paper explores the design of an expressive visual instrument that embraces the unique movement style of a dancer living with physical disability. Through a collaboration between the dancer and an interaction designer/visual artist, the creative qualities of wearable devices for motion tracking are investigated, with emphasis on integrating the dancer's specific movement capabilities with their creative goals. The affordances of this technology for imagining new forms of creative expression play a critical role in the design process. These themes are drawn together through an experiential performance which augments an improvised dance with an ephemeral real-time visualisation of the performer's movements. Through practice-based research, the design, development and presentation of this performance work is examined as a 'testbed' for new ideas, allowing for the exploration of HCI concepts within a creative context. This paper outlines the creative process behind the development of the work, the insights derived from the practice-based research enquiry, and the role of movement technology in encouraging new ways of moving through creative expression.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Transhuman Ansambl - Voice Beyond Language
Authors:
Lucija Ivsic,
Jon McCormack,
Vince Dziekan
Abstract:
In this paper we present the design and development of the Transhuman Ansambl, a novel interactive singing-voice interface which senses its environment and responds to vocal input with vocalisations using human voice. Designed for live performance with a human performer and as a standalone sound installation, the ansambl consists of sixteen bespoke virtual singers arranged in a circle. When perfor…
▽ More
In this paper we present the design and development of the Transhuman Ansambl, a novel interactive singing-voice interface which senses its environment and responds to vocal input with vocalisations using human voice. Designed for live performance with a human performer and as a standalone sound installation, the ansambl consists of sixteen bespoke virtual singers arranged in a circle. When performing live, the virtual singers listen to the human performer and respond to their singing by reading pitch, intonation and volume cues. In a standalone sound installation mode, singers use ultrasonic distance sensors to sense audience presence. Developed as part of the 1st author's practice-based PhD and artistic practice as a live performer, this work employs the singing-voice to explore voice interactions in HCI beyond language, and innovative ways of live performing. How is technology supporting the effect of intimacy produced through voice? Does the act of surrounding the audience with responsive virtual singers challenge the traditional roles of performer-listener? To answer these questions, we draw upon the 1st author's experience with the system, and the interdisciplinary field of voice studies that consider the voice as the sound medium independent of language, capable of enacting a reciprocal connection between bodies.
△ Less
Submitted 5 May, 2024;
originally announced May 2024.
-
Holon: a cybernetic interface for bio-semiotics
Authors:
Jon McCormack,
Elliott Wilson
Abstract:
This paper presents an interactive artwork, "Holon", a collection of 130 autonomous, cybernetic organisms that listen and make sound in collaboration with the natural environment. The work was developed for installation on water at a heritage-listed dock in Melbourne, Australia. Conceptual issues informing the work are presented, along with a detailed technical overview of the implementation. Indi…
▽ More
This paper presents an interactive artwork, "Holon", a collection of 130 autonomous, cybernetic organisms that listen and make sound in collaboration with the natural environment. The work was developed for installation on water at a heritage-listed dock in Melbourne, Australia. Conceptual issues informing the work are presented, along with a detailed technical overview of the implementation. Individual holons are of three types, inspired by biological models of animal communication: composer/generators, collector/critics and disruptors. Collectively, Holon integrates and occupies elements of the acoustic spectrum in collaboration with human and non-human agents.
△ Less
Submitted 5 April, 2024;
originally announced April 2024.
-
Towards A Diffractive Analysis of Prompt-Based Generative AI
Authors:
Nina Rajcic,
Maria Teresa Llano,
Jon McCormack
Abstract:
Recent developments in prompt-based generative AI has given rise to discourse surrounding the perceived ethical concerns, economic implications, and consequences for the future of cultural production. As generative imagery becomes pervasive in mainstream society, dominated primarily by emerging industry leaders, we encourage that the role of the CHI community be one of inquiry; to investigate the…
▽ More
Recent developments in prompt-based generative AI has given rise to discourse surrounding the perceived ethical concerns, economic implications, and consequences for the future of cultural production. As generative imagery becomes pervasive in mainstream society, dominated primarily by emerging industry leaders, we encourage that the role of the CHI community be one of inquiry; to investigate the numerous ways in which generative AI has the potential to, and already is, augmenting human creativity. In this paper, we conducted a diffractive analysis exploring the potential role of prompt-based interfaces in artists' creative practice. Over a two week period, seven visual artists were given access to a personalised instance of Stable Diffusion, fine-tuned on a dataset of their work. In the following diffractive analysis, we identified two dominant modes adopted by participants, AI for ideation, and AI for production. We furthermore present a number of ethical design considerations for the future development of generative AI interfaces.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
No Longer Trending on Artstation: Prompt Analysis of Generative AI Art
Authors:
Jon McCormack,
Maria Teresa Llano,
Stephen James Krol,
Nina Rajcic
Abstract:
Image generation using generative AI is rapidly becoming a major new source of visual media, with billions of AI generated images created using diffusion models such as Stable Diffusion and Midjourney over the last few years. In this paper we collect and analyse over 3 million prompts and the images they generate. Using natural language processing, topic analysis and visualisation methods we aim t…
▽ More
Image generation using generative AI is rapidly becoming a major new source of visual media, with billions of AI generated images created using diffusion models such as Stable Diffusion and Midjourney over the last few years. In this paper we collect and analyse over 3 million prompts and the images they generate. Using natural language processing, topic analysis and visualisation methods we aim to understand collectively how people are using text prompts, the impact of these systems on artists, and more broadly on the visual cultures they promote. Our study shows that prompting focuses largely on surface aesthetics, reinforcing cultural norms, popular conventional representations and imagery. We also find that many users focus on popular topics (such as making colouring books, fantasy art, or Christmas cards), suggesting that the dominant use for the systems analysed is recreational rather than artistic.
△ Less
Submitted 24 January, 2024;
originally announced January 2024.
-
Creative Discovery using QD Search
Authors:
Jon McCormack,
Camilo Cruz Gambardella,
Stephen James Krol
Abstract:
In creative design, where aesthetics play a crucial role in determining the quality of outcomes, there are often multiple worthwhile possibilities, rather than a single ``best'' design. This challenge is compounded in the use of computational generative systems, where the sheer number of potential outcomes can be overwhelming. This paper introduces a method that combines evolutionary optimisation…
▽ More
In creative design, where aesthetics play a crucial role in determining the quality of outcomes, there are often multiple worthwhile possibilities, rather than a single ``best'' design. This challenge is compounded in the use of computational generative systems, where the sheer number of potential outcomes can be overwhelming. This paper introduces a method that combines evolutionary optimisation with AI-based image classification to perform quality-diversity search, allowing for the creative exploration of complex design spaces. The process begins by randomly sampling the genotype space, followed by mapping the generated phenotypes to a reduced representation of the solution space, as well as evaluating them based on their visual characteristics. This results in an elite group of diverse outcomes that span the solution space. The elite is then progressively updated via sampling and simple mutation. We tested our method on a generative system that produces abstract drawings. The results demonstrate that the system can effectively evolve populations of phenotypes with high aesthetic value and greater visual diversity compared to traditional optimisation-focused evolutionary approaches.
△ Less
Submitted 8 May, 2023;
originally announced May 2023.
-
Is Writing Prompts Really Making Art?
Authors:
Jon McCormack,
Camilo Cruz Gambardella,
Nina Rajcic,
Stephen James Krol,
Maria Teresa Llano,
Meng Yang
Abstract:
In recent years Generative Machine Learning systems have advanced significantly. A current wave of generative systems use text prompts to create complex imagery, video, even 3D datasets. The creators of these systems claim a revolution in bringing creativity and art to anyone who can type a prompt. In this position paper, we question the basis for these claims, dividing our analysis into three are…
▽ More
In recent years Generative Machine Learning systems have advanced significantly. A current wave of generative systems use text prompts to create complex imagery, video, even 3D datasets. The creators of these systems claim a revolution in bringing creativity and art to anyone who can type a prompt. In this position paper, we question the basis for these claims, dividing our analysis into three areas: the limitations of linguistic descriptions, implications of the dataset, and lastly, matters of materiality and embodiment. We conclude with an analysis of the creative possibilities enabled by prompt-based systems, asking if they can be considered a new artistic medium.
△ Less
Submitted 1 February, 2023; v1 submitted 25 January, 2023;
originally announced January 2023.
-
Message Ritual: A Posthuman Account of Living with Lamp
Authors:
Nina Rajcic,
Jon McCormack
Abstract:
As we become increasingly entangled with digital technologies, the boundary between human and machine is progressively blurring. Adopting a performative, posthumanist perspective resolves this ambiguity by proposing that such boundaries are not predetermined, rather they are enacted within a certain material configuration. Using this approach, dubbed `Entanglement HCI', this paper presents \emph{M…
▽ More
As we become increasingly entangled with digital technologies, the boundary between human and machine is progressively blurring. Adopting a performative, posthumanist perspective resolves this ambiguity by proposing that such boundaries are not predetermined, rather they are enacted within a certain material configuration. Using this approach, dubbed `Entanglement HCI', this paper presents \emph{Message Ritual} -- a novel, integrated AI system that encourages the re-framing of memory through machine generated poetics. Embodied within a domestic table lamp, the system listens in on conversations occurring within the home, drawing out key topics and phrases of the day and reconstituting them through machine generated poetry, delivered to household members via SMS upon waking each morning. Participants across four households were asked to live with the lamp over a two week period. We present a diffractive analysis exploring how the lamp \emph{becomes with} participants and discuss the implications of this method for future HCI research.
△ Less
Submitted 26 January, 2023;
originally announced January 2023.
-
What is in a Name: Defining -High Entropy- Oxides
Authors:
Matthew Brahlek,
Maria Gazda,
Veerle Keppens,
Alessandro R. Mazza,
Scott J. McCormack,
Aleksandra Mielewczyk-Gryń,
Brianna Musico,
Katharine Page,
Christina Rost,
Susan B. Sinnott,
Cormac Toher,
Thomas Z. Ward,
Ayako Yamamoto
Abstract:
High entropy oxides are emerging as an exciting new avenue to design highly tailored functional behaviors that have no traditional counterparts. Study and application of these materials are bringing together scientists and engineers from physics, chemistry, and materials science. The diversity of each of these disciplines comes with perspectives and jargon that may be confusing to those outside of…
▽ More
High entropy oxides are emerging as an exciting new avenue to design highly tailored functional behaviors that have no traditional counterparts. Study and application of these materials are bringing together scientists and engineers from physics, chemistry, and materials science. The diversity of each of these disciplines comes with perspectives and jargon that may be confusing to those outside of the individual fields, which can result in miscommunication of important aspects of research. In this perspective, we provide examples of research and characterization taken from these different fields to provide a framework for classifying the differences between compositionally complex oxides, high entropy oxides, and entropy stabilized oxides, which is intended to bring a common language to this emerging area. We highlight the critical importance of understanding a materials crystallinity, composition, and mixing length scales in determining its true definition.
△ Less
Submitted 4 November, 2022; v1 submitted 26 August, 2022;
originally announced August 2022.
-
Towards the Generation of Musical Explanations with GPT-3
Authors:
Stephen James Krol,
Maria Teresa Llano,
Jon McCormack
Abstract:
Open AI's language model, GPT-3, has shown great potential for many NLP tasks, with applications in many different domains. In this work we carry out a first study on GPT-3's capability to communicate musical decisions through textual explanations when prompted with a textual representation of a piece of music. Enabling a dialogue in human-AI music partnerships is an important step towards more en…
▽ More
Open AI's language model, GPT-3, has shown great potential for many NLP tasks, with applications in many different domains. In this work we carry out a first study on GPT-3's capability to communicate musical decisions through textual explanations when prompted with a textual representation of a piece of music. Enabling a dialogue in human-AI music partnerships is an important step towards more engaging and creative human-AI interactions. Our results show that GPT-3 lacks the necessary intelligence to really understand musical decisions. A major barrier to reach a better performance is the lack of data that includes explanations of the creative process carried out by artists for musical pieces. We believe such a resource would aid the understanding and collaboration with AI music systems.
△ Less
Submitted 11 May, 2022;
originally announced June 2022.
-
Explainable Computational Creativity
Authors:
Maria Teresa Llano,
Mark d'Inverno,
Matthew Yee-King,
Jon McCormack,
Alon Ilsar,
Alison Pease,
Simon Colton
Abstract:
Human collaboration with systems within the Computational Creativity (CC) field is often restricted to shallow interactions, where the creative processes, of systems and humans alike, are carried out in isolation, without any (or little) intervention from the user, and without any discussion about how the unfolding decisions are taking place. Fruitful co-creation requires a sustained ongoing inter…
▽ More
Human collaboration with systems within the Computational Creativity (CC) field is often restricted to shallow interactions, where the creative processes, of systems and humans alike, are carried out in isolation, without any (or little) intervention from the user, and without any discussion about how the unfolding decisions are taking place. Fruitful co-creation requires a sustained ongoing interaction that can include discussions of ideas, comparisons to previous/other works, incremental improvements and revisions, etc. For these interactions, communication is an intrinsic factor. This means giving a voice to CC systems and enabling two-way communication channels between them and their users so that they can: explain their processes and decisions, support their ideas so that these are given serious consideration by their creative collaborators, and learn from these discussions to further improve their creative processes. For this, we propose a set of design principles for CC systems that aim at supporting greater co-creation and collaboration with their human collaborators.
△ Less
Submitted 11 May, 2022;
originally announced May 2022.
-
Quality-diversity for aesthetic evolution
Authors:
Jon McCormack,
Camilo Cruz Gambardella
Abstract:
Many creative generative design spaces contain multiple regions with individuals of high aesthetic value. Yet traditional evolutionary computing methods typically focus on optimisation, searching for the fittest individual in a population. In this paper we apply quality-diversity search methods to explore a creative generative system (an agent-based line drawing model). We perform a random samplin…
▽ More
Many creative generative design spaces contain multiple regions with individuals of high aesthetic value. Yet traditional evolutionary computing methods typically focus on optimisation, searching for the fittest individual in a population. In this paper we apply quality-diversity search methods to explore a creative generative system (an agent-based line drawing model). We perform a random sampling of genotype space and use individual artist-assigned evaluations of aesthetic quality to formulate a computable fitness measure specific to the artist and this system. To compute diversity we use a convolutional neural network to discriminate features that are dimensionally reduced into two dimensions. We show that the quality-diversity search is able to find multiple phenotypes of high aesthetic value. These phenotypes show greater diversity and quality than those the artist was able to find using manual search methods.
△ Less
Submitted 3 February, 2022;
originally announced February 2022.
-
Complexity and Aesthetics in Generative and Evolutionary Art
Authors:
Jon McCormack,
Camilo Cruz Gambardella
Abstract:
In this paper we examine the concept of complexity as it applies to generative and evolutionary art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different evolutionary art data…
▽ More
In this paper we examine the concept of complexity as it applies to generative and evolutionary art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different evolutionary art datasets and look at the correlations between complexity and individual aesthetic judgement by the artist (in the case of two datasets) or the physically measured complexity of generative 3D forms. Our results show that the degree of correlation is different for each set and measure, indicating that there is no overall "better" measure. However, specific measures do perform well on individual datasets, indicating that careful choice can increase the value of using such measures. We then assess the value of complexity measures for the audience by undertaking a large-scale survey on the perception of complexity and aesthetics. We conclude by discussing the value of direct measures in generative and evolutionary art, reinforcing recent findings from neuroimaging and psychology which suggest human aesthetic judgement is informed by many extrinsic factors beyond the measurable properties of the object being judged.
△ Less
Submitted 5 January, 2022;
originally announced January 2022.
-
Growing and Evolving 3D Prints
Authors:
Jon McCormack,
Camilo Cruz Gambardella
Abstract:
Design - especially of physical objects - can be understood as creative acts solving practical problems. In this paper we describe a biologically-inspired developmental model as the basis of a generative form-finding system. Using local interactions between cells in a two-dimensional environment, then capturing the state of the system at every time step, complex three-dimensional (3D) forms can be…
▽ More
Design - especially of physical objects - can be understood as creative acts solving practical problems. In this paper we describe a biologically-inspired developmental model as the basis of a generative form-finding system. Using local interactions between cells in a two-dimensional environment, then capturing the state of the system at every time step, complex three-dimensional (3D) forms can be generated by the system. Unlike previous systems, our method is capable of directly producing 3D printable objects, eliminating intermediate transformations and manual manipulation often necessary to ensure the 3D form is printable. We devise fitness measures for optimising 3D printability and aesthetic complexity and use a Covariance Matrix Adaptation Evolutionary Strategies algorithm (CMA-ES) to find 3D forms that are both aesthetically interesting and physically printable using fused deposition modelling printing techniques. We investigate the system's capabilities by evolving and 3D printing objects at different levels of structural consistency, and assess the quality of the fitness measures presented to explore the design space of our generative system. We find that by evolving first for aesthetic complexity, then evolving for structural consistency until the form is 'just printable', gives the best results.
△ Less
Submitted 6 July, 2021;
originally announced July 2021.
-
Searching for Designs in-between
Authors:
Camilo Cruz Gambardella,
Jon McCormack
Abstract:
The use of evolutionary methods in design and art is increasing in diversity and popularity. Approaches to using these methods for creative production typically focus either on optimisation or exploration. In this paper we introduce an evolutionary system for design that combines these two approaches, enabling users to explore landscapes of design alternatives using design-oriented measures of fit…
▽ More
The use of evolutionary methods in design and art is increasing in diversity and popularity. Approaches to using these methods for creative production typically focus either on optimisation or exploration. In this paper we introduce an evolutionary system for design that combines these two approaches, enabling users to explore landscapes of design alternatives using design-oriented measures of fitness, along with their own aesthetic preferences. We test our methods using a biologically-inspired generative system capable of producing 3D objects that can be exported directly as 3D printing toolpath instructions. For the search stage of our system we combine the use of the CMA-ES algorithm for optimisation and linear interpolation between generated objects for feature exploration. We investigate the system`s capabilities by evolving highly fit artefacts and then combining them with aesthetically interesting ones.
△ Less
Submitted 11 February, 2021;
originally announced February 2021.
-
The Enigma of Complexity
Authors:
Jon McCormack,
Camilo Cruz Gambardella,
Andy Lomas
Abstract:
In this paper we examine the concept of complexity as it applies to generative art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different generative art datasets and look at th…
▽ More
In this paper we examine the concept of complexity as it applies to generative art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different generative art datasets and look at the correlations between complexity and individual aesthetic judgement by the artist (in the case of two datasets) or the physically measured complexity of 3D forms. Our results show that the degree of correlation is different for each set and measure, indicating that there is no overall "better" measure. However, specific measures do perform well on individual datasets, indicating that careful choice can increase the value of using such measures. We conclude by discussing the value of direct measures in generative and evolutionary art, reinforcing recent findings from neuroimaging and psychology which suggest human aesthetic judgement is informed by many extrinsic factors beyond the measurable properties of the object being judged.
△ Less
Submitted 3 February, 2021;
originally announced February 2021.
-
Deep Learning of Individual Aesthetics
Authors:
Jon McCormack,
Andy Lomas
Abstract:
Accurate evaluation of human aesthetic preferences represents a major challenge for creative evolutionary and generative systems research. Prior work has tended to focus on feature measures of the artefact, such as symmetry, complexity and coherence. However, research models from Psychology suggest that human aesthetic experiences encapsulate factors beyond the artefact, making accurate computatio…
▽ More
Accurate evaluation of human aesthetic preferences represents a major challenge for creative evolutionary and generative systems research. Prior work has tended to focus on feature measures of the artefact, such as symmetry, complexity and coherence. However, research models from Psychology suggest that human aesthetic experiences encapsulate factors beyond the artefact, making accurate computational models very difficult to design. The interactive genetic algorithm (IGA) circumvents the problem through human-in-the-loop, subjective evaluation of aesthetics, but is limited due to user fatigue and small population sizes. In this paper we look at how recent advances in deep learning can assist in automating personal aesthetic judgement. Using a leading artist's computer art dataset, we investigate the relationship between image measures, such as complexity, and human aesthetic evaluation. We use dimension reduction methods to visualise both genotype and phenotype space in order to support the exploration of new territory in a generative system. Convolutional Neural Networks trained on the artist's prior aesthetic evaluations are used to suggest new possibilities similar or between known high quality genotype-phenotype mappings. We integrate this classification and discovery system into a software tool for evolving complex generative art and design.
△ Less
Submitted 23 September, 2020;
originally announced September 2020.
-
High precision meteor observations with the Canadian Automated Meteor Observatory -- Data reduction pipeline and application to meteoroid mechanical strength measurements
Authors:
Denis Vida,
Peter G. Brown,
Margaret Campbell-Brown,
Robert J. Weryk,
Gunter Stober,
John P. McCormack
Abstract:
Context. The mirror tracking system of the Canadian Automated Meteor Observatory (CAMO) can track meteors in real time, providing an effective angular resolution of 1 arc second and a temporal resolution of 100 frames per second.
Aims. We describe the upgraded hardware and give details of the data calibration and reduction pipeline. We investigate the influence of meteor morphology on radiant an…
▽ More
Context. The mirror tracking system of the Canadian Automated Meteor Observatory (CAMO) can track meteors in real time, providing an effective angular resolution of 1 arc second and a temporal resolution of 100 frames per second.
Aims. We describe the upgraded hardware and give details of the data calibration and reduction pipeline. We investigate the influence of meteor morphology on radiant and velocity measurement precision, and use direct observations of meteoroid fragmentation to constrain their compressive strengths.
Methods. On July 21, 2017, CAMO observed a ~4 second meteor on a JFC orbit. It had a shallow entry angle ~8 deg and 12 fragments were visible in the narrow-field video. The event was manually reduced and the exact moment of fragmentation was determined. The aerodynamic ram pressure at the moment of fragmentation was used as a proxy for compressive strength, and strengths of an additional 19 fragmenting meteoroids were measured in the same way. The uncertainty in the atmosphere mass density was estimated to be +/-25% using NAVGEM-HA data.
Results. We find that meteor trajectory accuracy significantly depends on meteor morphology. The CAMO radiant and initial velocity precision for non-fragmenting meteors with short wakes is ~0.5' and 1 m/s, while that for meteors with fragments or long wakes is similar to non-tracking, moderate field of view optical systems (5', ~50 m/s). Measured compressive strengths of 20 fragmenting meteoroids (with less precise radiants due to their morphology) was in the range of 1-4 kPa, which is in excellent accord with Rosetta in-situ measurements of 67P. Fragmentation type and strength do not appear to be dependent on orbit. The mass index of the 12 fragments in the July 21 meteoroid was very high (s = 2.8), indicating possible progressive fragmentation.
△ Less
Submitted 2 September, 2020;
originally announced September 2020.
-
A scriptable, generative modelling system for dynamic 3D meshes
Authors:
Jon McCormack,
Ben Porter,
James Wetter
Abstract:
We describe a flexible, script-based system for the procedural generation and animation of 3D geometry. Dynamic triangular meshes are generated through the real-time execution of scripts written in the Lua programming language. Tight integration between the programming environment, runtime engine and graphics visualisation enables a workflow between coding and visual results that encourages experi…
▽ More
We describe a flexible, script-based system for the procedural generation and animation of 3D geometry. Dynamic triangular meshes are generated through the real-time execution of scripts written in the Lua programming language. Tight integration between the programming environment, runtime engine and graphics visualisation enables a workflow between coding and visual results that encourages experimentation and rapid prototyping. The system has been used successfully to generate a variety of complex, dynamic organic forms including complex branching structures, scalable symmetric manifolds and abstract organic forms. We use examples in each of these areas to detail the main features of the system, which include a set of flexible 3D mesh operations integrated with a Lua-based L-system interpreter that creates geometry using generalised cylinders.
△ Less
Submitted 21 April, 2020;
originally announced April 2020.
-
Mirror Ritual: An Affective Interface for Emotional Self-Reflection
Authors:
Nina Rajcic,
Jon McCormack
Abstract:
This paper introduces a new form of real-time affective interface that engages the user in a process of conceptualisation of their emotional state. Inspired by Barrett's Theory of Constructed Emotion, `Mirror Ritual' aims to expand upon the user's accessible emotion concepts, and to ultimately provoke emotional reflection and regulation. The interface uses classified emotions -- obtained through f…
▽ More
This paper introduces a new form of real-time affective interface that engages the user in a process of conceptualisation of their emotional state. Inspired by Barrett's Theory of Constructed Emotion, `Mirror Ritual' aims to expand upon the user's accessible emotion concepts, and to ultimately provoke emotional reflection and regulation. The interface uses classified emotions -- obtained through facial expression recognition -- as a basis for dynamically generating poetry. The perceived emotion is used to seed a poetry generation system based on OpenAI's GPT-2 model, fine-tuned on a specially curated corpus. We evaluate the device's ability to foster a personalised, meaningful experience for individual users over a sustained period. A qualitative analysis revealed that participants were able to affectively engage with the mirror, with each participant developing a unique interpretation of its poetry in the context of their own emotional landscape.
△ Less
Submitted 20 April, 2020;
originally announced April 2020.
-
Mirror Ritual: Human-Machine Co-Construction of Emotion
Authors:
Nina Rajcic,
Jon McCormack
Abstract:
Mirror Ritual is an interactive installation that challenges the existing paradigms in our understanding of human emotion and machine perception. In contrast to prescriptive interfaces, the work's real-time affective interface engages the audience in the iterative conceptualisation of their emotional state through the use of affectively-charged machine generated poetry. The audience are encouraged…
▽ More
Mirror Ritual is an interactive installation that challenges the existing paradigms in our understanding of human emotion and machine perception. In contrast to prescriptive interfaces, the work's real-time affective interface engages the audience in the iterative conceptualisation of their emotional state through the use of affectively-charged machine generated poetry. The audience are encouraged to make sense of the mirror's poetry by framing it with respect to their recent life experiences, effectively `putting into words' their felt emotion. This process of affect labelling and contextualisation works to not only regulate emotion, but helps to construct the rich personal narratives that constitute human identity.
△ Less
Submitted 15 April, 2020;
originally announced April 2020.
-
Understanding Aesthetic Evaluation using Deep Learning
Authors:
Jon McCormack,
Andy Lomas
Abstract:
A bottleneck in any evolutionary art system is aesthetic evaluation. Many different methods have been proposed to automate the evaluation of aesthetics, including measures of symmetry, coherence, complexity, contrast and grouping. The interactive genetic algorithm (IGA) relies on human-in-the-loop, subjective evaluation of aesthetics, but limits possibilities for large search due to user fatigue a…
▽ More
A bottleneck in any evolutionary art system is aesthetic evaluation. Many different methods have been proposed to automate the evaluation of aesthetics, including measures of symmetry, coherence, complexity, contrast and grouping. The interactive genetic algorithm (IGA) relies on human-in-the-loop, subjective evaluation of aesthetics, but limits possibilities for large search due to user fatigue and small population sizes. In this paper we look at how recent advances in deep learning can assist in automating personal aesthetic judgement. Using a leading artist's computer art dataset, we use dimensionality reduction methods to visualise both genotype and phenotype space in order to support the exploration of new territory in any generative system. Convolutional Neural Networks trained on the user's prior aesthetic evaluations are used to suggest new possibilities similar or between known high quality genotype-phenotype mappings.
△ Less
Submitted 15 April, 2020;
originally announced April 2020.
-
Adaptive Music Composition for Games
Authors:
Patrick Hutchings,
Jon McCormack
Abstract:
The generation of music that adapts dynamically to content and actions has an important role in building more immersive, memorable and emotive game experiences. To date, the development of adaptive music systems for video games is limited by both the nature of algorithms used for real-time music generation and the limited modelling of player action, game world context and emotion in current games.…
▽ More
The generation of music that adapts dynamically to content and actions has an important role in building more immersive, memorable and emotive game experiences. To date, the development of adaptive music systems for video games is limited by both the nature of algorithms used for real-time music generation and the limited modelling of player action, game world context and emotion in current games. We propose that these issues must be addressed in tandem for the quality and flexibility of adaptive game music to significantly improve. Cognitive models of knowledge organisation and emotional affect are integrated with multi-modal, multi-agent composition techniques to produce a novel Adaptive Music System (AMS). The system is integrated into two stylistically distinct games. Gamers reported an overall higher immersion and correlation of music with game-world concepts with the AMS than with the original game soundtracks in both games.
△ Less
Submitted 2 July, 2019;
originally announced July 2019.
-
Autonomy, Authenticity, Authorship and Intention in computer generated art
Authors:
Jon McCormack,
Toby Gifford,
Patrick Hutchings
Abstract:
This paper examines five key questions surrounding computer generated art. Driven by the recent public auction of a work of `AI Art' we selectively summarise many decades of research and commentary around topics of autonomy, authenticity, authorship and intention in computer generated art, and use this research to answer contemporary questions often asked about art made by computers that concern t…
▽ More
This paper examines five key questions surrounding computer generated art. Driven by the recent public auction of a work of `AI Art' we selectively summarise many decades of research and commentary around topics of autonomy, authenticity, authorship and intention in computer generated art, and use this research to answer contemporary questions often asked about art made by computers that concern these topics. We additionally reflect on whether current techniques in deep learning and Generative Adversarial Networks significantly change the answers provided by many decades of prior research.
△ Less
Submitted 5 March, 2019;
originally announced March 2019.
-
Camera Obscurer: Generative Art for Design Inspiration
Authors:
Dilpreet Singh,
Nina Rajcic,
Simon Colton,
Jon McCormack
Abstract:
We investigate using generated decorative art as a source of inspiration for design tasks. Using a visual similarity search for image retrieval, the \emph{Camera Obscurer} app enables rapid searching of tens of thousands of generated abstract images of various types. The seed for a visual similarity search is a given image, and the retrieved generated images share some visual similarity with the s…
▽ More
We investigate using generated decorative art as a source of inspiration for design tasks. Using a visual similarity search for image retrieval, the \emph{Camera Obscurer} app enables rapid searching of tens of thousands of generated abstract images of various types. The seed for a visual similarity search is a given image, and the retrieved generated images share some visual similarity with the seed. Implemented in a hand-held device, the app empowers users to use photos of their surroundings to search through the archive of generated images and other image archives. Being abstract in nature, the retrieved images supplement the seed image rather than replace it, providing different visual stimuli including shapes, colours, textures and juxtapositions, in addition to affording their own interpretations. This approach can therefore be used to provide inspiration for a design task, with the abstract images suggesting new ideas that might give direction to a graphic design project. We describe a crowdsourcing experiment with the app to estimate user confidence in retrieved images, and we describe a pilot study where Camera Obscurer provided inspiration for a design task. These experiments have enabled us to describe future improvements, and to begin to understand sources of visual inspiration for design tasks.
△ Less
Submitted 5 March, 2019;
originally announced March 2019.
-
In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound
Authors:
Jon McCormack,
Toby Gifford,
Patrick Hutchings,
Maria Teresa Llano Rodriguez,
Matthew Yee-King,
Mark d'Inverno
Abstract:
Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of th…
▽ More
Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.
△ Less
Submitted 18 February, 2019;
originally announced February 2019.
-
A phylogeny of birds based on over 1,500 loci collected by target enrichment and high-throughput sequencing
Authors:
John E. McCormack,
Michael G. Harvey,
Brant C. Faircloth,
Nicholas G. Crawford,
Travis C. Glenn,
Robb T. Brumfield
Abstract:
Evolutionary relationships among birds in Neoaves, the clade comprising the vast majority of avian diversity, have vexed systematists due to the ancient, rapid radiation of numerous lineages. We applied a new phylogenomic approach to resolve relationships in Neoaves using target enrichment (sequence capture) and high-throughput sequencing of ultraconserved elements (UCEs) in avian genomes. We coll…
▽ More
Evolutionary relationships among birds in Neoaves, the clade comprising the vast majority of avian diversity, have vexed systematists due to the ancient, rapid radiation of numerous lineages. We applied a new phylogenomic approach to resolve relationships in Neoaves using target enrichment (sequence capture) and high-throughput sequencing of ultraconserved elements (UCEs) in avian genomes. We collected sequence data from UCE loci for 32 members of Neoaves and one outgroup (chicken) and analyzed data sets that differed in their amount of missing data. An alignment of 1,541 loci that allowed missing data was 87% complete and resulted in a highly resolved phylogeny with broad agreement between the Bayesian and maximum-likelihood (ML) trees. Although results from the 100% complete matrix of 416 UCE loci were similar, the Bayesian and ML trees differed to a greater extent in this analysis, suggesting that increasing from 416 to 1,541 loci led to increased stability and resolution of the tree. Novel results of our study include surprisingly close relationships between phenotypically divergent bird families, such as tropicbirds (Phaethontidae) and the sunbittern (Eurypygidae) as well as between bustards (Otididae) and turacos (Musophagidae). This phylogeny bolsters support for monophyletic waterbird and landbird clades and also strongly supports controversial results from previous studies, including the sister relationship between passerines and parrots and the non-monophyly of raptorial birds in the hawk and falcon families. Although significant challenges remain to fully resolving some of the deep relationships in Neoaves, especially among lineages outside the waterbirds and landbirds, this study suggests that increased data will yield an increasingly resolved avian phylogeny.
△ Less
Submitted 31 December, 2012; v1 submitted 4 October, 2012;
originally announced October 2012.
