arXiv subjects

Jon McCormack

Publications and source records attributed to Jon McCormack.

33 records · Page 2Linked to original sources

arXiv2022

Towards the Generation of Musical Explanations with GPT-3

Open AI's language model, GPT-3, has shown great potential for many NLP tasks, with applications in many different domains. In this work we carry out a first study on GPT-3's capability to communicate musical decisions through textual explanations when prompted with a textual representation of a piece of music. Enabling a dialogue in human-AI music partnerships is an important step towards more engaging and creative human-AI interactions. Our results show that GPT-3 lacks the necessary intelligence to really understand musical decisions. A major barrier to reach a better performance is the lack of data that includes explanations of the creative process carried out by artists for musical pieces. We believe such a resource would aid the understanding and collaboration with AI music systems.

cs.CL↗

arXiv2022

Quality-diversity for aesthetic evolution

Many creative generative design spaces contain multiple regions with individuals of high aesthetic value. Yet traditional evolutionary computing methods typically focus on optimisation, searching for the fittest individual in a population. In this paper we apply quality-diversity search methods to explore a creative generative system (an agent-based line drawing model). We perform a random sampling of genotype space and use individual artist-assigned evaluations of aesthetic quality to formulate a computable fitness measure specific to the artist and this system. To compute diversity we use a convolutional neural network to discriminate features that are dimensionally reduced into two dimensions. We show that the quality-diversity search is able to find multiple phenotypes of high aesthetic value. These phenotypes show greater diversity and quality than those the artist was able to find using manual search methods.

cs.NE↗

arXiv2022

Complexity and Aesthetics in Generative and Evolutionary Art

In this paper we examine the concept of complexity as it applies to generative and evolutionary art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different evolutionary art datasets and look at the correlations between complexity and individual aesthetic judgement by the artist (in the case of two datasets) or the physically measured complexity of generative 3D forms. Our results show that the degree of correlation is different for each set and measure, indicating that there is no overall "better" measure. However, specific measures do perform well on individual datasets, indicating that careful choice can increase the value of using such measures. We then assess the value of complexity measures for the audience by undertaking a large-scale survey on the perception of complexity and aesthetics. We conclude by discussing the value of direct measures in generative and evolutionary art, reinforcing recent findings from neuroimaging and psychology which suggest human aesthetic judgement is informed by many extrinsic factors beyond the measurable properties of the object being judged.

cs.NE↗

arXiv2021

Growing and Evolving 3D Prints

Design - especially of physical objects - can be understood as creative acts solving practical problems. In this paper we describe a biologically-inspired developmental model as the basis of a generative form-finding system. Using local interactions between cells in a two-dimensional environment, then capturing the state of the system at every time step, complex three-dimensional (3D) forms can be generated by the system. Unlike previous systems, our method is capable of directly producing 3D printable objects, eliminating intermediate transformations and manual manipulation often necessary to ensure the 3D form is printable. We devise fitness measures for optimising 3D printability and aesthetic complexity and use a Covariance Matrix Adaptation Evolutionary Strategies algorithm (CMA-ES) to find 3D forms that are both aesthetically interesting and physically printable using fused deposition modelling printing techniques. We investigate the system's capabilities by evolving and 3D printing objects at different levels of structural consistency, and assess the quality of the fitness measures presented to explore the design space of our generative system. We find that by evolving first for aesthetic complexity, then evolving for structural consistency until the form is 'just printable', gives the best results.

cs.NE↗

arXiv2021

Searching for Designs in-between

The use of evolutionary methods in design and art is increasing in diversity and popularity. Approaches to using these methods for creative production typically focus either on optimisation or exploration. In this paper we introduce an evolutionary system for design that combines these two approaches, enabling users to explore landscapes of design alternatives using design-oriented measures of fitness, along with their own aesthetic preferences. We test our methods using a biologically-inspired generative system capable of producing 3D objects that can be exported directly as 3D printing toolpath instructions. For the search stage of our system we combine the use of the CMA-ES algorithm for optimisation and linear interpolation between generated objects for feature exploration. We investigate the system`s capabilities by evolving highly fit artefacts and then combining them with aesthetically interesting ones.

cs.NE↗

arXiv2021

The Enigma of Complexity

In this paper we examine the concept of complexity as it applies to generative art and design. Complexity has many different, discipline specific definitions, such as complexity in physical systems (entropy), algorithmic measures of information complexity and the field of "complex systems". We apply a series of different complexity measures to three different generative art datasets and look at the correlations between complexity and individual aesthetic judgement by the artist (in the case of two datasets) or the physically measured complexity of 3D forms. Our results show that the degree of correlation is different for each set and measure, indicating that there is no overall "better" measure. However, specific measures do perform well on individual datasets, indicating that careful choice can increase the value of using such measures. We conclude by discussing the value of direct measures in generative and evolutionary art, reinforcing recent findings from neuroimaging and psychology which suggest human aesthetic judgement is informed by many extrinsic factors beyond the measurable properties of the object being judged.

cs.NE↗

arXiv2020

Deep Learning of Individual Aesthetics

Accurate evaluation of human aesthetic preferences represents a major challenge for creative evolutionary and generative systems research. Prior work has tended to focus on feature measures of the artefact, such as symmetry, complexity and coherence. However, research models from Psychology suggest that human aesthetic experiences encapsulate factors beyond the artefact, making accurate computational models very difficult to design. The interactive genetic algorithm (IGA) circumvents the problem through human-in-the-loop, subjective evaluation of aesthetics, but is limited due to user fatigue and small population sizes. In this paper we look at how recent advances in deep learning can assist in automating personal aesthetic judgement. Using a leading artist's computer art dataset, we investigate the relationship between image measures, such as complexity, and human aesthetic evaluation. We use dimension reduction methods to visualise both genotype and phenotype space in order to support the exploration of new territory in a generative system. Convolutional Neural Networks trained on the artist's prior aesthetic evaluations are used to suggest new possibilities similar or between known high quality genotype-phenotype mappings. We integrate this classification and discovery system into a software tool for evolving complex generative art and design.

cs.NE↗

arXiv2020

A scriptable, generative modelling system for dynamic 3D meshes

We describe a flexible, script-based system for the procedural generation and animation of 3D geometry. Dynamic triangular meshes are generated through the real-time execution of scripts written in the Lua programming language. Tight integration between the programming environment, runtime engine and graphics visualisation enables a workflow between coding and visual results that encourages experimentation and rapid prototyping. The system has been used successfully to generate a variety of complex, dynamic organic forms including complex branching structures, scalable symmetric manifolds and abstract organic forms. We use examples in each of these areas to detail the main features of the system, which include a set of flexible 3D mesh operations integrated with a Lua-based L-system interpreter that creates geometry using generalised cylinders.

cs.GR↗

arXiv2020

Mirror Ritual: An Affective Interface for Emotional Self-Reflection

This paper introduces a new form of real-time affective interface that engages the user in a process of conceptualisation of their emotional state. Inspired by Barrett's Theory of Constructed Emotion, `Mirror Ritual' aims to expand upon the user's accessible emotion concepts, and to ultimately provoke emotional reflection and regulation. The interface uses classified emotions -- obtained through facial expression recognition -- as a basis for dynamically generating poetry. The perceived emotion is used to seed a poetry generation system based on OpenAI's GPT-2 model, fine-tuned on a specially curated corpus. We evaluate the device's ability to foster a personalised, meaningful experience for individual users over a sustained period. A qualitative analysis revealed that participants were able to affectively engage with the mirror, with each participant developing a unique interpretation of its poetry in the context of their own emotional landscape.

cs.HC↗

arXiv2020

Understanding Aesthetic Evaluation using Deep Learning

A bottleneck in any evolutionary art system is aesthetic evaluation. Many different methods have been proposed to automate the evaluation of aesthetics, including measures of symmetry, coherence, complexity, contrast and grouping. The interactive genetic algorithm (IGA) relies on human-in-the-loop, subjective evaluation of aesthetics, but limits possibilities for large search due to user fatigue and small population sizes. In this paper we look at how recent advances in deep learning can assist in automating personal aesthetic judgement. Using a leading artist's computer art dataset, we use dimensionality reduction methods to visualise both genotype and phenotype space in order to support the exploration of new territory in any generative system. Convolutional Neural Networks trained on the user's prior aesthetic evaluations are used to suggest new possibilities similar or between known high quality genotype-phenotype mappings.

cs.NE↗

arXiv2020

Mirror Ritual: Human-Machine Co-Construction of Emotion

Mirror Ritual is an interactive installation that challenges the existing paradigms in our understanding of human emotion and machine perception. In contrast to prescriptive interfaces, the work's real-time affective interface engages the audience in the iterative conceptualisation of their emotional state through the use of affectively-charged machine generated poetry. The audience are encouraged to make sense of the mirror's poetry by framing it with respect to their recent life experiences, effectively `putting into words' their felt emotion. This process of affect labelling and contextualisation works to not only regulate emotion, but helps to construct the rich personal narratives that constitute human identity.

cs.HC↗

arXiv2019

Adaptive Music Composition for Games

The generation of music that adapts dynamically to content and actions has an important role in building more immersive, memorable and emotive game experiences. To date, the development of adaptive music systems for video games is limited by both the nature of algorithms used for real-time music generation and the limited modelling of player action, game world context and emotion in current games. We propose that these issues must be addressed in tandem for the quality and flexibility of adaptive game music to significantly improve. Cognitive models of knowledge organisation and emotional affect are integrated with multi-modal, multi-agent composition techniques to produce a novel Adaptive Music System (AMS). The system is integrated into two stylistically distinct games. Gamers reported an overall higher immersion and correlation of music with game-world concepts with the AMS than with the original game soundtracks in both games.

cs.MM↗

arXiv2019

Camera Obscurer: Generative Art for Design Inspiration

We investigate using generated decorative art as a source of inspiration for design tasks. Using a visual similarity search for image retrieval, the \emph{Camera Obscurer} app enables rapid searching of tens of thousands of generated abstract images of various types. The seed for a visual similarity search is a given image, and the retrieved generated images share some visual similarity with the seed. Implemented in a hand-held device, the app empowers users to use photos of their surroundings to search through the archive of generated images and other image archives. Being abstract in nature, the retrieved images supplement the seed image rather than replace it, providing different visual stimuli including shapes, colours, textures and juxtapositions, in addition to affording their own interpretations. This approach can therefore be used to provide inspiration for a design task, with the abstract images suggesting new ideas that might give direction to a graphic design project. We describe a crowdsourcing experiment with the app to estimate user confidence in retrieved images, and we describe a pilot study where Camera Obscurer provided inspiration for a design task. These experiments have enabled us to describe future improvements, and to begin to understand sources of visual inspiration for design tasks.

cs.MM↗

arXiv2019

Autonomy, Authenticity, Authorship and Intention in computer generated art

This paper examines five key questions surrounding computer generated art. Driven by the recent public auction of a work of `AI Art' we selectively summarise many decades of research and commentary around topics of autonomy, authenticity, authorship and intention in computer generated art, and use this research to answer contemporary questions often asked about art made by computers that concern these topics. We additionally reflect on whether current techniques in deep learning and Generative Adversarial Networks significantly change the answers provided by many decades of prior research.

cs.AI↗

arXiv2019

In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound

Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

cs.HC↗