NATURALISM
An interactive painting, co-created by an artist, a machine, and an audience
Early experimental outputs from the Naturalism system
The work embraces new models of artistic authorship and co-creation, as popularized by MIT’s Open Documentary Lab, and opens the process of making to both the audience and non-human agents (in the form of machine learning models). I, as an artist and initiator, assume to role of a system developer, rather than the author of a single, final deliverable. With this project I am interested to create work that is, by-design, open to chance, interpretation and intervention by both artificial intelligence and the audience, creating a platform for the two to interact and interpret each other.
Naturalism is a new Work-in-Progress that explores the shifting role of the artist in the age of artificial creative intelligence. Presented as an interactive installation, the project chains together several machine learning models, some tasked with boy segmentation and others with image generation, in the creation of a compound automatic artist that transforms audiences into paintings of imagined natural landscapes.
Concept
In Naturalism, I assume the role of a connector-of-dots, a system-maker that acts as a bridge between the machine and its audience. In fact, in the lack of an audience, which act as the machine’s input, the work does not exist at all. While some decisions must be made (which networks and models to use, how to filter data, how to present the imagery), I tried to leave most of the audience <-> machine connection as uninterrupted as possible.
Does this make me less of an artist and more of a tool maker? Do I still have the right to claim the work as my own, even though it is co-created by a machine and a viewer? What is the artwork itself, the machine or the images it produces?
Machine learning and artificial intelligence have become common-place in modern life, in anything from smart assistants to shopping recs., driving aids and chat bots. The models and algorithms that drive this virtual intelligence are vastly different from traditional logical algorithms, and rely on internal networks and convolutions that are trained in a statistical process, this yields a box that, if not black, is definitely more opaque. As models and agents grow in complexity, so our intuitive understanding of their inner workings diminishes, and their reactions, decisions and outputs become unpredictable, surprising, and full of chance.
This fact inspired me to explore artificial intelligence as a an equal, autonomous creative collaborator in the creation of visual art. A machine that will not only aid me in the development and creation process, but will in-fact be at the core of the work itself. The formalized process was inspired by MIT’s Open Doc Lab’s principles of co-creation, noting particularly the creative possibility in open collaboration with non-human agents and opening the creation process to outside input and contribution. It was also inspired by recent cries that marked machine-made images as the end of art as we know it.
Naturalism’s detection steps and final output
Process
Naturalism’s real-time generative system is made of several machine learning models combined to create a single purpose-built network that chains together detection, segmentation, generation and stylization operations.
Body Segmentation
I used OpenPose to detect the presence of human figures in a live video feed, the result of the model is a UV-map of the detected body. This allows not only pose estimation but texturing and keying the surface of the body in the image, making it possible for the system to accomodate different body shapes and even clothing.
SCENE GENERATION
To transform the pose of audiences into a natural scene, I use NVidia’s GauGAN, a neural network capable of converting color-indexed doodles into images of natural landscapes. Different elements: mountains, seas, skies and trees are encoded as different colors. Depending on the viewer’s pose, their detected outline is painted in a different color and placed in the scene, it is then drawn as the corresponding natural element. The final output of this model, as seen below, is a composed natural landscape image made of the indexed elements.
Transforming pose-segmentation into color-index doodles.
These are then transformed into a natural landscape using GauGAN.
Model chaining
To build the project, I had to create an entirely new pipeline for the real-time connection of image-based machine learning models. This pipeline is implemented on the graphics processing unit (GPU) and maintains the output of one model in memory, to be fed directly as the input of the next. This approach cuts on heavy computational time in moving data between the GPU and CPU, which is a significant bottleneck of image-based machine learning, and allows the project’s system to reach real-time performance speeds (nearly 20 FPS).
Presentation (proposal)
Paintings mounted in a gallery space. The order and grouping of mounting creates a visual story that is unfolded as visitors walk through a space.
The installation will be presented as a series of projected framed canvases, each with an attached camera for audience detection and interaction. Each canvas presents a base natural scene: ocean-side, mountain and meadow, with various elements slowly animated over time.
Mounted beside each other, the canvases form a path for users to follow. As they approach the first, the interaction of the installation is revealed and viewers are encouraged to explore different poses and how these translate to different natural elements. Each canvas interprets poses differently, as each network is initialized individually. This opens up the possibility of comparative exploration, and for audiences to find the scenery they relate to most during play.
TOOLS USED
Hardware
Short-throw projector
Wooden frame
Windows PC
Software
PyTorch models
BodyPix
PoseNet
GauGAN
