Art direction vs artificial intelligence: A helpful tool or an added hassle?

A recent example is Infinite Vibes’ music video for Jessy Lanza, I Hate Myself, created entirely by Dom and 15 separate AI tools. A dreamscape of images of Jessy presented as a variety of artworks, I Hate Myself began with Dom’s usual starting point of stepping away from the digital world and heading to the park or a cafe with a notebook to develop ideas.

The initial visualisation of these ideas will be created by Dom, usually in Blender, but then fed into AI tools for expansion. In this case, that first step began with the idea of a model of Jessy. To make a custom model, Dom was sent press photos of the artist to use in DreamBooth within Stable Diffusion. Here, the model can generate contextualised iterations of the image it is supplied with, allowing Dom to use “a few different methods of teaching Stable Diffusion the concept of Jessy Lanza,” as any creative would when building out the moodboard of a concept. From here, Dom moved into Stable Diffusion XL to create images in high resolution, before switching to Photoshop to extend these to the 16:9 ratio needed for a music video, using Photoshop’s Generative Fill tool. To give the AI Jessy Lanza the ability to sing, Jessy sent Dom an acapella version of the song to be used in another AI tool called SadTalker, which Dom used to make Jessy’s mouth and face move in sync with the song.

The animation phase then called for a whole host of extra tools. Stable Diffusion is back open again here, but specifically Stable Diffusion Warp Fusion, which specialises in taking in video files so Dom can augment his animations with AI generation. Stable Diffusion can also be run locally on your own computer and, as it’s open source, “there’s a huge community of people building tools for it,” unlike similar tools Midjourney and Dall-E, Dom says. In Warp Fusion, Stable Diffusion is applied on top of the footage supplied, but uses a process called “optical flow” to keep the finished product coherent. “The issue with AI animation is if you’re feeding frames through, it will see every frame as a new image, applying a new diffusion over the top. You then get these crazy, stuttering, flickering videos,” Dom explains. “Warp Fusion predicts where the movement is and keeps that the same.” Animate Diff can also be used to do this, “which completely removes the flicker, but you have less control over the prompting.”

This plethora of tools is then also combined with interfaces that the animator uses, such as Google Colabs and Automatic 1111, but also node-based interfaces like Comfy UI, where he can create his own workflows from nodes. If this all sounds a bit complex, Dom clarifies that “if you’ve ever used Touch Designer, Houdini or Blender, that’s a node-based workflow.”

As and when to pick up and put down these tools then follows a natural process. Like any other medium, using any other tool, “It’s just about iterating towards a feeling, and then iterating further until it feels right,” Dom says. In the output of Infinite Vibes, Dom uses these programmes like others pick up different paintbrushes and his own ideas and references still lead the work. As you can see in his lengthy process, AI doesn’tautomate his creativity, a process which still needs clarification from those who commission him: “Clients still don’t know the limitations... It’s amazing, but it’s not a magic button.” But, it’s also a process he treasures. “I instantly lose interest if I feel like I’m filling in the blocks a computer came up with,” he says, referring to his aversion to using AI for idea generation. “Yes, it’s useful for corporate work when you need something that’s good enough but – as someone who doesn’t come to artistic work with that perspective – I have no interest in manually colouring in the book an AI has come up with. Why bother doing it at all?”

Unfortunately, opportunities to curb these developments are often out of the hands of illustrators. It’s on the shoulders of art directors, studios, agencies, brands and so on to create work with the individuals who have inspired its direction, and it’s our view that this level of honesty and transparency should be a given in our industry (whether you’re using an AI tool or not). However, until potential regulations are in place, one way to gain a better grasp on AI’s impact on illustration is to experiment, as Seba has.

Mainly working in Midjourney, Seba’s current technique is to blend already made illustrations into new ideas, while maintaining his original style. “If the result meets my expectations, I keep one of the images,” he says. “If I feel there is more to explore, I request additional options from the AI.” Then, he’ll take matters back into his own hands, opening Illustrator and vectorising the result and editing how he pleases. “Sometimes, I use only a portion of the image from Midjourney and illustrate something new around that piece,” he expands. “Once I have an image I like, I take it to Photoshop to work on textures, lighting and shadows.”

Across Seba’s current portfolio you’ll see original 2D illustrations lifted into 3D renderings, still with a tint of an analogue shade thanks to this process. But there are also more photographic renderings, with nods to the original via colour palette and objects rendered into real life. “In this case, I use one of my illustrations as an input to which I add prompts (text) to help generate a new image,” the illustrator explains. “The AI takes that information and returns a set of four images based on it, which sometimes align with my expectations and sometimes take unexpected turns.” At times it can take Seba days to reach an image he likes “and that is the part I enjoy the most of the whole process,” he says, “the search for something new and unusual.” In short: “AI empowers me to extend the visual narrative of my artworks.”

In only using his own work to generate new work, Seba’s quick to clarify he can “only speak about the way I use artificial intelligence”, when it comes to artists’ work being used in wider data sets. Interestingly, Seba’s AI experiments are also continuously well received amongst his community online. Much of this respect seems to be the result of Seba outwardly demonstrating his process, posting the original, the failures and the final AI-collaborated piece together. “I believe that showing the visual progression of a work from conception to completion can inspire curiosity and value,” he says. “In addition, my approach to AI focuses on free exploration… This constant search for innovation and authenticity is what can help create a connection with those who appreciate my work.”

This openness to “multiply my artistic possibilities” has encouraged Seba to also see future possibilities for collaboration. Considering much of his work will be in tandem with an art director’s brief, the illustrator states how integrating AI into this back and forth could be viewed “as a positive thing,” he says. “I think it opens a door to a closer collaboration between [illustrators and art directors].” For example, Seba recently collaborated with fellow AI-inquisitive artists to generate prompts separately “and, once we had the images we liked, we merged them to create something new,” he says. “I imagine this same approach could be applied with an art director who set out to work with AI. We could each bring our perspective to the table and, from there, we would begin to generate images that would gradually approach a final result that we both agreed on.”

What Base was developing at this stage was a series of visual imaginings in relation to the opera being created. Developed in tandem with the opera house, the Base team would input parts as they received them into Dall-E, which was chosen as it “provided the best results in terms of its weirdness and imperfect generation,” adds Arthur Dubois, Base’s motion designer. Arthur then used Google Colab to “inject life into still images” through animation, also working with Tokkin Heads, “an AI that generates characters and the facial features to apply a templated animation, or just record with your webcam to create an animation of the face,” he says. Runway was then also used, due to its framing interpolation, which generates in-between frames for a type-to-image transition. The final AI tool the team employed was Upscayl, used to upscale every image generated to principal resolution.

Although these steps appear simple when noted down, the actual process of working with AI in a design context wasn’t without its challenges. Firstly, the imagery generated by Dall-E tested the creative limits of an identity system such as this. “It really allowed for an aesthetic that felt less corporate or commercial,” says Bruce Vansteenwinkel, a designer at Base. But at times these images weren’t appealing from an aesthetic standpoint. Thankfully, the animation techniques used “really helped us to increase the weirdness of each visual,” adds Aurore. “Even more so when we weren’t really fond of an image, adding motion could help bring the result we wanted.”

The process of then delivering the work also proved harder than originally anticipated. When it came to collaborating with La Monnaie’s team “we realised that it was a bit more hassle,” explains Bruce. “We sold it as a toolbox. A way of collaborating where we could create images, they could create images, but that creates a lot of opportunities for new misunderstandings and new frustrations. Things became out of hand.”

However, losing control led Base to realise the necessity of human-led design when collaborating with artificially generated artworks. “It’s funny. We sold them a concept of losing control, but then we lost so much control we had to gain it back,” Bruce says. “Right before launching there was an internal crisis because some of the images we’d developed were less striking than the originals we had presented. That was the concept, but because we’re designers – neat freaks, control freaks – we wanted to regain control of the image. The delight we had in the beginning turned into despair, and maybe even disappointment.”

The final visual not representing an art director or designer’s initial vision is a possibility on any project. In the end, Base’s team placed themselves back in their usual role in a brief such as this, inputting details like the aforementioned animation techniques to tie the campaign together. “We were a bit naive,” says Aurore. “A tool where both we can create images, and the client can create images sounds wonderful, but looking at what they’d done we realised we were a very important part of the process. We have a culture of image. They have a culture of content. We understood that we were still needed, which is the whole question around AI.”

Like Seba’s communication approach, since completing the La Monnaie project Base has been extremely open, and proud, of this use case of AI as art direction and design. As pointed out by Manon: “It’s our duty to use these tools and see where they can bring us to.” There has also been little backlash towards the agency, even though its team have used it so obviously.

Arguably this is due to the fact that AI was only implemented due to its relevance for the overall theme of the opera season at hand, rather than “an excuse not to work, or have AI as a buzzword,” says Bruce. “On this project I received a lot of questions about whether AI is going to replace our jobs. In our personal experience, I think in many ways it can. But that’s more of a choice than an actual fate you have to accept. There are a few agencies working towards owning AI as their unique selling point, but I’m not sure whether that’s the strongest way to move forward with design at large. Choosing when and how to use it, maybe a little sparingly as well, evades the question of whether it will take our jobs.” Interestingly, since completing the project Base hasn’t used AI tools to this extent again, because, with this experience in mind, the concept hasn’t called for it.