Then he got DALL-E 2, and things were dialed up to 11.
Artificial intelligence applications of text-to-image synthesis as DALL-E 2 Let anyone with an imagination create almost anything they want by simply typing in a few instruction words known as “requests”. You can imagine a dramatic love story between John Oliver and a cabbage or turn David Bowie’s lyrics into surreal works of art worthy of an album by Ziggy Stardust himself. But in the hands of someone as creative as Trillo, these AI tools are the equivalent of going from using a zoetrope to having Industrial Light & Magic follow your every whim using just one command line.
“It’s an incredibly exciting and overwhelming time for creators,” Trillo tells me over email. “In a sense, AI democratizes image creation so that people who are more verbal can express themselves visually. It also gives people who are already visual a way to evolve their work and pursue avenues they may never have explored.”
“[The experiment] it worked much better than I expected,” says Trillo. “My next impulse was to do a clothes swap, which is also one effect that I have practically done in previous projects, but i wanted to experiment with some other ideas before using it with a full-bodied person.” It was there that the power of the tool became so apparent to him. The generative work with AI is so fast that “we can now access a variety of ideas,” he says.
Overwhelming, limitless creative power
To create the outfits seen in the video, he used variations of several texts to guide the AI, from “oversized iridescent t-shirt” to “lavender purple puffy jumpsuit” to “futuristic fashion combo in lavender purple with faux collar. Puffy feather shoulder pads, avant-garde fashion, 2040 Japanese minimalism, Barbarella.”
“[It] it opened the door to some pretty wild designs that I would never have been able to achieve on my own,” says Trillo, who describes the power as “boundless and overwhelming.”
https://www.instagram.com/p/ChInrv1DfwW/
Another stop-motion DALL-E 2 experiment from Trillo, who also used AI text-to-speech synthesis to narrate it with the voice of Sir David Attenborough.
The difference between using a traditional technique and AI is staggering. As Trillo says, you’ll literally have to design and build 100 outfits and then have the model change in them every few frames while controlling the panning camera. Another way would be to design and build the clothes in 3D, creating the fabrics and textures, then lighting and doing the necessary compositing work over the video. But DALL-E 2 can do this with a text prompt. Not only does it generate an object like clothing, but it can recreate a picture and instantly compose something into that picture. This is a feature unique to DALL-E 2 that other AI synthesis programs do not: “DALL-E analyzes the original image’s aesthetics, lighting, perspective, everything, and seamlessly blends something new into the original image. It is extremely good at adding objects to a scene or filling in blanks, a process known as ‘painting,’“, says Trillo.
Creating an AI fashion show
Making the actual video is a straightforward and even tedious process. First, you need to capture the basic video, which is just a doll view of his wife, Shyama, on a garden path. You then need to extract the video frames and feed them one at a time to DALL-E, entering a text prompt for each one. “It’s basically AI-generated stop motion,” he says.
Some of the adaptations that didn’t make the cut pic.twitter.com/7pbIRvVZo2
– Paul Trillo (@paultrillo) August 23, 2022
“Once I explored a few different directions to go, I settled on a particular style for the clothing,” Trillo says, considering the balance of visual design variation and visual consistency. He ended up using 115 outfits, with countless more left on the cutting room floor. They had to be arranged in a way that flowed together organically, he says, but it was also unexpected. He finally used another AI program – called RunwayML – to rotoscope the image sequence in the source video. And then, to finalize the sequence, he created the floating objects you can see using DALL-E, again using stop motion and layering them for added depth.
There is no fear of AI
One of the dangers of DALL-E, Trillo believes, is that it is time-consuming. His “imagination” is so fascinating that you can easily lose hours and hours in the process of exploration. But you can curb that compulsion, he believes, and that’s precisely why it’s just such a powerful tool in a creator’s arsenal.
Trillo knows that DALL-E could have negative impacts on the creative industry, but, he insists, “Tt won’t take any work away from the visual effects artists.” If anything, he predicts, “it will create efficiencies in the work they’re already doing. It will open the door to completely new types of techniques as well as allow lower budget projects to have photorealistic VFX.”
Which makes sense. I can see how truly creative people will be confident and empowered by these new tools rather than threatened by them. I can imagine people who are great at using After Effects or Photoshop, but have limited creativity, losing jobs in the same way that many other jobs were lost to technology that empowered others to do amazing things.
Trillo makes another good point: “If everyone can create spectacle, then spectacle will become boring.”
