GenAI as a collage of mental percepts

During my PhD I developed an app called Sketch-to-Collage. It was a tool to create collages using simple color strokes as input. It used image retrieval to extract blobs from an image dataset and combine them in a new image that resembled the input sketch. See the example below, and this video: Sketch-to-Collage (Siggraph 2007 Posters).

Because I had to write scientific papers, I needed to come up with a method to evaluate the quality of the techniques proposed. In image retrieval, a common strategy is measuring recall and precision. With that in mind, I used the concept of “mental percepts”, a mental representation of the perceived world state. The sketch from the user represents a draft of their visual mental percepts and I can use the sketch to measure the recall and precision of my app. It’s a bit more complex than that, but that’s the main idea. You can read my PhD thesis here: “Region-based image abstraction for retrieval and synthesis” [Tokyo Tech, 2008] (PDF).

To me, Generative AI, or GenAI for short, sounds like a very sophisticated version of such a tool. Your mental percept can be represented by words now, if the input is a text prompt. But you can even use images as input, or combine several inputs, or even combine several tools, i.e. you could create a collage created with my tool as an input to create a better image from it.

Originality and copyright

Can I collage anything and stay original? We need to check the images used in the original data set. If the dataset contains a copyrighted drawing of an apple and I use it in my collage, I may be infringing copyright, even if one could argue that the collage itself is an original piece of work.

But what if I simply sample the color of the apple to paint another fruit? Are colors copyrighted? And what if I copy the brush stroke lines as well? Perhaps you see where I’m going: with GenAI is sometimes hard to find traces of any original images because the final result is a “collage” of very small image features.

Another thing to take into account when thinking of originality is the limit of what we can create. As with my “sketch-to-collage” app, GenAI depends on an initial dataset for training, so there needs to be an a-priori match between your mental percepts and what’s already there. That means that it won’t be able to draw a cat if it’s never seen one.

What is art?

We doubted the originality of GenAI, but there’s a more fundamental question to answer: Can GenAI produce art at all?

The Wikipedia says that there’s no generally agreed definition of what constitutes art, but it gives this one:

Art is a diverse range of human activity and its resulting product that involves creative or imaginative talent, generally expressive of technical proficiency, beauty, emotional power, or conceptual ideas.

The word “human” suggests that it must be made by humans to be art. I also highlighted the word “talent”, because I think many will agree that we need to appreciate some skill that excels in some way.

Here’s my attempt to give my own definition, based on the above, but rewritten so it’s easier to evaluate whether something is art or not:

Art is any piece of work created by combining mental percepts into something to be consumed by human senses, using a set of sophisticated skills. It evokes some emotion in the consumer because of its quality or originality.

Let us discuss my definition over some examples (please leave comments on each section on Medium if you want to discuss).

A banana duct-taped to a wall

❌🙅‍♂️ This doesn’t match my definition of art because it doesn’t require any sophisticated skills or talent, even though this is sold as art in art galleries.

Spanish Artist and communicator Antonio García Villarán tagged that piece as “hamparte”, a term he popularized to define art fiascos. “Hamparte” is a portmanteau of the words “hampa” (group of rascals, scammers, or con men) and “arte” (art). Since the word “con art” already exists in English with a different meaning, let me create a new portmanteau for “hamparte” in English with the words “fiasco” and “art”: fiascart.

A pianist performance of “Clair de Lune”

✅🙆‍♂️ This matches my definition of art, because you need skills to play the piano and it evokes emotions to people who listen to it. It may not be original, but it has quality.

This art is co-created by 2 people: the composer and the performer. The composer puts down his mental percepts in paper in the form of music notation. But it may not fully represent how he feels it should be played. They may write some “prompts” in the score to express if it needs to be played softer, or “virtuoso”. But in the end it’s the performer’s own mental model and interpretation what we hear.

A photograph taken with an iPhone

❌🙅‍♂️ There are no skills needed for this, so I would argue it’s not art. That doesn’t mean the picture is not pretty. Perhaps you have travelled far to take a picture of the Northern Lights, the Aurora Borealis. But the travel is not art and the picture that you take with your phone isn’t art either.

What we can derive from this is that not all pretty pictures are art. Instagram is mostly not art. These pictures can evoke emotions as well, and they can be of high quality. They can even be original, if it captures some rare event. But if they don’t require skills, I wouldn’t consider it art.

One could argue that the engineers that created the iPhone were the ones who had the skills. But there was not direct intentionality in the engineers to take that particular photo that you’ve taken. The skills should be needed in the person who directly creates the artwork.

A painting by Miró

❌🙅‍♂️ This may be controversial, but Antonio García Villarán considers Miró’s work “hamparte” —or “fiascart”, as I defined earlier. I do agree it’s not art. But it’s not the same as a banana on a wall either, so I wouldn’t classify it as “fiascart”. I think Miró’s drawings are designs. A design is created with artistic tools, but it doesn’t need to be art.

For instance, a straight line on a white T-shirt is a design. But it doesn’t require any special skills to draw such line. Similarly, if you can’t tell apart Miró’s drawings from those from a 5-year old is because what you see is not art. That doesn’t mean the design doesn’t evoke any emotions or that it’s not valuable. But as with the case with the iPhone photograph, it’s just an interesting picture.

A drawing made with Photoshop

🤔 This depends on how much talent is needed to create that drawing. If anyone can do it with a few clicks, it’s not art. Any digital tools make our lives much easier. For instance, I use Procreate on iPad to draw clean brush strokes that I wouldn't be able to reproduce with a real brush. Above there’s a dragon I’ve drawn with ink and color pencils, and then there's the same drawing I made with Procreate. You can still recognize my style in the Procreate drawing, but the finish may look a bit neater. Neither of them can be considered art, I suppose, because I’m not talented enough to create something of good quality. They can be categorized as “artistic expression”, or simply as cute drawings.

Andy Warhol was a pioneer in digital drawing. He created some iconic drawings with Deluxe Paint in the Amiga computer. Drawing with the mouse wasn’t easier than drawing with paper at the time, though. There were many talented pixel-art artists at the time that spent a lot of time and effort in their creations.

A GenAI image from a text prompt

❌🙅‍♂️ I would argue that a GenAI image from a text prompt is not art. Even if the prompt is very long and complicated, I wouldn’t consider that an artistic skill. That’s why people that master the current prompt inputs are called “prompt engineers”, and not “prompt artists”. Also note that AI is getting simpler to use, so there may no longer be any need for “engineering” either.

Some people may confuse originality with art. Even if we assume that GenAI can be original, originality alone does not define art. If there’s no talent involved, a GenAI image is just another pretty image. But this time around is created from other means, other than an iPhone camera.

A drawing improved with GenAI

🤔 There are two things to check here: the drawing and the transform. If the drawing could be considered art by itself, and the transform doesn’t make the original drawing unrecognizable, I would say it can be considered art. If we accepted a Photoshop drawing by a talented artist as art, why not in this case? The artist could be applying premade textures with Photoshop on their drawing, and it would still be art.

Another way to think of it is that we allow certain level of “cheating”, i.e. the use of certain good tools. If I were talented and the drawing I made with Procreate (left in the figure above) could be considered art, the image generated with GenAI in the middle may still be art because the original drawing is still recognizable. However, the image on the right is a completely different thing. It does represent the same mental percepts I had in mind, that is, a woman with blue hair and black tears, but it doesn’t require much talent to obtain good quality like that. In this example, it completely hides my lack of talent, so it should smell suspicious.

A GenAI song from an original poem

🤔 This is somewhat similar to creating a GenAI image from an original drawing. But there’s a difference: there’s no music at all to begin with. If the poem can be considered art, only the poem will be art, but not the song. I think this is more similar to taking a photograph of the Mona Lisa. That photograph contains a piece of art, but the photograph itself is not art.

(The video above is a poem I wrote in 2009, “Sonnet pretensions of Pluto”, now turned into a song with Suno AI.)

A GenAI song from an original song

🤔 This is close to the case of creating a GenAI image from an original drawing. It works the same way: if the original song can be considered art, then the result will also be art if the original can still be recognized in it. The linked YouTube video above is a song I created with Suno AI by making a cover of my own voice. It produces something of decent quality from something that sounds awful (if you are not afraid, here's my embarrassing original voice: “Sonnet pretensions of Pluto”, improvised version by myself). Again, it is not art because it completely hides my lack of talent. It’s just a nice song, in the same way we had nice GenAI drawings, nice iPhone photographs, and interesting designs.

A drawing from a GenAI image

✅🙆‍♂️ If the drawing is made by a talented artist, I would consider that art (not the case of my drawings of seals I shared above for illustration purposes). If we accept that searching for reference images on Internet is not cheating, then writing a text prompt to generate an image for inspiration should be OK as well. The worry is not whether it is art or not, but whether that art is original. For instance, in the extreme case that the learning dataset contains just images of Van Gogh sunflowers, you’d be copying a Van Gogh drawing.

In music, the equivalent would be a band creating a song with AI and then playing it live. The performance may be art, but there would be a worry of whether that song is original or not.

So let’s go back to the subject of originality.

Can humans be original?

After that long discussion of what art is, I think we can conclude that there are many pretty or interesting creations, either created by humans or by GenAI, that are not necessarily art. Another takeaway is that art doesn’t need to be original. A good reproduction of Las Meninas or a good performance of Gymnopédie can be considered art.

Even if GenAI is not art, it may have some value, though. And that value may be greater if it represents something original. I discussed this a bit earlier when I talked about originality and the limits of what GenAI can create, depending on the dataset used for learning. But are humans any different? Do we have any limits?

For me, the creative process starts as “collaging” in the brain, that’s why I talked about collages in the introduction. We copy and paste mental percepts in our brains and then we try to put the result down in paper. When drawing, we repeat this “copy-pasting” by copying familiar shapes, or familiar drawing styles. GenAI just reveals the human tricks that most mortals use. Only a few individuals can be truly inventive. For instance, in pop music Björk and Rosalia come to mind. Everyone else is mostly repeating the same chords, using the same instruments, and repeating the same patterns.

There may even be a limit to originality. Anime faces may all look the same because it’s hard to create original faces and still have the distinctive anime look. And even something particular to a person, like their voice, may not be original. Some of the songs I created with Suno AI have familiar voices. This song I made from a Spanish poem I wrote 20 years ago, “Si así debiera ser en el mar”, sounds like Spanish singer Anna Torroja. And this other one, “Word Wormhole”, something I wrote for the first chapter of my attempt to write a sci-fi book (“Surrealitales”), sounds to me as if sung by David Bowie. Are those singers in the dataset, or is it just that other singers have a similar tone and the “collaging” of voices end up sounding familiar? Since the AI is a black box, it’s hard to know what went into creating the final result.

I think it is very hard to be truly original. That’s why I put the “or” clause in my definition of art. But originality is tangential to talent, i.e. it’s another dimension. That means that we can create something original, interesting by itself, and GenAI can help us improve its quality.

Saturation of the senses and mediocrity rise

Since I work in Computer Graphics, I studied a bit how cameras work, in terms of lenses, sensors, and light interactions. However, I never dug deeper into photography because I became saturated with digital photographs early on. When I moved to Tokyo in 2002, all phones in Japan had cameras already. They weren’t great at the time, just 240×320 pixels on my first mobile phone. But it already meant that tons of photographs were shared continuously. It didn’t help that I focused my research on creating a database of such photographs and worked on image retrieval. And then when Facebook, and then later Instagram, appeared, we got bombarded with photographs from all our family and friends.

At the beginning those photographs were not of very good quality. But now with smartphones with intelligent processing that create beautiful moon pictures that are more fake than real, your Instagram feed looks probably amazing. It doesn’t help that phones tend to saturate colors to make images more appealing. In the end our visual cortex receives too many stimuli and we may be too tired to appreciate real art.

Let pause for a bit and think of what we've discussed earlier: most of what we see is probably not art. It’s just pretty pictures. But since everyone can create pretty pictures now, we may become immune to beauty. GenAI exacerbates the problem because now it’s even easier for everyone to create pretty pictures.

And it’s not only pretty pictures. We can create all kinds of appealing content. Now we can write like a professional novelist, or create catchy songs with just a few clicks. I’ve been pretty excited myself, specially with music. It’s so cool to be able to listen to my poems turn into songs. But when I hear a silly song about eating cookies all day made by a 6-yo, I get a bit down because I start thinking that perhaps my poems were crap to begin with and it’s just the music that makes them sound good.

Real talent is hidden in a sea of mediocrity. And the problem is that our senses can no longer tell the difference.

Conclusions

I’ve provided a definition of art that tries to reflect what I believe to be our common intuition of what art is: a piece of work for the senses created by a talented person, either original or of great quality. Under that definition, GenAI content is generally not art, because it can be produced by anyone without requiring any special skills or talent. However, it can be argued that GenAI may produce original content, or as original as the human “mental collaging” process can be.

Let me summarize my final thoughts in a few bullet points:

• The majority of the pretty pictures we see are not art.
• Art doesn’t need to be original.
• Original creations aren’t necessarily art.
• Our senses are increasingly saturated with stimuli.
• Non-talented people can now create good quality content that looks like art.
• Real talent will increasingly become hidden in a sea of good quality GenAI content.

This means that digital artists may be replaced by GenAI at places, unfortunately. But on the bright side, I suppose people will appreciate even more the art created with human hands, like paintings and sculptures, or performance arts, like theater or live concerts. Art won’t disappear, and with more people creating content, there will be more chances of finding interesting and original works.

References and links

• Marcus Pivato. “Models of philosophy”, May 2003 (PDF)— Discussion on mental percepts.
• My PhD thesis: “Region-based image abstraction for retrieval and synthesis” [Tokyo Tech, 2008] (PDF)
• Antonio García Villarán website: antoniogarciavillaran.es
• My instagram account: endavidg
• Playlist of music I generated with Suno AI: Focotaku Music playlist (YouTube), suno.com/@endavid
• A quiz I prepared to test yourself (I don't collect any answers): What is art? (Quiz)

Tweet