You may have heard of Austrian artist Gustav Klimt, and if not, you more than likely have come across one of his striking pieces without realizing it. All around the world, from prominent art galleries to souvenir shops, and even at your local Ikea store, Klimt’s work can be found triumphantly displayed in many forms. Often basking in a signature golden glow, and almost always inclusive of brilliant, defined colour dynamics, the artist consistently produced unique, visually rich pieces that remain culturally relevant, and revered, today.

The most famous of Klimt’s pieces can be found at the Belvedere Museum, in Vienna. The Kiss (Der Kuss in German) occupies a full wall, an honor reserved for key pieces, and is kept in the company of works owed to Klimt’s imaginative mind. The Belvedere owns the largest collection of Klimt’s work, which is displayed in a permanent exposition not far from other historically prominent works such as the French Enlightenment painter David’s take on a glorious Napoleon, or German Romantic Caspar David Friedrich’s dreamy Rocky Landscape in the Elbe Sandstone Mountains.

But what if The Kiss wasn’t Klimt’s true masterpiece?

In modern times, The Kiss has joined the ranks of highly commercialized, and mass-replicated pieces such as Da Vinci’s Mona Lisa, or Van Gogh’s Starry Night. Its composition, originally brought to life using oil paint on canvas, and lovingly adorned with gold leaf, has been stretched, shrunk, and parodied to fit almost any surface or purpose imaginable. The pillow covers and delicate China sets produced for quick and profitable resale purposes don’t care about doing the work justice. The work is both a tourist favourite in its hometown, and an international staple which flies off digital and physical small businesses with widely varied product offerings, experiencing something comparable to the commercialization of successful franchises (think Marvel anything plastered on absolutely everything), or celebrities whose names have become synonymous with too many products to keep track of.

Still, it’s missing a few strong competitors. In late 2021, AI technology was tapped to help solve a mystery dating back hundreds of years. Three pieces created at Klimt’s artistic peak were lost to a fire, leaving the world clutching little more than black and white photographs of their former glory, forever wondering whether we are missing out on something big.  During the same period that produced Klimt’s best-known works (The Kiss, of course, and Portrait of Adele Bloch-Bauer, a close second in fame), three works cumulatively dubbed The Faculty Paintings rose to social prominence in Vienna, stirring up a controversial scene around the artist, only to soon to soon vanish.

Through the practice of learning from Klimt’s canon of work, AI supported by Dr. Franz Smola of the Belvedere Museum, sought to recolour three prominent works, delivering a first look at what they may have truly looked like when still dressed in Klimt’s mesmerizing colors. But first, why are these works so important, and what is their story?   

At the request of the Austrian Ministry of Education, in 1894, Klimt and fellow artist Franz Matsch were commissioned to paint allegorical pieces for the University of Vienna’s festival hall ceiling. The five paintings were set to showcase four depictions of the University’s faculties, along with a grand centerpiece. All works were to be of monumental proportions and complexity, measuring in at more than 13 feet in length, each. Matsch would paint the largest, “Religion,” while Klimt was given “Philosophy,” “Medicine,” and “Jurisprudence,” or law.

We can think back to 1509, when Italian High Renaissance artist Raphael was approached by Pope Julius II. He was to fill four connected walls in the Palace of the Vatican with frescoes depicting significant areas of human knowledge and enlightenment, resulting in a collection of artworks that are studied, and visited by millions around the world every year. The fate of Klimt’s work for the University of Vienna would be starkly different. His completed works were unveiled one by one, each being disastrously received by the Austrian public, and rejected by the University. This all caused an increasingly tense societal scene to unfold. Interestingly, Parisian society found the pieces wonderful, even earning Klimt a cultural prize for “Medicine.”

Frustrated, but full of belief in his work, Klimt swayed two benefactors into purchasing the pieces for their own collections, thereby refunding the University and somewhat reclaiming ownership of his work. Unfortunately, they would change hands several times before ending up stolen by the National Socialist party in 1938. In 1944 they were brought to Scholl Immendorf, an estate Northwest of Vienna, where they would perish the following year as the area became engulfed by military combat.  

For many years, the mystery of the destroyed Faculty Paintings remained dormant. Art Historians swerved around the subject, infrequently studying or teaching on the subject of the pieces due to their incomplete and uncertain nature.

Then, Google stepped in, recruiting Dr. Franz Smola, curator at the Belvedere Museum, and Klimt expert, for help.

The process, as described by Google Arts & Culture lab lead, Emil Wallner; “At the Google Arts & Culture Lab, we had 80 images of Klimt’s colored artworks to teach the algorithm how to colorize the Faculty Paintings. When we only used these images to teach the Pix2pix algorithm, an algorithm developed by Isola et al. (2016), it learned about Klimt’s color palette, however, it did not have enough understanding of the scenes in the paintings to make a coherent colorization.

As a rough indicator, an algorithm needs 5000 images to learn one object, and 80 images are not enough for the algorithm to model Klimt’s coloring style. We tried the DeOldify algorithm, proposed by Jason Antic (2018). The model is trained on one million pictures of things in the real world, including people, animals, and buildings. The colorization is more coherent and it mimics the real-world, giving people vibrant skin tones, a range of hair hues, and a blue-grey sky. However, this model has no understanding of art nor Klimt’s colorizing style.”

More refinement was needed to create the correct level of understanding that would allow the AI system to reproduce the stylistic choices Klimt would have employed.

Google then experimented with “guiding the colorization with human-made color annotations, a model made by Zang et al. (2017). A user adds a handful of color dots to the black and white painting which informs the algorithm how to colorize it. The machine learning model detects textures and objects and propagates the color hints to similar regions.”

Finally, a novel model was developed by combining these approaches. “We developed a novel model by combining these approaches. The model has a similar structure to DeOldify, a U-net with a pre-trained ResNet-34 with self-attention, spectral normalization, and a 3-channel RGB input with color hints. It’s progressively trained with a custom feature loss from a pre-trained GAN critic.

The algorithm is trained on 91749 artworks from Google Arts & Culture. This allows the machine learning model to learn object boundaries, textures, and frequent compositions in artworks. It makes the colorization coherent and it learns how to adapt to colorization styles from several thousand artists.

As a final step, we trained it on Klimt’s colored paintings. This creates a colorization bias towards color themes from Klimt’s artworks. Although it does not model Klimt’s artistic style in full, it has a prejudice for moods, colors, and incurring motifs in Klimt’s paintings.”

The monumental works, close in size even to Michelangelo’s David, once held enough power to dazzle and revolt European audiences, and even led to the abrupt end to Klimt’s artistic partnership with the Austrian state. Google’s Arts & Culture lab has dedicated itself to providing an answer to the unanswerable, and has done quite well, but experts are still divided. To some, the digitally colorized works are no different than painted souvenirs; they aren’t originals, and never will be, no matter how close in likeness. To others, AI has managed to mirror one of the art world’s masters, completing a feat that is not to be scoffed at.

Would Klimt really have made the same decisions as the Google AI trained on his artwork? We’ll never know. Though technology can coax a close reproduction through arduous AI exercises, the slant of the maker’s hand will always remain unique.

Google ends its exploration of the lost Faculty Paintings with a brief but hopeful note. “It’s been an exciting journey to unravel the enigma left by the Faculty Paintings. In 1905, when Klimt showed the paintings together for the first time, many visitors were in shock. We too were surprised several times as we applied the clues we found. As our journey comes to an end, we hope it creates a new beginning for three of the world’s most controversial and important paintings during the Vienna Secession.”

So, did AI solve one of the art world’s mysteries? The answer is – not exactly. Still, it did provide us a few fun options of what could have been, while adding to a larger discussion around AI generated art.

As Google, and companies in line with its trajectory, pump out better, and better AI models, we can only imagine what’s next. What is clear is that AI has cemented its place in the art world, and whether you support or reject it, it isn’t going anywhere. Between art recreation, and conceptual output automation efforts like those of DALL-E 2, the growing artistic abilities of AI cannot, and should not be ignored.

Read Emil Wallner’s full account of the exploration HERE.