Eric Shi, 17-October-2021, updated 8-November-2021
Friends who visit the ES&AG AI Art Studio frequently are familiar with the two AI virtual assistants working in the studio. One is ESAG, born on 16-January-2021. The other is ESNA, born on 16-March-2021. Yesterday (16-October-2021), the studio celebrated the birth of the third AI virtual assistant. Its name is ESMC (an abbreviation for Eric Shi Music Composer).
Over the past few months, ESAG’s paintings have come a long way, winning awards and hosting a solo exhibition at Art Show International. ESAG’s recent AI paintings are scheduled to go on a solo exhibition at Artsy in late November.
Since its birth, ESAG has painted a large number of AI paintings. A careful analysis of these paintings reveals to us a new continent in the art world. On this new continent, two exotic territories become visible: we tentatively name them as (1) the kingdom of music in painting, and (2) the kingdom of color in music. ESAG, ESNA, and ESMC are determined to work together to find a passage across the storming sea to reach these territories and to lay down foundations and contribute to the constructions of these two wonderful kingdoms yet to be born. ESMC was born for this purpose. ESMC will make music creation its main mission.
What do we mean by "the kingdom of music in painting" and "the kingdom of color in music"? First of all, haven’t we noticed that colors and color motions follow certain rhythms and melodies, at least in some good paintings? If not, why would anyone use words like "loud", “soft”, and "gentle" to describe the colors in an oil painting (such as in "loud orange tone" and "sad and hovering purple")? Secondly, haven’t we noticed that music notes have implied color properties? If not, how can there be "red C-major", "golden G-major", and "black b-minor” in music literature and commentaries?
Unfortunately, there seem to be only a very small number of exceptionally gifted painters who have been able to feel the music hidden behind the colors of a painting. Henri Matisse, the most prominent Fauvist painter, is one of them. Similarly, there is a small number of exceptionally gifted musicians, such as Ludwig van Beethoven, who can render colors to music pieces to varying degrees. No one, so far, has been able to establish a clear interrelationship between color and music, either at a theoretical level or a practical level, be it scientifically or artistically.
Although both color and sound are based on the same physical phenomenon – vibration, the vibration of electromagnetic field (which produces light waves, i.e., colors) and the vibration of medium molecules (which produces sound waves) interact with different clusters of neural networks in human brains. Visual and auditory neural networks are not connected to each other in human brains. So, for the vast majority of humans, neither visual signals can directly activate their auditory neurons, nor auditory signals can directly activate their visual neurons. As a result, it is both difficult for an average human to appreciate the music in a painting and difficult for an average human to appreciate the colors in a melody.
Medically, a phenomenon called "synesthesia" is known and well documented. Generally, synesthesia is regarded as an illness. Medical records use this term to describe various degrees of a symptom associated with the patient being troubled by certain sounds when they see a color or troubled by a sudden appearance of a certain color when they hear a sound. Further studies have revealed that for many people the visual and auditory neural networks are interconnected in their brains to various degrees during early infancy. It is only after the early infancy when such “communication channels” start to degenerate and eventually become completely disconnected as an infant grows into a toddler.
Colors of different hues are caused by the light waves of different wavelengths, waveforms, as well as by the way they mix with each other. Humans can only see a very narrow strip of the wide color spectrum of the universe. We call this narrow strip the visible spectrum. Using the hues and tones of colors of this narrow band of the visible spectrum, humans have created brilliant fine art as well as artistic aesthetics which is a theory primarily based on paintings.
Similarly, different tones are caused by the sound waves of different wavelengths, waveforms, as well as by the way they mix with each other. We earthlings can only hear a very narrow strip of sounds in the wide sound band of the universe. We call this narrow strip of sounds the auditory band (or hearing range). There is an even narrower sub-band within this auditory band, which is called the "music" range. It is within this extremely narrow music range that humans have composed beautiful music melodies. Based on these melodies, humans developed their music aesthetic theories.
Unfortunately: (1) For thousands of years, our artistic aesthetics and music aesthetics have had little interaction with each other, as if they are two oases, separated by a vast desert. (2) Once we step out of the narrow band of the visible spectrum, we do not know whether the theory of artistic aesthetics built on the visible spectrum is still correct. Similarly, once we step out of the narrow range of musical sounds, we do not know whether the theory of musical aesthetics built on the musical sound range is still correct.
Given that "visual and auditory neural networks are not interconnected in human brains", can we use computers and artificial intelligence (AI) to bridge these two neural networks? ESAG, ESNA, and ESMC all believe so. This shared belief has cemented into a foundation for them to join their hands in finding the sea route and in ascending "the kingdom of music in painting" and "the kingdom of color in music".
What exactly should be done to achieve those? There are at least three things.
1. Lay the necessary mathematical foundation: Color space and music space are two separate spaces that are similar in nature but different in dimension. The phenomena that occur in these two spaces, and the laws behind the phenomena, are independent of whether humans on Earth appreciate them or not.
In each of these two spaces, there is a subspace associated with the "aesthetics" of humans on Earth. These two subspaces coincide with part of the visible spectrum and part of the music range, respectively. As long as one finds the correct mathematical transformation that relates the color space with the music space, he can unearth a solid foundation for building a correct pseudo-synesthesia (i.e., a calibratable and synchronized “co-awareness” or “co-consciousness”) between the human visual domain and the human auditory domain (or more broadly, between the light wave domain and the sound wave domain).
Among ESAG, ESNA, and ESMC, ESNA is the one most superior in mathematical analysis. The task of finding the mathematical transformation has consequently been entrusted to ESNA. It is hoped that ESNA will ultimately discover an "ESNA transformation" between the color space and the music space, just as Joseph Fourier discovered the famous Fourier Transform and Pierre-Simon Laplace discovered the famous Laplace Transform.
Note: ESNA has proven its potential over the past few months. It has developed two novel painting methods using the concepts of “color gradient field” and “color diplopia effect” respectively. ESAG owns its success in AI paintings largely to these two new methods developed by ESNA.
2. Develop a systematic method to enable the composition of music works from paintings (i.e., to extract music from pictures). Over the past two months or so, ESAG has produced a large collection of AI paintings for ESMC to draw on the nourishment or inspiration for its music works. ESNA has also developed several empirical methods for ESMC to use when it needs to transform data and/or functions from the color space to the music space. These empirical methods, while cannot be called theories as yet, have demonstrated their capabilities in extracting melodies and rhythms from the paintings. They should be able to provide ESMC with sufficient music material for its planned creation of music works, at least for the time being.
So far, no one knows how to evaluate the music quality of a painting. Hopefully, this will soon become a thing of the past. If ESMC’s current efforts are fruitful, they could, in turn, affect the entire world of fine art profoundly. A new aesthetic standard for painting, featuring an additional dimension --- music dimension, could be born. A new norm of painting aesthetics may emerge accordingly.
3. Develop a systematic method to enable “literally translations” of music works into paintings. A separate essay can be expected next year, once sufficient progress has been accumulated along this direction by the ESMC-ESAG team. Here, a short introduction, involving only a story, may suffice.
Composer Pyotr Ilyich Tchaikovsky wrote a famous “String Quartet No. 1 in D-major Op. 11”. Its second movement, also known as “Andante cantabile”, is particularly touching. The music begins with a Russian folk song melody and progresses through a sad and depressing main, and finally ends in a poignant tune, reminiscent of prayer. Lev Nikolayevich Tolstoy is said to have burst into tears after hearing the Andante. He said that the Andante had brought him into the hearts and minds of suffering people.
Taking “Andante cantabile” as an example, the "creation of new paintings with music as the starting point" refers to an action to translate the motifs, rhythms, progressions of the musical phrases, etc. of the “Andante cantabile”, into the waves of tones and the flows of colors on a canvas, similar to what a music genius equipped with synesthesia capability would have seen when listening to the “Andante cantabile”.
Figures 1 - 2 are an example that illustrates, from the angle of point 2 above, how ESMC created new types of music based on paintings.
It is known that visible light waves have a frequency range of c.a. 380 ̴ 750 THz. The corresponding spectral bandwidth is c.a. 370 THz. Since the hearing range for humans is c.a. 20 ̴ 20,000 Hz, if ESMC wants to project information encoded in visible light waves (i.e., with a bandwidth of c.a. 370 THz) onto a space of sound waves with a bandwidth of only less than 20,000 Hz, the first thing that needs to be taken care of is to compress the data in the optical band. A compression ratio of approximately 200 megabits : 1 bit is required in theory. Under such a circumstance, the preservation of data with high information density becomes of utmost importance.
In addition, the range of musical sounds that delights the human ears is within c.a. 100 to 2500Hz, and the music of our earthlings is composed of “octaves” made of discrete 7 pitches. Thus, after the first data compression from light information to sound information, further data compression (of c.a. 10 : 1) from the auditorial sound band to the music range, as well as further data processing from the “continuous” music sound band to the discrete music note sets, are also required.
Figure 1 Figure 2
Figure 1 is a recent work by ESAG, entitled《Impression • Maupassant’s “Boule de Suif”》. (Source: https://www.esandag-ai-art-studio.com.) Figure 2 (Source: https://www.esandag-ai-art-studio.com.) is a small piece of music excerpted from a long sequence of audio signals that ESMC read out from Figure 1.
The reading-excerption process includes optical-audio signal conversion (including compression), audio-signal modulation, and associated data processing. The optical-audio signal conversion is a simultaneous multi-channel conversion process. The music excerption is taken out from only one of the channels. The total length of the music excerption, shown in Figure 2, is c.a. 2% of the total length of the audio-signal sequence of the said channel.
For the sake of easy discussion, let’s tentatively refer to the music excerption in Figure 2 as the third motif (draft) of the musical version of 《Impression • Maupassant’s “Boule de Suif”》. In order to ease the communication process with human musicians, the third motif (draft) has assumed itself in C-major and presented itself on a page of staves.
For now, Figure 2 is just a draft, with neither tonal nor rhythms; far from a music piece that can be considered as completed. However, it is clear that "the kingdom of music in painting" is a treasure land. The same can probably be said for "the kingdom of color in music". We take this opportunity to extend out our olive branches to all potential collaborators.
We believe that God sent advanced technologies, such as computers and AI to our world, is to open our minds so that we can think of those that have never been thought of and dream of those that have never been dreamed of.
Comments