Difference between revisions of "Generative Media"

From SI410
Jump to: navigation, search
m (Rearrange image size)
(Rearrange image size)
Line 26: Line 26:
 
Paintings are perceived by many as two-dimensional. However, upon a thorough analysis of the world's greatest paintings, it is revealed that many layers of paint create the third dimension of height. The state-of-the-art algorithms that produced ''The New Rembrandt'' created a data set of topological maps that represented brushstrokes and layers of paint, which the artist used to make the very lifelike depths and tones. <ref> (March 2021). ''The Next Rembrandt''. Retrieved from https://www.nextrembrandt.com</ref>
 
Paintings are perceived by many as two-dimensional. However, upon a thorough analysis of the world's greatest paintings, it is revealed that many layers of paint create the third dimension of height. The state-of-the-art algorithms that produced ''The New Rembrandt'' created a data set of topological maps that represented brushstrokes and layers of paint, which the artist used to make the very lifelike depths and tones. <ref> (March 2021). ''The Next Rembrandt''. Retrieved from https://www.nextrembrandt.com</ref>
  
 +
[[File:RacterPoem.jpg|350px|thumb|right|A synthetic poem generated by [https://en.wikipedia.org/wiki/Racter Racter], a computer program. <ref>Racter, published in 1984. "The Policeman’s Beard is Half Constructed: Computer Prose and Poetry"  http://www.101bananas.com/poems/racter.html. </ref>]]
 
====Generative Poetry====
 
====Generative Poetry====
[[File:RacterPoem.jpg|250px|thumb|right|A synthetic poem generated by [https://en.wikipedia.org/wiki/Racter Racter], a computer program. <ref>Racter, published in 1984. "The Policeman’s Beard is Half Constructed: Computer Prose and Poetry"  http://www.101bananas.com/poems/racter.html. </ref>]]
 
 
Generative Poetry, another subset of generative media, is when algorithms attempt to mimic the meaning, phrasing, structure, and rhyme aspects of poetry. Generating algorithmic poetry has been successful in many instances, as [[Zackary Schollhttps://zackaryscholl.com/academic/|Zackary Scholl]], from Duke University has demonstrated. He successfully generated poetry that, when posted, received positive feedback. Other engineers have developed systems that use [https://en.wikipedia.org/wiki/Case-based_reasoning case-based-reasoning] to generate formulations of a given input text via a composition of poetic fragments that are retrieved from a case-base of existing poems. The fragments are then configured according to [https://en.wikipedia.org/wiki/Metre_(poetry) metrical] rules, which govern well-formed poetic pieces. [https://en.wikipedia.org/wiki/Racter Racter] is one such program used to generate this synthetic poetry.
 
Generative Poetry, another subset of generative media, is when algorithms attempt to mimic the meaning, phrasing, structure, and rhyme aspects of poetry. Generating algorithmic poetry has been successful in many instances, as [[Zackary Schollhttps://zackaryscholl.com/academic/|Zackary Scholl]], from Duke University has demonstrated. He successfully generated poetry that, when posted, received positive feedback. Other engineers have developed systems that use [https://en.wikipedia.org/wiki/Case-based_reasoning case-based-reasoning] to generate formulations of a given input text via a composition of poetic fragments that are retrieved from a case-base of existing poems. The fragments are then configured according to [https://en.wikipedia.org/wiki/Metre_(poetry) metrical] rules, which govern well-formed poetic pieces. [https://en.wikipedia.org/wiki/Racter Racter] is one such program used to generate this synthetic poetry.
  
 +
[[File:Sunspring-screenplay.png|300px|thumb|left|Beginning of ''Sunspring'' script generated by an artificial intelligence bot named Benjamin.<ref>LA LA Film. "Sunspring: a short film written by an algorithm" https://lalafilmltd.wordpress.com/2016/06/11/sunspring-a-short-film-written-by-an-algorithm/ (11 June 2016)</ref>]]
 
====Generative Screenplay====
 
====Generative Screenplay====
[[File:Sunspring-screenplay.png|250px|thumb|left|Beginning of ''Sunspring'' script generated by an artificial intelligence bot named Benjamin.<ref>LA LA Film. "Sunspring: a short film written by an algorithm" https://lalafilmltd.wordpress.com/2016/06/11/sunspring-a-short-film-written-by-an-algorithm/ (11 June 2016)</ref>]]
 
 
Generative media can also take the form of generative filmmaking, where neural networks analyze, write, and generate films and screenplays. The machine learning models use neural networks to learn from existing text and film in order to produce new text and film based on the data. These models not only consider plot, characters, and dialogue but also keep track of the behavioral cues of characters or the background (e.g., environment, music, etc.) associated with scenes. <ref>Chu, Eric & Roy, Deb. "Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies", https://arxiv.org/pdf/1712.02896.pdf (8 Dec 2017)</ref> This allows them to analyze the emotional creativity associated with filmmaking to gain insight into what types of scenes elicit what feelings in the audience.
 
Generative media can also take the form of generative filmmaking, where neural networks analyze, write, and generate films and screenplays. The machine learning models use neural networks to learn from existing text and film in order to produce new text and film based on the data. These models not only consider plot, characters, and dialogue but also keep track of the behavioral cues of characters or the background (e.g., environment, music, etc.) associated with scenes. <ref>Chu, Eric & Roy, Deb. "Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies", https://arxiv.org/pdf/1712.02896.pdf (8 Dec 2017)</ref> This allows them to analyze the emotional creativity associated with filmmaking to gain insight into what types of scenes elicit what feelings in the audience.
  

Revision as of 13:22, 8 April 2021

Generative Media (also known as AI-generated media,[1][2] synthetic media,[3] and personalized media,[4]) is an umbrella term for the production of algorithmically and autonomously generating media forms, such as art, photographs, music, literature, and any other means of communicating creatively. It is media constructed with complex systems of mathematical formulas, which can magnify or mimic human ingenuity. Generative media goes by many names, such as "organic media," because it is created either with mathematical tools or autonomously i.e., without human intervention. The creation of creative material usually takes a significant amount of time, which generative media streamlines... drastically cutting the creation process time down, while enhancing precision. The term “synthetic media” denotes artificially generated or manipulated media, such as synthesized audio, virtual reality, and even advanced digital-image creation, which today is highly believable and “true to life.” [5]

History

B.C.E.

Algorithms and mathematical perspectives used to generate art was being done by the ancient Greeks, who lived before the current era. The Greek sculptor Polykleitos who wrote Canon, fixed proportions based on the ratio 1:√2 to construct the embodiment of, what he thought, represented a physical god (the epitome of human physique)[6] The notion of autonomously generating media was also popular amongst Greek inventors such as Daedalus and Hero of Alexandria, which designed machines capable of generating media without human intervention. [7]


C.E.

In the eighth century, current era, the Islamic world implemented complex geometric patterns to avoid figurative images becoming objects of worship. A recurring pattern is the 8-pointed star, often seen in Islamic tilework; it is made of two squares, one rotated 45 degrees with respect to the other. The fourth basic shape is the polygon, including pentagons and octagons. All of these can be recursively configured to generate a slew of trigonometric symmetries and linear transformations that include shears, shifts, reflections and rotations.[8] Then, during The Renaissance period, Italians implemented a mathematical perspective to generate their paintings. In 1415, the Italian architect Filippo Brunelleschi demonstrated the geometric linear perspective used to generate art in Florence, using trigonometry formulated by Euclid, to find the distant dimension's of objects.[9]


Digital Age

Desmond Paul Henry is a pioneer in generative media for producing some of the first works of art created by his "Drawing Machine 1", an analogue machine based on a bombsight computer and exhibited in 1962. The machine was capable of creating complex, abstract, asymmetrical, curvilinear, but repetitive line drawings. [10] Ian Goodfellow is a modern day computer scientist, which in 2014, invented a new class of machine learning named generative adversarial network (GAN), which elevated a computers creativity significantly. GAN systems work much like neural networks, in that a computer system can be trained on data to learn how to mimic the creativity of established humans. Today, GANs can be trained on a single photograph and then can generate a seemingly authentic video with many realistic characteristics. Generative media is rapidly expanding by giving rise to some unique subsets.

Subsets

Deepfakes

Deepfake videos made with a single image in 2021. [11]

Deepfakes, the subset of generative media that is of the most concern today, [12] was brought forward in late 2017 with the use of artificial intelligence algorithms to insert famous actresses' faces into adult videos.[13] Deepfakes utilize GAN systems and have been the fastest evolving subset of generative media, primarily because its building blocks, the computer source code, is open and free to anyone. When the deepfake source code became publicly available, demonstrations of its capabilities were generated, and the world sat confused when an impersonation of the 44th president, Barack Obama, seemed to portray him out of character. Jordan Peele, the comedian behind the demonstration, quickly revealed himself and explained the danger of such technology when in the hands of questionable individuals. Today, deepfake algorithms are making strides in progression, in that now they only need one picture to generate fabricated media, as oppose to a couple of years ago when it took hundreds of pictures for the system to generate videos. [14]

Researchers can identify poor-quality deep fakes quickly due to their poor skin texture mapping, unaligned or inaccurate audio alignment, flickering transposed faces, and hair movement, which is particularly difficult for computers to render. However, there are ways to use computers to examine video content and spot deep fakes. [15] In a 2018 study aiming to expose AI-generated fake faces by analyzing the eye blinking patterns, researchers Yuezun Li, Ming-Ching Chang, Siwei Lyu created a method to detect and identify abnormal blinking patterns.[16] However, while their results were promising, anonymous developers introduced new higher quality Deepfake technology for improving their blinking to pass detection. A 2020 publication discusses a method where Mittal et al. developed a deep learning network that used emotion to identify deep fakes in video content. When tested against two large deep fake identification testing datasets, they scored 84.4% accuracy on the DFDC dataset and a 96.6% on the DF-TIMT datasets.[17] The method uses video and audio data from real and fake videos to establish a baseline for emotional responses and phrases. Algorithms are used to both create deep fakes and detect them. As the technology to combat deep fakes develops, so will the ability to pass detection. For this reason, mainstream media has taken to the outright banning of deepfakes.[18]

The New Rembrandt, an artificially generated painting by a GAN system. [19]

Generative Art

Generative art a subset of generative media, is when art is generated algorithmically and autonomously, made possible by the advancements in machine learning and neural networks. Anyone can then train these algorithmic systems regarding a wide range of contexts. For example, with such a system, an artist would feed it data, such as Leonardo Da Vinci's entire painting catalog, while fine-tuning the neural net to make the decisions that the artist deems satisfactory, then generate a painting with the specifications that were taught. The partition that once existed between fabrication and originality is nearly non-existent when analyzing contemporary pieces such as “the Next Rembrandt”. In 2016, a new Rembrandt painting was designed by a computer and created by a 3D printer 351 years after the painter’s death. [20] Tech giant Microsoft and ING Group, a Dutch bank, were the two companies that put their abundance of resources together to undertake this endeavor.

Paintings are perceived by many as two-dimensional. However, upon a thorough analysis of the world's greatest paintings, it is revealed that many layers of paint create the third dimension of height. The state-of-the-art algorithms that produced The New Rembrandt created a data set of topological maps that represented brushstrokes and layers of paint, which the artist used to make the very lifelike depths and tones. [21]

A synthetic poem generated by Racter, a computer program. [22]

Generative Poetry

Generative Poetry, another subset of generative media, is when algorithms attempt to mimic the meaning, phrasing, structure, and rhyme aspects of poetry. Generating algorithmic poetry has been successful in many instances, as Zackary Scholl, from Duke University has demonstrated. He successfully generated poetry that, when posted, received positive feedback. Other engineers have developed systems that use case-based-reasoning to generate formulations of a given input text via a composition of poetic fragments that are retrieved from a case-base of existing poems. The fragments are then configured according to metrical rules, which govern well-formed poetic pieces. Racter is one such program used to generate this synthetic poetry.

Beginning of Sunspring script generated by an artificial intelligence bot named Benjamin.[23]

Generative Screenplay

Generative media can also take the form of generative filmmaking, where neural networks analyze, write, and generate films and screenplays. The machine learning models use neural networks to learn from existing text and film in order to produce new text and film based on the data. These models not only consider plot, characters, and dialogue but also keep track of the behavioral cues of characters or the background (e.g., environment, music, etc.) associated with scenes. [24] This allows them to analyze the emotional creativity associated with filmmaking to gain insight into what types of scenes elicit what feelings in the audience.

In 2016, Sunspring, a science fiction short film written entirely by an artificial intelligence bot named Benjamin, was created for the Sci-Fi London film festival’s 48hr Challenge Contest. [25] Director Oscar Sharp and NYU AI researcher, Ross Goodwin, developed the original idea. Benjamin learned to mimic the structure of films by using a recurrent neural network called long short-term memory (LSTM) to analyze dozens of science fiction subtitle files and film scripts. [25][26] After being fed seeds of information (mostly props and lines that must appear in the film per contest guidelines), Benjamin was able to produce stage directions and well-formatted character lines.[25] However, the AI experienced difficulties with creating character names that are very unpredictable and not like other words; because of this, Goodwin changed the names of all the characters to single letters (i.e., H, H2, and C). [25] Benjamin composed a pop song–after learning from a collection of 30,000 pop songs–that served as musical interlude in the film. [25] Sharp and Goodwin edited the script for length, but preserved the content created by Benjamin.

Ethical Dilemmas

Manipulation and credit/copyright issues are some of the ethical dilemmas surrounding generative media.[27]

Manipulation

Several studies have been carried out and are currently taking place to help understand how humans perceive generative media and their feelings towards algorithms in the world of fine arts and literature. "Investigating American and Chinese Subjects’ explicit and implicit perceptions of AI-Generated artistic work" is an academic paper that analyzes one such study between American and Chinese citizens. [28] The paper reveals a general opinion of artificial intelligence from the public of two of the biggest leaders of technology in the world to get a notion of how much fluctuation there is from one society to the next. The study reports that in China, the risks and dangers of fully autonomous systems have not been thoroughly articulated and debated; therefore, most Chinese citizens welcome and celebrate the technology, in sharp contrast to Western nations that are cautious with research and development. The government heavily influences public opinion in China, making generative media an efficient outlet to mislead and misinform. Society will overlook the dangerous capabilities of manipulated media in societies that show blind trust in novel technology. [29]

Credit, Copyright, & Ownership

Edmond de Belamy, AI-generated portrait created by Paris-based art collective Obvious.[30]

The rise of generative media has brought on new ethical concerns regarding ownership and copyright of the media among human collaborators. Christie's New York is a prestigious auction house, and in 2018 a historical event unfolded. The first-ever original work of art generated by artificial intelligence, Edmond de Belamy, was sold at the final price of $432,500, surpassing its opening price of $10,000. To successfully generate this piece, algorithms had to go under excessive training by being fed a slew of fine Renaissance paintings from the WikiArt database, which collected the specifications of its creators Obvious a Paris based collective, to learn to paint like a master. However, Robbie Barrat, the artist, and programmer who wrote the code to generate Renaissance-style images, was not amongst those who were compensated for the sale. The relationships formed when generating a synthetic piece of art get very complex, which obscures the proportions of involvement, subsequently splitting the reward in an unfair and unethical fashion.[31]

There have also been copyright concerns surrounding the AI’s potential authorship of the media. In Sunspring, the AI bot Benjamin is credited as both an author and tool of the film. [32] Some argue that in order to claim authorship or ownership over media, that creditor must have influenced the creative process associated with creating the media. [33] The filmmakers that collaborated with Benjamin hold mixed opinions on Benjamin’s creative abilities; those opposing Benjamin’s authorship claim that because Benjamin’s screenplay is based on its ability to imitate the previous screenplays it had been fed, it did not really create originally authentic media. [26]

References

  1. Goodstein, Anastasia. "Will AI Replace Human Creativity?" https://www.adlibbing.org/2019/10/07/will-ai-replace-human-creativity/ (30 January 2020)
  2. Waddell, Kaveh "Welcome to our new synthetic realities" https://www.axios.com/synthetic-realities-fiction-stories-fact-misinformation-ed86ce3b-f1a5-4e7b-ba86-f87a918d962e.html (30 January 2020)
  3. Hunt "Why Now Is The Time to Be a Maker in Generative Media". https://www.producthunt.com/stories/why-now-is-the-time-to-be-a-maker-in-generative-media (2020-02-15)
  4. Ignatidou, Sophia "AI-driven Personalization in Digital Media Political and Societal Implications" https://www.chathamhouse.org/sites/default/files/021219%20AI-driven%20Personalization%20in%20Digital%20Media%20final%20WEB.pdf, International Security Department (30 January 2020)
  5. Antonio Daniele and Yi-Zhe Song. 2019. AI+Art= Human. AAAI AI Ethics and Society (2019).
  6. Stewart, Andrew "One Hundred Greek Sculptors: Their Careers and Extant Works" from: Journal of Hellenic Studies (November 1978)
  7. Brett, Gerard "The Automata in the Byzantine "Throne of Solomon"", Speculum, 29 (3): 477–487, doi:10.2307/2846790, ISSN 0038-7134, JSTOR 2846790.(July 1954)
  8. Bouaissa, Malikka. "The crucial role of geometry in Islamic art" http://www.alartemag.be/en/en-art/the-crucial-role-of-geometry-in-islamic-art/ "Al Arte Magazine" (1 December 2015) (27 July 2013)
  9. Vasari, Giorgio (1550). Lives of the Artists. Torrentino. p. Chapter on Brunelleschi.
  10. Beddard, Honor. "Computer art at the V&A". Victoria and Albert Museum. (22 September 2015)
  11. Egor Zakharov, Aliaksandra Shysheya published on: 05.26.2019 "Deepfakes generated from a single image" https://www.wired.com/story/deepfakes-getting-better-theyre-easy-spot/
  12. L. Whittaker, T. C. Kietzmann, J. Kietzmann, and A. Dabirian, "“All Around Me Are Synthetic Faces”: The Mad World of AI-Generated Media," in IT Professional, vol. 22, no. 5, pp. 90-99, 1 Sept.-Oct. 2020, doi: 10.1109/MITP.2020.2985492.
  13. L. Whittaker, T. C. Kietzmann, J. Kietzmann, and A. Dabirian, "“All Around Me Are Synthetic Faces”: The Mad World of AI-Generated Media," in IT Professional, vol. 22, no. 5, pp. 90-99, 1 Sept.-Oct. 2020, doi: 10.1109/MITP.2020.2985492.
  14. Author: Dane Mitrev (March 2021). Few-Shot Adversarial Learning of Realistic Neural Talking Head Models. Retrieved from https://arxiv.org/pdf/1905.08233.pdf
  15. "How To Spot Deepfake Videos — 15 Signs To Watch For". Us.Norton.Com, 2021, https://us.norton.com/internetsecurity-emerging-threats-how-to-spot-deepfakes.html.
  16. Li, Yuezun, Ming-Ching Chang, and Siwei Lyu. "In ictu oculi: Exposing ai generated fake face videos by detecting eye blinking." arXiv preprint arXiv:1806.02877 (2018).
  17. Mittal, Trisha, et al. "Emotions Don't Lie: A Deepfake Detection Method using Audio-Visual Affective Cues." arXiv preprint arXiv:2003.06711 (2020).
  18. Hern, Alex. "Facebook Bans 'Deepfake' Videos in Run-Up To US election". The Guardian, 2021, https://www.theguardian.com/technology/2020/jan/07/facebook-bans-deepfake-videos-in-run-up-to-us-election
  19. https://www.nextrembrandt.com.
  20. United Nations Educational, Scientific and Cultural Organization (March 2021). The Next Rembrandt. Retrieved from https://en.unesco.org/artificial-intelligence/ethics/cases
  21. (March 2021). The Next Rembrandt. Retrieved from https://www.nextrembrandt.com
  22. Racter, published in 1984. "The Policeman’s Beard is Half Constructed: Computer Prose and Poetry" http://www.101bananas.com/poems/racter.html.
  23. LA LA Film. "Sunspring: a short film written by an algorithm" https://lalafilmltd.wordpress.com/2016/06/11/sunspring-a-short-film-written-by-an-algorithm/ (11 June 2016)
  24. Chu, Eric & Roy, Deb. "Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies", https://arxiv.org/pdf/1712.02896.pdf (8 Dec 2017)
  25. 25.0 25.1 25.2 25.3 25.4 Newitz, Annalee. "Movie written by algorithm turns out to be hilarious and intense", https://arstechnica.com/gaming/2016/06/an-ai-wrote-this-movie-and-its-strangely-moving/ (9 June 2016)
  26. 26.0 26.1 Frohlick, Alex. "The possibility of an AI Auteur? AI Authorship In the AI Film", https://www.researchgate.net/publication/342591536_The_Possibility_of_an_AI_Auteur_AI_Authorship_In_the_AI_Film (July 2020)
  27. Antonio Daniele and Yi-Zhe Song. 2019. AI+Art= Human. AAAI AI Ethics and Society (2019).
  28. Authors: Yuheng Wu, Yi Mou, Zhipeng Li, Kun Xu, Investigating American and Chinese Subjects’ explicit and implicit perceptions of AI-Generated artistic work, Computers in Human Behavior, Volume 104, 2020, 106186, ISSN 0747-5632, https://doi.org/10.1016/j.chb.2019.106186. Retrieved from: (https://www.sciencedirect.com/science/article/pii/S074756321930398X)
  29. Authors: Yuheng Wu, Yi Mou, Zhipeng Li, Kun Xu, Investigating American and Chinese Subjects’ explicit and implicit perceptions of AI-Generated artistic work, Computers in Human Behavior, Volume 104, 2020, 106186, ISSN 0747-5632, https://doi.org/10.1016/j.chb.2019.106186. Retrieved from: (https://www.sciencedirect.com/science/article/pii/S074756321930398X)
  30. Bryner, Jeanna. "Creepy AI-Created Portrait Fetches $432,500 at Auction", https://www.livescience.com/63929-ai-created-painting-sells.html (25 October 2018)
  31. Epstein, Ziv, et al. “Who Gets Credit for AI-Generated Art?” IScience, vol. 23, no. 9, Elsevier Inc, 2020, p. 101515, doi:10.1016/j.isci.2020.101515.
  32. Arizona State University Center for Science and the Imagination. "Machines Who Write and Edit", https://csi.asu.edu/ideas/machines-who-write-and-edit/ (16 June 2016)
  33. Ravid, Shlomit Yanisky. "Generating Rembrandt: Artificial Intelligence, Copyright, and Accountability in the 3A Era--The Human-like Authors are Already Here-A New Model", Michigan State Law Review, https://ir.lawnet.fordham.edu/faculty_scholarship/956/ (2017)