IBM used its artificial intelligence platform Watson to create a movie trailer for the film Morgan, a tale of the fate of an artificially created human.
The IBM team worked with 20th Century Fox, it said.
John R. Smith, IBM Fellow, Manager, Multimedia and Vision, said in a blog: “How do you create a movie trailer about an artificially enhanced human? You turn to the real thing – artificial intelligence. 20th Century Fox has partnered with IBM Research to develop the first-ever “cognitive movie trailer” for its suspense/horror film, “Morgan”. Fox wanted to explore using artificial intelligence (AI) to create a horror movie trailer that would keep audiences on the edge of their seats.”
(Of course we have no idea if this Mr John Smith wrote this blog – it was posted by a Michael Zimmerman – or indeed if he actually exists or if he does exist if he is human. It could have been generated using Artificial Intelligence. At the risk of doing the man a disservice in his blog photo he looks very young to be an IBM fellow. It could be that IBM is very good at making Artificial Intelligence platforms but is not very imaginative when it comes to original character names, but we’ll go with John Smith’s blog for now.)
“Movies, especially horror movies, are incredibly subjective. Think about the scariest movie you know (for me, it’s the 1976 movie,“The Omen”). I can almost guarantee that if you ask the person next to you, they’ll have a different answer. There are patterns and types of emotions in horror movies that resonate differently with each viewer, and the intricacies and interrelation of these are what an AI system would have to identify and understand in order to create a compelling movie trailer. Our team was faced with the challenge of not only teaching a system to understand, “what is scary”, but then to create a trailer that would be considered “frightening and suspenseful” by a majority of viewers,” wrote Mr Smith.
According to the blog, using machine learning techniques and experimental Watson APIs, our Research team trained a system on the trailers of 100 horror movies by segmenting out each scene from the trailers. Once each trailer was segmented into “moments”, the system completed the following;
1) A visual analysis and identification of the people, objects and scenery. Each scene was tagged with an emotion from a broad bank of 24 different emotions and labels from across 22,000 scene categories, such as eerie, frightening and loving;
2) An audio analysis of the ambient sounds (such as the character’s tone of voice and the musical score), to understand the sentiments associated with each of those scenes;
3) An analysis of each scene’s composition (such the location of the shot, the image framing and the lighting), to categorize the types of locations and shots that traditionally make up suspense/horror movie trailers.
The analysis was performed on each area separately and in combination with each other using statistical approaches. The system now “understands” the types of scenes that categorically fit into the structure of a suspense/horror movie trailer.
“Then, it was time for the real test. We fed the system the full-length feature film, “Morgan”. After the system “watched” the movie, it identified 10 moments that would be the best candidates for a trailer. In this case, these happened to reflect tender or suspenseful moments. If we were working with a different movie, perhaps “The Omen”, it might have selected different types of scenes. If we were working with a comedy, it would have a different set of parameters to select different types of moments.”
“It’s important to note that there is no “ground truth” with creative projects like this one. Neither our team, or the Fox team, knew exactly what we were looking for before we started the process. Based on our training and testing of the system, we knew that tender and suspenseful scenes would be short-listed, but we didn’t know which ones the system would pick to create a complete trailer. As most creative projects go, we thought, “we’ll know it when we see it,” wrote Mr Smith.
Human Intervention:
In good news for creative types, or at least editing types Mr Smith says a resident IBM filmmaker arranged and edited the moments together into a comprehensive trailer complete with music.
Some of Watson’s ‘moments’ ended up on the cutting room floor.
“Not surprisingly, our system chose some moments in the movie that were not included in other “Morgan” trailers. The system allowed us to look at moments in the movie in different ways –moments that might not have traditionally made the cut, were now short-listed as candidates. On the other hand, when we reviewed all the scenes that our system selected, one didn’t seem to fit with the bigger story we were trying to tell –so we decided not to use it. Even Watson sometimes ends up with footage on the cutting room floor!”
Mr Smith says the process took just 24 hours from Watson viewing the movie for the first time to final edit of the trailer compared with a normal trailer production schedule of two weeks.
The results are available on Youtube.
