How Netflix Uses Matching To Pick The Best Thumbnail For You
If you log into a friend’s Netflix account, you will notice that the thumbnails for the movies and tv shows listed on their account differ from yours. Netflix does this intentionally and it’s in an effort to retain its customers’ attention and interest. Netflix estimates that they only have up to 90 seconds to grab your attention before you switch to something else, so they find creative ways to grab your attention, and one of the most successful methods is thumbnails. For example, during a one-hour episode of Money Heist, there are roughly 86,400 possible frames from the show that Netflix could choose to use as a thumbnail. So, how do they go about this process?
Netflix uses Aesthetic Visual Analysis (AVA) to pick the best thumbnail to display to the user. AVA is a set of algorithms that sift through movies and shows on Netflix to identify the best frames to use as thumbnails. The first part of the process is called frame annotation, a system where a program analyzes every frame from a show and creates metadata for each frame. Next, the metadata is used to group the images into different types of shots based on composition, characters, and the quality of the picture.
At this point, Netflix has identified which frames to use as possible thumbnails, but now the process of matching each thumbnail to every user comes into play. To do this, Netflix collects data on every aspect of your interaction with the site. For example, they analyze the genres of shows and movies you watch, how much time you spend watching, where in the world you watch, and much more. They also compare your data to people with similar data as to you and put you into groups with other people based on the data they analyze. One basic example of the kind of thumbnail matching they perform is based on the cast of programs you watch. For instance, if you recently watched a show with Uma Thurman, the thumbnail for Pulp Fiction on your account will have an image with Thurman’s character rather than Samuel L. Jackson.
More recently, Netflix has begun using another machine learning algorithm called Contextual Bandits to match thumbnails to each user in real-time. Contextual Bandits uses data from your previous viewing history and also collects data on your real-time interactions with the site to create the best matchings for you. You may notice that your thumbnails change from day to day, and this is a result of this machine learning working in the background.
https://www.youtube.com/watch?v=axCBA3VD5dQ
I came across this topic from watching this Youtube video, and from here dove into more research. This video addressed the topic on a surface level but served as a great introduction to the topic.
https://www.looper.com/274997/the-secret-behind-netflixs-personalized-thumbnails/
This source was very informative about the type of machine learning Netflix collects and the data it sifts through to make the correct matchings.