IMDB Top 100K Movies Analysis in Depth Part 1

Data

Data is from IMDB and it includes all of the popularly voted 100042 movies from 1950 to 2013.(I know why 100000 is there but have no idea how 42 movies get squeezed. Instead of blaming my web scraping skills, I blame the universe, though).

The reason why I chose the number of votes as a metric to order the movies is because, generally the information (title, certificate, outline, director and so on) about movie are more likely to be complete for the movies that have high number of votes. Moreover, IMDB uses number of votes as a metric to determine the ranking as well so number of votes also correlate with the rating as well. Further, everybody at least has an idea on IMDB Top 250 or IMDB Top 1000 which are ordered by the ratings computed by IMDB.

Although the data is quite rich in terms of basic information, only year, rating and votes are complete for all of the movies. Only ~80% of the movies have runtime information(minutes). The categories are mostly 90% complete which could be considered good but the certificate information of the movies is the most sparse (only ~25% of them have it).

This post aims to explore data for diffferent aspects of data(categories, rating and categories) and also useful information(best movie in terms of rating or votes for each year).

I will let the data speak for itself by providing scatter plots and sometimes histograms as well as useful tables for movies.(I know tables are the worst but in some cases they are still useful.)

As categories go, I considered 23 categories although there are 25 categories in the top 100K movies. I removed Game-Show and TV-Reality movies as they have less than 5 movies in the top 100K movies.

Let's get started.

Best Rated Movies for Each Year

I put the rating as 8.5 and also filtered the movies that have less than 25000 votes as there are number of movies which get quite high ratings whereas the number of votes are quite small.

In [82]:
Year Movies
0 1950 Sunset Blvd.
1 1951
2 1952
3 1953
4 1954 Rear Window - Seven Samurai
5 1955
6 1956
7 1957 12 Angry Men
8 1958
9 1959
10 1960 Psycho
11 1961
12 1962
13 1963
14 1964 Dr. Strangelove or: How I Learned to Stop Worr...
15 1965
16 1966 The Good, the Bad and the Ugly
17 1967
18 1968 Once Upon a Time in the West
19 1969
20 1970
21 1971
22 1972 The Godfather
23 1973
24 1974 The Godfather: Part II
25 1975 One Flew Over the Cuckoo's Nest
26 1976
27 1977 Star Wars
28 1978
29 1979
30 1980 Star Wars: Episode V - The Empire Strikes Back
31 1981 Raiders of the Lost Ark
32 1982
33 1983
34 1984
35 1985
36 1986
37 1987
38 1988
39 1989
40 1990 Goodfellas
41 1991 The Silence of the Lambs
42 1992
43 1993 Schindler's List
44 1994 The Shawshank Redemption - Pulp Fiction - Forr...
45 1995 Se7en - The Usual Suspects
46 1996
47 1997 Life Is Beautiful
48 1998 Saving Private Ryan - American History X
49 1999 Fight Club - The Matrix
50 2000 Memento
51 2001 The Lord of the Rings: The Fellowship of the R...
52 2002 The Lord of the Rings: The Two Towers - City o...
53 2003 The Lord of the Rings: The Return of the King
54 2004
55 2005
56 2006 Rang De Basanti
57 2007 Like Stars on Earth
58 2008 The Dark Knight
59 2009
60 2010 Inception
61 2011 The Intouchables
62 2012 The Dark Knight Rises
63 2013

Highly Voted Movies per Year

In [175]:
Year Movies
0 1950 Sunset Blvd.
1 1951 Alice in Wonderland
2 1952 Singin' in the Rain
3 1953 Roman Holiday
4 1954 Rear Window
5 1955 Rebel Without a Cause
6 1956 The Searchers
7 1957 12 Angry Men
8 1958 Vertigo
9 1959 North by Northwest
10 1960 Psycho
11 1961 Breakfast at Tiffany's
12 1962 To Kill a Mockingbird
13 1963 The Great Escape
14 1964 Dr. Strangelove or: How I Learned to Stop Worr...
15 1965 The Sound of Music
16 1966 The Good, the Bad and the Ugly
17 1967 The Graduate
18 1968 2001: A Space Odyssey
19 1969 Butch Cassidy and the Sundance Kid
20 1970 Patton
21 1971 A Clockwork Orange
22 1972 The Godfather
23 1973 The Exorcist
24 1974 The Godfather: Part II
25 1975 One Flew Over the Cuckoo's Nest
26 1976 Taxi Driver
27 1977 Star Wars
28 1978 The Deer Hunter
29 1979 Alien
30 1980 Star Wars: Episode V - The Empire Strikes Back
31 1981 Raiders of the Lost Ark
32 1982 Blade Runner
33 1983 Star Wars: Episode VI - Return of the Jedi
34 1984 The Terminator
35 1985 Back to the Future
36 1986 Aliens
37 1987 Full Metal Jacket
38 1988 Die Hard
39 1989 Indiana Jones and the Last Crusade
40 1990 Goodfellas
41 1991 The Silence of the Lambs
42 1992 Reservoir Dogs
43 1993 Schindler's List
44 1994 The Shawshank Redemption
45 1995 Se7en
46 1996 Trainspotting
47 1997 Titanic
48 1998 Saving Private Ryan
49 1999 Fight Club
50 2000 Gladiator
51 2001 The Lord of the Rings: The Fellowship of the Ring
52 2002 The Lord of the Rings: The Two Towers
53 2003 The Lord of the Rings: The Return of the King
54 2004 Eternal Sunshine of the Spotless Mind
55 2005 Batman Begins
56 2006 The Departed
57 2007 No Country for Old Men
58 2008 The Dark Knight
59 2009 Avatar
60 2010 Inception
61 2011 Harry Potter and the Deathly Hallows: Part 2
62 2012 The Dark Knight Rises
63 2013 Man of Steel

Category Distribution (Mean Values)

In [77]:
Out[77]:
<matplotlib.text.Text at 0x10f061c10>

Runtime vs Rating

In [19]:
Out[19]:
<matplotlib.text.Text at 0x1221edc50>

There are small number of movies which is shorter 50 minutes and longer than 200 minutes. What else can we get this graph? Well, one thing, not average runtimes(>120 and <70) are correlated high ratings. This may due to selection bias; people who prefer longer movies, have specific taste or interest to the movie so they enjoy movie more.

Votes vs Rating

In [20]:

There is a clear correlation between number of votes and rating as IMDB explains(only regular voters are considered for top 250 movies, though):

weighted rank = \(\frac{v*X + k*C}{v+k}\)
where

\(X\) = average for the movie (mean)
\(v\) = number of votes for the movie
\(k\) = minimum votes required to be listed in the top 250 (currently 25000)
\(C\) = the mean vote across the whole report (currently 6.90)

Therefore, it may be safe to assume this ranking more or less holds true for non-top 250 movies as well.

Rating vs Year

In [8]:
Out[8]:
<matplotlib.text.Text at 0x116847250>

I think this result is mostly due to selection bias as most people prefer good old movies whereas they are not that selective choosing contemporary movies.

Votes vs Year

In [13]:

Number of Movies for each Year

In [14]:
Out[14]:
<matplotlib.text.Text at 0x10b13e810>

Number of Movies with Rating

In [15]:
Out[15]:
<matplotlib.text.Text at 0x10abac890>

Number of Movies with Runtime

In [16]:
Out[16]:
<matplotlib.text.Text at 0x10acc4390>

Most of the movies are in the band of 90-100 minutes, it is almost like a gaussian if it is distributed a little evenly between 80 and 100 minutes.

Long Movies

In [13]:
Out[13]:
title year rating runtime votes
3373 Gangs of Wasseypur 2012 8.6 320 20236
4087 1900 1976 7.8 317 14487
4241 The Best of Youth 2003 8.5 383 13439
7405 Satantango 1994 8.5 450 4584
7996 War and Peace 1966 7.7 427 3958
16123 The Deluge 1974 8.0 315 1009
19135 Little Dorrit 1988 7.6 357 714
21310 La r&#xE9;volution fran&#xE7;aise 1989 7.5 360 579
21321 The Great Battle 1969 8.0 470 578
22608 La Commune (Paris, 1871) 2000 7.8 345 517
22766 My Uncle Napoleon 1976 8.5 770 511
23770 Aszparuh 1981 8.4 323 471
24230 Hitler: A Film from Germany 1977 7.7 442 454
24485 And Quiet Flows the Don 1957 7.8 340 446
25102 Broken Saints 2003 7.5 720 425
29248 Out 1 1971 7.9 729 314
29605 The Cure for Insomnia 1987 5.9 5220 306
32547 Sleep 1963 4.1 321 252
33165 Shchit i mech 1968 7.7 344 243
37346 Noce i dnie 1975 7.5 632 189
40239 Bitva za Moskvu 1985 7.3 358 160
42022 Khleb - imya sushchestvitelnoe 1988 7.4 580 146
42526 The Longest Most Meaningless Movie in the World 1970 6.8 2880 141
43661 Legenda o Tile 1977 7.6 311 133
46124 Sobri, ponyvafilm 2002 3.9 301 117
48382 Melancholia 2008 7.1 450 104
50639 Star Spangled to Death 2004 7.2 402 93
52108 Evolution of a Filipino Family 2004 7.3 540 86
53045 The Satin Slipper 1985 7.2 410 82
55760 Death in the Land of Encantos 2007 7.3 540 72
60766 Batang West Side 2001 7.1 315 57
61110 Century of Birthing 2011 7.3 360 56
61546 Apocalypse Now: The Workprint 1978 7.6 307 55
63955 Heremias: Unang aklat - Ang alamat ng prinsesa... 2006 7.5 540 50
64344 The Cremaster Cycle 2003 6.3 398 49
64742 **** 1967 6.5 1100 48
65313 The Big Red One: The Reconstruction 2005 7.6 316 47
70520 The Weathered Underground 2010 6.6 310 38
72636 Florentina Hubaldo, CTE 2012 6.1 360 35
80673 Soldaty Svobody 1977 5.5 420 26
81284 Die Nacht 1985 8.0 360 25
83139 Konstantin Filosof 1983 5.8 308 24
83321 Scooby-Doo! Laff-A-Lympics: Spooky Games 2012 7.3 308 23
84497 Matrjoschka 2006 5.4 5700 22
84621 The New Media Bible: Book of Genesis 1979 6.7 312 22
90440 In Search of Myster Ey 2008 4.3 2100 18

Well, what can we say? Apparently, there are long movies and people are not watching most of them(looking at the votes).

High Rating Movies

In [14]:
Out[14]:
title year rating votes
0 The Shawshank Redemption 1994 9.3 1157455
1 The Dark Knight 2008 9.0 1125214
3 Pulp Fiction 1994 9.0 893719
4 Fight Club 1999 8.9 881325
5 The Lord of the Rings: The Fellowship of the Ring 2001 8.9 855325
6 The Lord of the Rings: The Return of the King 2003 8.9 828522
8 The Godfather 1972 9.2 805802
21 Schindler&#x27;s List 1993 8.9 587233
32 The Godfather: Part II 1974 9.1 529579
94 The Good, the Bad and the Ugly 1966 9.0 347858
154 12 Angry Men 1957 8.9 286958
3419 Waar 2013 9.4 19775
4553 The Rascals&#x27; Class Misses its Grades 1976 8.9 11858
4985 The Foster Brothers 1976 8.9 9973
5646 The Marathon Family 1982 8.9 7811
10937 The Moromete Family 1988 8.9 2157
11107 Yesterday 1988 9.1 2095
52166 Hunter X Hunter: The Last Mission 2013 9.0 86
55565 How to Act and Eat at the Same Time 2012 8.9 73
56594 Betty I Am 2012 8.9 69
56624 Angels of Darkness 1954 9.0 69
69376 The Voice of La Raza 1972 8.9 40
71004 Annabel Lee 2009 8.9 37
73185 Santiago 2011 8.9 34
73896 Muse - Live at Rome Olympic Stadium 2013 8.9 33
74674 Ilegales 2010 8.9 32
76091 Nawab Sirajuddaula 1967 8.9 31
80372 Kraljeva zavrsnica 1987 8.9 26
80535 Kljuc 1965 8.9 26
82902 Od petka do petka 1985 9.0 24
84791 Akher ayam el ard 2009 8.9 22
85750 Rat Rod Rockers! 2010 8.9 21
88748 Bella Biondina 2011 8.9 19
92352 Silnee vsekh inykh veleniy (Syuita na russkie ... 1987 8.9 17
92436 Timon 1973 9.1 17
93706 Poslijepodne jednog fazana 1972 9.1 16
93981 Whichever Way the Ball Bounces 1974 9.1 16
94001 Benny Hill: A Tribute 1992 9.1 16
95737 Idu dani 1970 9.0 15
96666 Zaghlul and Zaharias: Zagreb in My Life 1994 8.9 15
97327 Agnes 2013 9.1 14
97374 Separate Checks 2011 8.9 14
97670 The Way to Paradise 1970 9.1 14

High Rating and Best Voted Movies

In [15]:
Out[15]:
title year rating votes
0 The Shawshank Redemption 1994 9.3 1157455
1 The Dark Knight 2008 9.0 1125214
3 Pulp Fiction 1994 9.0 893719
4 Fight Club 1999 8.9 881325
5 The Lord of the Rings: The Fellowship of the Ring 2001 8.9 855325
6 The Lord of the Rings: The Return of the King 2003 8.9 828522
8 The Godfather 1972 9.2 805802
21 Schindler&#x27;s List 1993 8.9 587233
32 The Godfather: Part II 1974 9.1 529579
94 The Good, the Bad and the Ugly 1966 9.0 347858
154 12 Angry Men 1957 8.9 286958

Low Rating Movies

In [16]:
Out[16]:
title year rating votes
14397 Ax &#x27;Em 1992 1.2 1262
17138 Gulczas, a jak myslisz... 2001 1.2 896
17467 Yyyreek!!! Kosmiczna nominacja 2002 1.1 860
19121 Proud American 2008 1.1 715
19277 The Maize 2: Forever Yours 2006 1.2 704
19398 The Legend of the Titanic 1999 1.3 695
20206 Santa and the Ice Cream Bunny 1972 1.3 644
20255 Xuxa Popstar 2000 1.3 641
20662 Rocket Attack U.S.A. 1961 1.4 616
21230 Hunting of Man 2003 1.4 583
21901 Ricky 1 1988 1.3 549
22187 Leather and Iron 2002 1.3 535
23079 Kac Wawa 2012 1.4 498
23553 99 2006 1.4 480
23971 Szuperbojz 2009 1.3 464
25054 Xuxinha and Guto Against the Space Monsters 2005 1.3 427
25215 Cockfight 2002 1.3 422
25310 Lost in the Woods 2009 1.4 418
25986 Holnap t&#xF6;rt&#xE9;nt - A nagy bulv&#xE1;rfilm 2009 1.0 397
26109 Killer Bitch 2010 1.4 393
27263 Tangy Guacamole 2003 1.4 362
28018 Super Agent K9 2008 1.4 343
29872 The Sobbing Stone 2005 1.4 301
31267 Venni vidi e m&#x27;arrapaoh 1984 1.2 274
31331 Shirley Pimple in the John Wayne Temple of Doom 2000 1.4 273
34032 In Search of the Titanic 2004 1.3 231
36703 Xuxa Twins 2006 1.2 196
37730 Xuxa and the Lost Treasure&#x27;s City 2004 1.3 185
38930 Captain Battle: Legacy War 2013 1.4 172
44275 Body Motion Reads 1997 1.4 129
44572 The Auditor&#x27;s Code 1997 1.4 127
49928 The Warrior Didi and the Ninja Lili 2008 1.4 97
53898 After Party 2002 1.4 79
54542 Composition of the Bank 1966 1.4 77
63271 The Secret of Flag Results 1977 1.2 52
63272 The Bank and Its Pattern 1964 1.3 52
63276 Reality 1969 1.2 52
63750 General Information 1966 1.4 51
65180 The Pattern of the Bank 1964 1.4 48
67928 No Longer My Twin 2002 1.4 43
70941 Avise la fin 2012 1.0 38
72605 Ostra Randka 3D 2013 1.3 35
76254 Futures Past 2012 1.3 31
80445 Ich denk&#x27;, mich tritt ein Pferd 1975 1.1 26
84880 Genderuwo 2007 1.0 22
86465 Half Way to Hell 1961 1.2 21
87340 Primavera di granito 2002 1.1 20
99148 Curse of Kilimanjaro 1978 1.4 14
99366 Pride of Ireland 2000 1.2 14

Overrated Movies (High Number of Votes and low ratings)

In [17]:
Out[17]:
title year rating votes
394 The Twilight Saga: New Moon 2009 4.6 178416
531 Batman &#x26; Robin 1997 3.6 148286
566 The Twilight Saga: Eclipse 2010 4.9 142679
577 The Twilight Saga: Breaking Dawn - Part 1 2011 4.9 140406
816 Wild Wild West 1999 4.7 107150

Certificate

In [160]:
Out[160]:
<matplotlib.text.Text at 0x110dbc110>

Apparently, there are not good movies for people who are older than 7 years-old(however little, they are people, too!)(TV_Y7 => suitable for children age 7 or older)

In [159]:
Out[159]:
<matplotlib.text.Text at 0x117584390>

Sorry, under 14 years-olds, you are missing a lot of discussion as well! (TV_14 => is suitable for people who are 14 or older)

In [84]:
Out[84]:
Movie Count
Drama 48406
Comedy 30167
Romance 13408
Thriller 13399
Action 12408
Crime 9795
Horror 8697
Adventure 6545
Family 4853
Mystery 4356
Sci-Fi 4342
Fantasy 4155
War 2923
Musical 2735
Western 2180
History 1989
Animation 1955
Biography 1597
Music 1558
Sport 1039
Film-Noir 255
Adult 23
News 6

Drama and Comedy are big in IMDB and movie industry apparently.

Distribution of Movies for each Category over Rating

From the following distributions, purple is the movie distribution for each category and turquoise is the mean distribution of all of the movies across different categories.

In [167]:

Distribution of Movies for each Category over Year

In [141]:

Drama, Comedy and other main categories almost follow the mean distribution unsurprisingly. Other than mainstream categories, almost other categories have higher number of movies than their historical means except Western and Film-Noir. Adult and News categories have quite a few numbers so it may not be meaningful to reach any conclusions for these categories.

Distribution of Movies for each Category over Runtime

In [142]: