the netflix prize competition
In October of 2006, Netflix announced that it would give $1 million to anyone who could improve their Cinematch recommendation engine by at least 10%. This is the Netflix Prize competition. The Netflix dataset consists of over 100 million training examples, with over 500,000 users and 17,770 movies.
final update: we tied (and lost) the netflix prize
In the years since we formed Dinosaur Planet, we helped form the collaborative teams When Gravity and Dinosaurs Unite, Grand Prize Team, and The Ensemble. The contest came to a thrilling conclusion in July of 2009 with The Ensemble placing first on the Quiz Set and second (tied but late by 20 minutes) on the Test Set. You can still view the results of the competition on the Netflix Prize Leaderboard.
update 7-26-2009: the netflix prize is over
We won!!! Well, not really. Sort of. In fact, because of the way the contest is structured, there is no guarantee that we won the competition, because it's only the Test Set RMSE (which is unknown) that matters.
What happened? Along with our old comrades Gravity, we co-founded Grand Prize Team, an ''open'' collaborative Netflix team with a transparent application process. Once BellKor's Pragmatic Chaos laid down the $1 million glove, we joined with Vandelay Industries !, another open collaborative team, to form The Ensemble, the largest, greatest Netflix wrecking crew ever assembled. This super team snatched the hotseat from the favored leader with just 24 hours to go, eventually finishing with a Quiz score just one-hundredth of a percent better.
To learn more about the last exciting month of the competition, visit The Ensemble.
a brief history of dinosaur planet
Along with two of my friends, David Lin and Lester Mackey, we started a team, Team Dinosaur Planet, named after the first movie in the database. Our approach, like that of the other top teams, is to find algorithms that capture different aspects of the data and blend them: from October 2006 to October 2007, we have implemented pretty much everything we have come across, as well as some new ideas of our own.
The final month of the 2006-2007 competition (culminating in a $50,000 Progress Prize) was pretty intense, and in the last week of the competition, we teamed up with Team Gravity (from Hungary) to form When Gravity and Dinosaurs Unite. For the last 23 hours, we reigned at the top of the leaderboard:
Unforunately, team BellKor from AT&T Research beat us out in the end, but it was an excellent competition and an awesome finish to the first year.
We are currently discussing writing up our work, so more details will come later...in the meantime, track our progress on the leaderboard.