Doing the Math on College Rankings
by Taylor Trussell, Stein | October 7th, 2008An article in Science News describes how researchers at Berkeley and Carnegie Mellon used high-dimensional geometry to analyze the results of US News & World Report’s college rankings. What they found wasn’t that the rankings are based on flawed criteria (they didn’t address that question) but that the rankings shift dramatically depending on how you prioritize the seven factors USNWR considers.
The researchers argue that the magazine has prioritized these factors arbitrarily, which biases its rankings. Their goal was to see what the rankings looked like if the data were analyzed in ways that took this inherent bias into account.
After reverse-engineering the USNWR data, the researchers mapped the schools onto a seven-dimensional space correlated to the seven criteria. (And that will be the last we speak of seven-dimensional space.) This kind of arrangement viewed all seven categories equally, so the researchers avoided any ranking bias.
Then they examined the schools, ranking the seven criteria in every possible combination. In other words, they modeled all of the possible priorities prospective students might have (and not just the set of priorities USNWR assumes they have, whatever that might be).
The results? They found that the top schools were unaffected—Harvard, Yale, Princeton still rate the highest since they score the highest in all criteria.
Schools that were a bit more uneven could vary wildly, though. Penn State, for example, was 48 according to the magazine’s criteria, but it could also be as high as 1 or as low as 59. That variability evolves because Penn State is the best at making sure students graduate, … but weaker in other aspects. UC Berkeley, on the other hand, was strong in most categories except for one: alumni giving. … As a result, although U.S. News rates UC Berkeley as 21, the university could go as high as 14 or as low as 36.
The researchers suggest that USNWR publish a variety of rankings that reflect representative sets of priorities to combat this bias.
October 9th, 2008 at 10:09 am
Taylor, rather than publishing a variety of rankings, I would just dump the whole thing. You have brought out what many have suspected all along. The rankings have so many fallacies. One must always consider that they are produced for a very naive public in order to sell magazines. How in the world did colleges and universities get into such a situation? Colleges should make decisions only for the good of their students and not to improve their rankings with a magazine. Go figure! Cheers. Ross