Methodology FAQ

Are survey-based opinions from alumni valid? Won't grads inflate their responses to make their schools look good?

Opinion-based surveys have a long and proven history and are well known as a valid research approach, as long as certain conditions are met. The first of these is anonymity, without which, survey respondents may alter their responses out of fear of reprisal. The Alumni Factor does not have any way to identify our respondents, and ensures they know their survey information will be aggregated with other responses to preserve anonymity.

The second condition is that the research be conducted by a disinterested third party. Were the colleges and universities themselves conducting the survey, graduates might be tempted to "soften" their feedback out of a misguided sense of loyalty, or to sharpen it to "send a message" to the administration. The Alumni Factor, as an independent researcher, takes these emotional responses out of the equation and thereby gets a more accurate view.

The third condition, which is just good research "hygiene," is that any unusual outliers in the data are identified and removed. We can and do validate the accuracy of responses by cross-referencing independent data where possible. For example, we ask alumni to indicate the range into which their SAT scores fall. Since colleges and universities report SAT data as well, we can cross-check that the ranges alumni report are consistent with those reported by schools.

Finally, if alumni were systematically and regularly inflating their scores to make their schools look good, you would expect this to show up in the data as virtually no difference in ratings across the colleges and universities we examined, as well as virtually no difference across the ratings of different outcomes within an individual school. As you delve into the data, you will see this is not the case. Differences reported are significant. Keep in mind there would be no reason to believe graduates of one school would inflate at a higher or lower rate than those of other schools, so the relative comparison between schools - the ranking - would still be a valid measure, even if inflation of responses were to occur.

Why do you have only 227 schools featured in the book? How did you choose which 227 to feature? We began our data collection effort with over 450 universities and colleges popularly considered among the ranks of the elite. Schools that are not included in our ranking did not have enough responses to be statisti- cally valid at the time we went to press.

What does it mean to be ranked #1 versus #15 or #70 on a list? Rankings are a way to simplify lots of data into easily understandable terms. They also appeal to an inher- ent competitive spirit. We encourage readers to look beyond a simple rank to understand the nuances of any individual school. Broadly speaking, groups of 25 might be considered roughly comparable. And remember, all 227 schools on our list are elite and high performing. The key is to find the one best for you.