Rating the Rankings: University Ranking Processes and Methodologies

With the release of the US News and World Report College Rankings this morning, the annual college rankings season comes to a close. There is a wide range of different rankings, with schools at the top of one ranking looking mediocre in another.
[cough]Yale University
[cough]Reed College
[cough]Babson College
What do we make of these? In the spirit of the forthcoming (and, perhaps, always will be) federal Postsecondary Institution Ratings System (PIRS), I rate (but not rank!) five of the more prominent systems, and consider where UVU fits into the rankings.

US NEWS AND WORLD REPORT

US News and World Report's Best Colleges Ranking, generally the most widely cited postsecondary ranking system, aims to provide a consumer decision tool for choosing the “best” undergraduate educational programs. It uses 16 “indicators of academic excellence” collected from surveys of the schools or, if the school does not respond to the survey, from other public data sources. Only schools that use SAT or ACT scores in making admissions decisions for first-time, first-year, degree-seeking applicants are ranked; UVU is thus unranked in the US News data.
For the 2014 rankings (released in 2013), the indicators rank institutions on the basis of:

• Graduation and retention rates (30%). Emphasis is on the IPEDS six year graduation rate, which excludes transfer-in students and spring-entry students, and counts transfer-out students as dropouts. Rankings also consider actual graduation rate in comparison to a predicted rate calculated by US News based on spending and student characteristics.
• Undergraduate academic reputation (22.5%). Based on a “peer assessment survey” of “top academics – presidents, provosts and deans of admissions” and, in the case of national research universities and national liberal arts colleges, a survey of high school counselors.
• Faculty resources (20%). This is largely class size and faculty salary measures.
• Financial resources and alumni giving rate (15%). This includes spending per student on “instruction, research, student services and related educational expenditures. …Spending on sports, dorms and hospitals doesn't count.” Alumni giving rate is interpreted as a measure of student satisfaction, not financial condition.
• Student selectivity (12.5%). This is largely the SAT or ACT scores of enrolling students, but also includes high school class rank and selectivity (operationalized by acceptance rate, but only comprising a total of 1.25% of the score).

US News made no changes to the methodology for 2015. In the past, however, the US News rankings have been criticized for frequent methodological changes that have had the effect of keeping the top-ranked institutions relatively unchanged and consistent with general perceptions of the higher education landscape: Harvard, Princeton, and Yale are almost always the top three. US News made a major change in its methodology in 1999 which was praised for its statistical rigor but resulted in the California Institute of Technology being ranked first. In 2000 the methodology reverted to its previous approach, and the statistician responsible for the 1999 methodology left US News. There have also been a significant number of instances where institutions have falsified self-reported data to improve their rank.
The US News rankings should be seen as resting primarily on institutional reputation and prestige. US News offers no justification for the selection or weighting of these variables beyond “our researched view of what matters in education.” Considering that student selectivity is driven largely by reputation (more prestigious institutions will attract more competitive students with higher SAT/ACT scores), and graduation and retention is affected by it (students are less likely to transfer out of more prestigious institutions), at least 35% of the rankings are reputationally driven.
UVU has generally not responded to the survey in recent years as its admissions policy would not lead it to be ranked in any case. However, last year UVU's Institutional Research & Information office noted that the data listed in its entry in the US News system was wildly inaccurate in many cases (for instance, listing former President Sederburg as the current president). UVU is now participating in the survey, but is still not ranked because of its open admissions policy.
Overall, I consider these rankings part of the problem in higher education rather than the solution. They are driven by reputation alone, and so are a wonderfully self-fulfilling prophesy. That said, the prestige hierarchy is probably one of the most significant effects of higher education, and that doesn't depend on robust pedagogy. Grade: D

WASHINGTON MONTHLY

Washington Monthly's College Guide is explicit about their rankings philosophy, approaching rankings with the aim of identifying “the most public-minded institutions”:

Our rankings have always rejected the idea that expense, luxury, and exclusivity should be held up as the highest values for colleges and students to aspire to. Instead, we ask a different question: What are colleges doing for the country? Higher education, after all, doesn’t just affect students. We all benefit when colleges produce groundbreaking research that drives economic growth, when they put students from lower-income families on the path to a better life, and when they shape the character of future leaders. And we all pay for it, through hundreds of billions of dollars in government-financed financial aid, tax breaks, and other spending.

The Washington Monthly rankings are based on three broad categories:

• Social Mobility (recruiting and graduating low-income students)
• Research (producing cutting-edge scholarship and PhDs)
• Service (encouraging students to give something back to their country).

Each category, and the specific measures within them, are weighted equally, with measures statistically standardized to ensure rankings that reflect broad success within the categories rather than success in any one specific measure. This prevents institutions from gaming the rankings by artificially inflating specific measures (e.g., lowering thresholds for applications in order to encourage unqualified applicants, thereby increasing selectivity). The specific measures in each category vary somewhat across institutional types. All data is collected from external sources rather than the institutions themselves.
UVU is ranked 111th of 346 baccalaureate colleges in the 2014 rankings. We only slightly underperform on graduation rate, with a predicted rate of 29% and an actual rate of 25%, ranking 71st in graduation performance. Several aspects of the data may be reducing UVU’s ranking. Our historical problem of low FAFSA completion rates makes the percentage of Pell Grant recipients a low estimate of the extent to which UVU serves low-income students. UVU did not report any research spending to the Center for Measuring University Performance and the National Science Foundation. Community service information is taken from applications for the President’s Higher Education Community Service Honor Roll, in which UVU does not participate.
The Washington Monthly rankings have the virtue of being a reasonably good ranking of institutions according to the goals of the system. Unfortunately, those goals don't make them very useful for students and their families. Grade: B-

MONEY MAGAZINE

Money's rankings of The Best Colleges for Your Money are oriented primarily toward the return on investment of the degree: “Our goal: to give students and parents a much better indication of which colleges will provide real value for their tuition dollars and enhance a student’s earning potential.”
Money ranks institutions in a limited universe: those with above average six-year IPEDS graduation rates or rates at least 25% above the predicted rate and that did not have a speculative Moody’s bond rating; UVU was thus not ranked. Approximately 20% of the ratings are in the form of value-added measures that adjust for “the economic and academic profile of each school’s student body.” Data was collected by a consulting firm, College Measures.
Schools are rated in three equally weighted categories:

• Quality of Education, which includes graduation rate (50%), admissions (35%), and “instructor quality” (15%) ratings. The latter is composed of student to faculty ratio and “the college’s average grade from Ratemyprofessor.com,” making it an exceptionally dubious measure.
• Affordability, which is composed of a net price calculation performed by the magazine rather than as reported to IPEDS (40%) and various debt measures (60%).
• Outcomes, which are almost entirely measures of post-graduation earnings (90%). This data is collected the PayScale, and includes only graduates who have not pursued a graduate degree.

Forbes offers a similar set of rankings that gives somewhat more consideration to student satisfaction, though it measures that with a combination of Ratemyprofessor.com scores and retention rates. The combined net price, debt, and earnings measures account for 57.5% of the score. UVU was ranked 556th of 650 institutions.
In both cases, rankings are likely biased quite strongly toward institutions with relatively affluent student bodies. Overall, 20% of the Money score is based on debt and 30% is income. Both are influenced at least as strongly by family income as by institutional factors. While value-added measures that take into account the expected debt and income of the institution’s students, such measures are in all cases weighted less strongly than unadjusted measures.

Update, Sept. 10: Libby Nelson at Vox provided an analysis of PayScale's salary measures, which are used in both Money and Forbes. She makes a good case that they run high to begin with, and that early career rankings are not good indicators of long-term success.

The Money rankings make me want to pick up my hammer and sickle. Largely they are measures of the affluence of the students coming in rather than of the institutions themselves, and they perpetuate the myth that everyone has the same opportunities in contemporary American society. Grade: a gentleman's C on the "Only A students go to Stanford" grading scale. Which means an F.

NEW YORK TIMES

The Times list of The Most Economically Diverse Top Colleges uses a more specific ranking system that would address only the economic diversity of “top” colleges and universities. The aim is to identify “colleges that have changed policies and made compromises elsewhere to recruit the kind of talented poor students who have traditionally excelled in high school but not gone to top colleges”—often referred to as “undermatching.” It is explicitly not a general purpose ranking of institutions.
The Times ranking includes only institutions with a graduation rate of at least 75% in 2011-12; the Times refers to these as “selective” colleges but they are better understood as elite institutions. Only three public institutions are ranked: the University of North Carolina, the University of Virginia, and the College of William and Mary. The average enrollment of the ranked institutions is less than 5,000, and average Pell eligibility is 15.4% of the student body.
The Times ranks institutions using only two metrics:

• Average percentage of students who received Pell grants over the 2011-12 through 2013-14 academic years (50%)
• 2011-12 net price for students whose family income was between $30,000 and $48,000 (50%)

The standardized values for the two measures are summed to arrive at the Times’ “College Access Index” according to which institutions are ranked. Data for both metrics is available through IPEDS, but the Times is not specific about the sources beyond noting them as “Individual colleges; [and the] Department of Education.”
The Times rankings, released September 8, were immediately criticized by many higher education researchers. Some (OK, me, by it got retweeted a good bit) referred to the rankings as a measure of noblesse oblige. It should also be noted that that cost is a noteworthy but not primary factor driving undermatching; campus environment, support from family, and especially location tend to be more important considerations in college choices for low-income and minority students. As Sara Goldrick-Rab noted, the relationship between graduation rate, used as a population selection parameter, and income is itself an outcome of interest to low and middle income students.
UVU was not included in the New York Times rankings due to its graduation rate.
While I'll give them some credit for respectable stats that measure what they claim to measure—which top schools are best at promoting economic diversity—that's like measuring which endangered species makes the best fertilizer for your lawn. An average of 15.6% of students being Pell eligible is hardly economic diversity when nearly half of the students in the Cal State system are Pell eligible. Grade: C-

POSTSECONDARY INSTITUTION RATINGS SYSTEM

The proposed federal Postsecondary Institution Ratings System (PIRS) is part of the Department of Education’s larger plan to enhance college affordability through accountability. According to the Department, the ratings system will address metrics in three areas:

• Access, such as percentage of students receiving Pell grants
• Affordability, such as net price and loan debt
• Outcomes, such as graduation and transfer rates, earnings of graduates, and completion of advanced degrees

Specific goals beyond this, and specific metrics, and not defined at this time. The National Center for Education Statistics solicited preliminary public input last December. In a public symposium on September 2, however, Deputy Under Secretary of Education Jamienne S. Studley noted that her “own test of success” was the extent to which the ratings reward institutions that outperform expectations for graduation of disadvantaged students. She also stated that at least two ratings were possible: one to be used for accountability that would meet the Obama administration’s plan to tie ratings to financial aid eligibility, and another to provide more useful consumer information to guide college choice. There is discussion of mission-specific ratings (subscription required) as well, but no firm commitments to it.
The Department expects PIRS to be in place by Fall 2015, and to be linked to financial aid by 2018 (however one should note that President Obama’s term ends in January 2017, which introduces significant ambiguity into the political viability of this aspect of the system).
We'll see if this ever comes in. Grade: I
[Edit, Sept. 10, 10:07 AM: Added a bunch of links that I forgot to include when I first posted.]