Introduction

The National Report Card is a benchmarking exercise, focusing on the 50 states in the USA as the units of analysis. The Report Card assesses with quantitative data the educational performance of each of the states on six dimensions: preparation for higher education, participation, completion, affordability, benefits, and learning. The focus is solely on undergraduate education, as no indicators are included for either graduate education or research. Aggregate data from all forms of postsecondary education are incorporated, including public 2-year and 4-year institutions, private non-profit institutions, and (where the data allow) private for-profit institutions. No data for individual colleges or universities are reported, however; the measures are collective state data. Multiple indicators are used for each of the six categories, with each indicator weighted by its importance, and combined into a single numerical measure for each category. The result is a set of performance measures that are used to compare each state to the best-performing state on each measure, a classic benchmarking technique. Grades are assigned to each measure (A through F), allowing the report to be used by state-level policymakers to judge how well a given state is performing relative to the other 49.

The Report Card is the product of the National Center for Public Policy and Higher Education, a non-profit, non-affiliated, and non-partisan private organization located in San Jose, California. Founded in 1998, the Center is fully supported by private foundation grants, with core support from the Pew Charitable Trusts, The Atlantic Philanthropies, and The Ford Foundation; no state or federal governmental funds are involved. In that sense, the Center has no official governmental status. It provides the biennial reports on state performance as a public service, and no state agency is obligated to respond to, or even read, the reports. It is assumed, however, that the quality of the work and the salience of the measures provide useful information to policymakers as they consider policies that support and govern higher education in each state. (Information on the extent to which the states have used the reports will be discussed later.) The National Center has an appointed board of directors, a small professional staff, and a number of advisory committees that guide its work.Footnote 1

Policy Problem

The National Center was created and supported to be an independent, non-partisan voice in the debates about higher education policy. The decision to create a National Report Card emerged from discussions that the leadership held around the country in 1998 and 1999, seeking advice and comments on how a small organization could speak to issues of higher education policy in all 50 states. It was noted in several of those meetings that report cards (or similar benchmarking techniques) have operated effectively in areas such as medical care and social welfare, and thus the first step was to determine whether a 50-state report card on higher education was feasible. It was decided early on that such a report card would have to be constructed from existing data sources, as the Center did not have sufficient funds or staff to undertake independent data collection. In 1999, a small advisory group met several times with Center staff to design and implement a pilot project, using 10 states and relying heavily on nationally collected data from such agencies as the National Center for Education Statistics and the Census Bureau. The pilot studies indicated that sufficient data sources existed to make the project a reality, and the first Report Card was issued in the year 2000. The central problem was to locate relevant and comparable data at the state level; many data sources report information on the national level, but the underlying surveys often are not large enough to provide adequate data at the state level.Footnote 2 The feasibility committee also had to consider such issues as the interstate migration of students, and the differences among states in the structures of their higher education systems, in particular the extent to which states differ in their reliance on private institutions and community colleges in achieving educational opportunities. The pilot study indicated that, while some of these problems would remain, a meaningful set of measures, as designed by the committee, could be assessed quantitatively at the state level, and the board of directors authorized the Center to undertake the Report Card as its central project. Reports have been issued subsequently in 2002, 2004, and 2006.

It should be noted that the National Center is not intended to continue in perpetuity; while no definite date has been determined when it will shut its doors, at some point that will happen. Whether another organization will decide to continue producing the Report Card is unknown at this time, although the developmental work has been completed and the Report Card is designed now for publication on a website, precluding the need to produce expensive, book-length products. If it has demonstrated its value, one hopes that another agency will pick up the project and continue Web-based publication in future years.

Content of the Policy Instrument

As noted earlier, the Report Card assesses six measures of educational performance – preparation, participation, completion, affordability, benefits, and learning. Each measure is made up of several underlying quantitative indicators that are weighted and aggregated into a score for each state on each measure. In identifying indicators for each measure, the Center was limited to those data that are available at the state level, which constrained the choices one might have made in an ideal world. Nonetheless, a good number of reasonable and relevant indicators were found, and while additional data could improve the quality of the Report Card (and some of those data elements will be discussed subsequently), the result has clearly passed the test of face validity and plausibility within the policy community.

Preparation. The indicators that make up the measure of preparation for postsecondary education (and the weights in parentheses) include:

  • High School Completion (20%)

  • K-12 Course Taking (35%)

    • 9th to 12th graders taking at least one upper-level math course

    • 9th to 12th graders taking at least one upper-level science course

    • 8th grade students taking algebra

    • 12th graders taking at least one upper-level math course

  • K-12 Student Achievement (35%)

    • 8th graders scoring at or above “proficient” on the national assessment exam in math, in reading, in science, and in writing

    • Low-income 8th graders scoring at or above “proficient” on the national assessment exam in math

    • Number of scores in the top 20% nationally on SAT/ACT college entrance exam per 1,000 high school graduates

    • Number of scores that are 3 or higher on an Advanced Placement subject test per 1,000 high school juniors and seniors

  • Teacher Quality (10%)

    • 7th to 12th graders taught by teachers with a major in their subject

Participation. The indicators that make up the measure of participation in postsecondary education (and the weights in parentheses) include:

  • Young Adults (60%)

    • Chance for college by age 19

    • 18- to 24-year-olds enrolled in college

  • Working-Age Adults (40%)

    • 25- to 49-year-olds enrolled part-time in any type of postsecondary education

Completion. The indicators that make up the measure of completion of postsecondary education (and the weights in parentheses) include:

  • Persistence (20%)

    • 1st-year community college students returning their second year

    • Freshmen at 4-year colleges/universities returning their sophomore year

  • Completion (80%)

    • First-time, full-time students completing a bachelor’s degree within 6 years of college entrance

    • Certificates, degrees, and diplomas awarded at all colleges and universities per 100 undergraduate students

Affordability. The indicators that make up the measure of affordability for postsecondary education (and the weights in parentheses) include:

  • Family Ability to Pay (50%)

    • Percentage of income (average of all income groups) needed to pay for college expenses minus financial aid at community colleges, at public 4-year colleges and universities, and at private 4-year colleges and universities

  • Strategies for Affordability (40%)

    • State investment in need-based financial aid as compared to the federal investment

    • At lowest-priced colleges, the share of income that the poorest families need to pay for tuition

  • Reliance on Loans (10%)

    • Average loan amount that undergraduate students borrow each year

Benefits. The indicators that make up the measure of benefits from postsecondary education (and the weights in parentheses) include:

  • Educational Achievement (37.5%)

    • Population aged 25–65 with a bachelor’s degree or higher

  • Economic Benefits (31.25%)

    • Increase in total personal income as a result of the percentage of the population holding a bachelor’s degree

    • Increase in total personal income as a result of the percentage of the population with some college degree (including an associate’s degree), but not a bachelor’s degree

  • Civic Benefits (31.25%)

    • Residents voting in national elections

    • Of those who itemize on federal income taxes, the percentage declaring charitable gifts

    • Increase in volunteering rate as a result of college education

  • Adult Skills (0%)Footnote 3

    • Adults demonstrating high-level literacy skills, quantitative, prose, and document

Learning. This category was graded as incomplete in both 2000 and 2002, as the country has no well-defined indicators at the state level that measure college-level learning. With support from the Pew Charitable Trusts, staff of the National Center and outside consultants began exploring this topic to determine whether some collection of existing tests and new instruments might be assembled to provide indicators for the learning category. A National Forum on College-Level Learning was organized under the auspices of the National Center in November 2001, and a broad group of participants, including educators, business people, and state and federal policymakers, reviewed the initial work and agreed that the topic was sufficiently important that it should be pursued to a pilot phase. That work was accomplished, and the Report Card for 2004 did contain results from the pilot effort in five states – Illinois, Kentucky, Nevada, Oklahoma, and South Carolina.

Essentially, the pilot effort developed three broad types of indicators of learning – literacy levels of the state population, graduates ready for practice, and performance of the college educated. For the first indicator, the Center used the 1992 National Adult Literacy Survey (NALS) for residents ages 25–64, updated using the 2000 census, and weighted at 25%. The second indicator involved the use of various licensure examinations, competitive admissions tests, such as the Graduate Records Examination and the Medical School Admissions Test, and measures of teacher preparation. Finally, for the third indicator, the WorkKeys assessment administered by the American College Testing Service was used for 2-year institutions, and the Collegiate Learning Assessment (CLA) developed by an offshoot of the Rand Corporation, was used for graduates of 4-year colleges. For both WorkKeys and CLA, the Center engaged a sample of colleges and universities in the five states and arranged for the tests to be administered to a representative sample of students. Given that the Center’s general operating policy is not to collect original data itself, future efforts to measure learning will have to be done by the states themselves, but the Center has demonstrated the way forward. A 2005 publication of the National Center, Measuring Up on College-Level Learning, by Margaret A. Miller and Peter T. Ewell, provides considerable information on the results and issues confronted in the five pilot states and will be a valuable resource for anyone seeking further information in this area.

Implementation

As noted, the Center has now had experience producing and disseminating three Report Cards, for the years 2000, 2002, 2004, and 2006. Each time one has been published, a new set of issues and complications have been encountered. Most of the issues have surrounded the fact that the first Report Card was a one-time snap-shot, while each successive version opens up the potential for longitudinal analyses, i.e., comparisons of 2002–2000 within the same state, rather than simply comparing states against each other. Any group undertaking a similar exercise would encounter the same issues, but it is fair to say that many of the issues raised by subsequent reports were not originally foreseen. The result has been a series of hard decisions that have been made as each new project is underway.

For the first Report Card, the key tasks were assembling the data, dealing with missing observations, and refining the aggregation techniques. Each indicator was tabulated, and the weights that had been determined by consensus and best judgment were then applied to each indicator – as indicated earlier, the weights sum to 100%.Footnote 4 State results on each indicator were then converted to a scale from 0 to 100, a statistical method that allows for accurate comparisons of different measures. The top five states on each indicator were seen as high, but achievable, measures of performance. In practice, the median of the top five was assigned a score of 100, meaning that potential outliers were eliminated. Finally, each state’s score for each category was calculated using the index score on the indicator’s and the indicator weights. Once again, the raw category scores are scaled on a 0–100 basis, and grades were assigned to each state in each category using the standard A through F scale common to public schools. When the exercise was finished, each state received five-letter grades (A though F) and an incomplete in the category of learning.

This technique differs from ventures of a similar sort in which absolute standards of performance are determined abstractly, and each entity is measured against that standard. In the Report Card, the highest grades are not determined abstractly, but rather represent actual performance delivered by each state. The benchmarking, therefore, is against best practice rather than against a standard that no state may have achieved. The creators of the Report Card believe that this method removes the objection that arbitrary standards have been applied.

Having produced the first Report Card, entitled Measuring Up 2000, the next task was to publicize and explain it to the higher education community, the policy community, and the media. Unlike many private research and policy organizations, the National Center has devoted considerable resources to outreach and public relations. The president has met with numerous editorial boards of newspapers, a professional public relations firm worked on the press releases, with one tailored explicitly to each state, and the C-Span television network covered the press release of the first Report Card, an event held at the National Press Club in Washington, D.C. Several members of the board of directors were present, as were key staff and consultants. The effort was rewarded with wide coverage in the press, with the central message being that opportunities for higher education vary widely among the states. In other words, one’s chances for higher education depend to a disturbing degree on the state in which one happens to live. This message was portrayed vividly with colored maps identifying the high- and low-scoring states on each measure.

A fascinating aspect of the launch was the reaction of the higher education community, particularly as represented by the national associations of colleges and universities. Before the release, there was considerable nervousness about the Report Card, reflecting a fear that the institutions of higher education would be under attack and poorly graded. It took some time for people to realize that no institution was named, and that indeed, the focus was on state performance, not institutional performance. Once that realization sank in, most college and university presidents simply turned their backs on the report, seeing it as not doing much to help them, as institutions were not identified. In a few instances (Georgia and New York most prominently) there was strong criticism of the grades, particularly the low grades received in the category of affordability. Georgia higher education officials were upset because the method of calculating affordability gave minimal credit to the HOPE scholarship program, a merit-based scholarship that does little to enhance affordability for low-income students. New York officials argued that their state student aid program was not recognized sufficiently by the Report Card methodology; once again, the issue has to do with how those funds are distributed and the impact they make in reducing net cost relative to family income. But beyond some of these debates, the general response of leaders of higher education was to ignore the report, not seeing how they could use it to their advantage.

An important point emerged from reaction to the first Report Card. At least since World War II, most of the policy debate about higher education in the states has focused on the institutions and how well (or how poorly) they are supported, how many new colleges were needed, where they should be built, their missions, and so forth. The Report Card takes a radically different approach, focusing instead on the citizens of the state and the opportunities they have (or do not have) for higher education. This is not the policy discussion with which most higher education leaders are familiar, and the de-centering of the institution from the heart of the conversation was a blow. Indeed, the National Center has taken the unusual stance of focusing its efforts on state policymakers rather than on college and university presidents, and only time will tell whether this effort to change the nature of the state higher education policy debate takes root. Early evidence suggests that this new direction is finding a ready audience in several states.

Having produced a successful first Report Card, the Center began work on the 2002 report and realized as the data were being assembled that now one could not only compare the states to each other in a given year, but one could also determine whether performance in each state in 2002 was improved or declining relative to 2000. This observation posed a communications challenge, as the designers argued that, on the one hand, it was important to replicate the basic design of the first Report Card for comparability purposes, but also that change over time within a state was at least as significant a measure as interstate comparisons. Furthermore, some of the measures (particularly affordability) were objectively getting worse, but the benchmarking approach still required that best practice states be given an A grade. Thus, a state could receive a top grade (even a better grade than in 2000) while actually performing less well. The metaphor that eventually was adopted was that of a race – individual runners might be doing better in the race in 2002 than in 2000, but if other runners outstripped them, their relative standing would decline, and it was the relative standing that determined the letter grade. The second Report Card, Measuring Up 2002, made a valiant effort to communicate this complicated situation, but the designers realized that they had a substantive problem to solve. If grades were to be used, then a move by a state from a C to a B should mean objective improvement over time, rather than just relative gain (or even loss). The grades were in danger of not revealing directly and simply the information they were intended to convey. The solution came in the third Report Card, Measuring Up 2004.

In the third version, the designers extended the measures back 10 years, to give each state a baseline measure in 1992. That allowed each state to be judged in 2004 not only in comparison to other states in that year, but over a decade against their own earlier performance. The Center published short reports for each state that provided both the current results and the 10-year change. Comments received from the field indicate that this blend of current comparison together with time-trend data provides the most useful information to date. As subsequent editions of the Report are produced, new challenges will undoubtedly arise, but the designers by necessity are becoming adept at finding creative solutions.

Over time other issues have arisen, generally involving data problems, as a survey question may be changed from one year to the next, or a data element may be dropped, or updated surveys not provided. For example, the National Adult Literacy Survey is conducted every 10 years, and while the Center decided it could use older data on adult literacy in Measuring Up 2000 and 2002, by 2004 the data were so out of date that the indicator had to be dropped. Similar data problems bedeviled one part of the affordability measure, as availability of data required to determine net cost after financial aid became a problem. In some cases, however, new indicators become available; an example would be the indicator on teacher quality, used for the first time in 2004.Footnote 5 On the one hand, it is important to improve the data underlying the graded categories; on the other hand, adding a new indicator means altering the weights and reduces comparability over time. The technical reports accompanying each Report Card give detailed information on how each problem was handled and can be found on the National Center’s website. Communicating changes without losing the reader in a shower of technicalities remains a constant challenge for this sort of exercise.

Impact

It is difficult to determine the impact of a project such as this one, as one can never trace precisely the changes in thinking that the Report Cards may have produced. One approach is to examine the media coverage that the Report Cards have achieved, for if the reports failed to gain substantial press coverage, that would be likely to reduce their impact. The National Center has carefully collected information on media coverage; for Measuring Up 2004, their statistics show that 2,030 newspaper articles covered the report, including 282 editorials, 38 op-ed pieces, and 34 columns across the country. Coverage was also excellent in the major papers, such as the New York Times, USA Today, the Los Angeles Times, the Washington Post, the Chicago Tribune, and the Dallas Morning News. At least 474 televisions news segments covered the report, including “CNN Headline News” and the “News Hour with Jim Lehrer.” Radio and internet coverage were also strong. Most significantly, this third report in the series garnered considerably more coverage than the previous two, indicating a growing familiarity with the report on the part of the media and the sense that the message is important.

Among the states, the response has varied from largely ignoring the reports to making active use of them. Several states, including New Mexico and Oklahoma, have borrowed the format and put out their own state-focused reports, often incorporating state data that are not available in all 50 states, but which give policymakers a better sense of how a given state is doing. One form of follow-up has been creation of the National Collaborative for Postsecondary Education – co-sponsored by the National Center, The National Center for Higher Education Management Systems (NCHEMS), and the Education Commission of the States (ECS). This entity has been the primary vehicle for state follow-up, supported by a grant from the Pew Charitable Trusts to the three organizations. The five states involved are Washington, Virginia, Rhode Island, Missouri, and West Virginia. In each of these states the NCHEMS “drill down” methodology was used, providing results at the county level on each of the Measuring Up categories. This effort was followed by “policy audit” discussions around the state with local education, business, and political leaders. Feedback from these local meetings was given to a statewide leadership group that each state had to form, including the governor, business, community, public/private 2- and 4-year college university leaders, and K-12 state leaders. This group then determined the priorities, based on the “drill down” exercise and policy audit, for policy focus and change. The Educational Collaborative Program ended in December 2005. Results have varied depending on state leadership – definitely more effective when the governor was involved significantly or chaired the leadership group, as in Virginia and Rhode Island.

In addition, the National Center has worked directly with many more states on one or more of these issues (Kentucky, South Carolina, Arizona, Oregon, Pennsylvania, and Oklahoma, to name a few) and is currently working in-depth in Minnesota through the governor’s office to address access and affordability issues.

As a further indication of impact, several national organizations, including the National Conference of State Legislators and the Committee for Economic Development, have devoted resources to producing guides to the Report Card, designed and published for their members. The respective reports are entitled The Legislator’s Guide to the National Report Card on Higher Education and Cracks in the Education Pipeline: A Business Leader’s Guide to Higher Education Reform.

A further strategic point about the Report Cards should be noted. No policy recommendations are included in the reports; the data are presented, the state comparisons are made, and the presumption is that the information should start conversations within each state regarding its relative performance and how it might improve. Indeed, the National Center sees its role primarily as moving the policy debates within states forward, while not prescribing any particular set of policies to be adopted. For example, in the first Report Card, two of the states that received A grades in affordability were Illinois and North Carolina. In the case of Illinois, the policy tool was a well-funded program of need-based student financial aid; in North Carolina, the policy tool was relatively low tuition. The Center’s report simply demonstrated that there is more than one way to achieve affordability, and circumstances in each state may determine which approach works best in a given state context. In short, the National Center’s efforts will be seen as successful if the states investigate and work on the issues measured by the Report Card, and not on the choice of a single set of policy options.

Costs

Over the 2-year development of Measuring Up, the National Center spent about $1.5 million, all costs calculated, including staff and consultant time. After that, for each of the subsequent editions the Center spent about $1 million. Core funders have included the Pew Charitable Trusts, The Ford Foundation, and Atlantic Philanthropies. In addition, The John S. and James L. Knight Foundation, The Carnegie Corporation of New York, The John D. and Catherine T. MacArthur Foundation, the William R. Kenan, Jr. Charitable Trust, and the Andrew W. Mellon Foundation provided support for specific aspects of the project.

Comparison

The closest reports to those of the National Center are those prepared by the OECD, entitled Education at a Glance (OECD 2004). These periodic publications report on educational trends in the OECD countries, although the apparatus of benchmarking to best practice and providing letter grades is not used in those reports. Individual OECD countries, however, such as Ireland and the Netherlands, have undertaken benchmarking exercises that rely on comparisons with “peer” countries. The National Center has commissioned Dr. Alan Wagner, an economist who worked at OECD for many years, to prepare a paper indicating how the Center’s Report Card measures could be integrated with the OECD data to generate comparative international measures of higher education performance.

Wagner notes that several OECD countries are roughly the same size as individual states within the USA, suggesting that comparisons of countries with states may have some value. He points out, however, that countries such as France, Germany, Italy, Poland, and the UK are larger, by an order of magnitude, than California, New York, and Texas (Wagner 2005). Other large OECD countries are Mexico, Canada, Japan, and Korea. As Wagner’s work progresses to publication stage, it will be possible to learn more about how the Measuring Up reports can be used effectively to enhance international comparisons of educational performance.

A second project worth noting is the publication of the Educational Policy Institute, a private, non-profit organization with offices in the U.SA, Canada, and Australia, entitled Global Higher Education Rankings: Affordability and Accessibility in Comparative Perspective (Usher and Cervevan 2005). This report builds on measures of tuition and fees, student maintenance costs, financial aid and public subsidies (including tax expenditures), GDP per capita, and relevant participation and population statistics for several countries. EPI apparently expects to continue this publication, which reflects the growing interest in these types of measures.