Let’s Kill the GRE

Photo: Cpl. Jo Jones [Public domain], via Wikimedia Commons

The Background

Most of you are probably aware of the GRE or Graduate Record Examination. Those applying for graduate programs are required to report scores from this standardized test. The GRE, along with the resume/curriculum vitae illustrating depth and breadth of experience, GPA, letters of recommendation, and an essay are evaluated for acceptance into many graduate programs. Although the weight given to the applicant’s GRE varies among institutions, many schools and departments require the GRE and set minimum scores for application/acceptance. Typically top institutions and programs institute a high minimum GRE score to reduce the number of applicants and to ensure highly accomplished applicant pools. However, to what extent does the GRE predict success in graduate school? Does the GRE accurately measure an applicant’s ability to synthesize and apply knowledge, acquire and assimilate new information, or familiarity with mathematics, vocabulary, biology, history, chemistry, etc.?

Criticisms of the GRE

Critics of the GRE, myself included, feel that the GRE does not evaluate any of these but simply provides a metric for how well a person can master GRE test-taking procedures.  ETS, the nonprofit that administers the test, states “the tests are intended to measure a portion of the individual characteristics that are important for graduate study: reasoning skills, critical thinking, and the ability to communicate effectively in writing in the General Test, and discipline-specific content knowledge through the Subject Test” However, the Princeton Review guide states, “ETS has long claimed that one can not be coached to do better on its tests. If the GRE were indeed a test of intelligence, then that would be true. But the GRE is NOT [emphasis original] a measure of intelligence; it’s a test of how well you handle standardized tests.” Indeed, training for the test, either in a GRE course or with a book, involves more learning test-taking strategies rather than a focus on acquirement or application of specific knowledge, i.e. “cracking the system”. Before the days of computerized exams, more math questions were included on the test than the average mathematically literate undergraduate could solve in the designated time period. To prep for the math section of the GRE, I and others learned not geometry or algebra, or even logic, but rather methods, i.e. cheats or tricks, to quickly move through the question. Of course, these tricks provide little help when math is encountered in either academia or the real world.

Who evaluates and writes the questions on the GRE? The Princeton Review states “You might be surprised to learn that the GRE isn’t written by distinguished professors, renowned scholars, or graduate school admissions officers. For the most part, it’s written by ordinary ETS employees, sometimes with freelance help from local graduate students.” Unfortunately, I am not able to retrieve information about question writers online either from ETS or other sources, so there is little I can comment on. However, the lack of transparency and accessible information should give all of us pause.

I am from the old guard and took my GRE with a No. 2 pencil with which I spent hours incessantly filling little circles. Modern tests are computerized and use a computer-adaptive methodology. Simply, the difficulty of questions is automatically adjusted as the test taker correctly or incorrectly answers questions. A complicated formula is used based on the level of questions and how many you answered correctly to determine your “real GRE score”. There is a substantial criticism of the computer-adaptive methodology. One, if a test taker suddenly encounters an easy question mid-exam, they may deduce they have not been performing well thereby affecting their performance through the rest of the exam. Test takers may also be discouraged if relatively difficult questions are presented earlier on. Second, unequal weighting is given to questions, with earlier questions receiving more and setting precedent for later questions, biasing against test takers who become more comfortable as the test continues. ETS is aware of these issues. In 2006, they announced plans to radically redesign the test structure but later announced, “Plans for launching an entirely new test all at once were dropped, and ETS decided to introduce new question types and improvements gradually over time.”

The GRE, a financial burden for students.

The GRE is expensive to take, $160 per test for U.S citizens.  Most students will take it at least twice to try to get their score higher.  This encouraged by both academia and ETS.  If you buy GRE prep books, which is also encouraged, you could easily be out another $100.    At the minimum,  a student is putting out $420.

It is often encouraged that the prospective graduate student also takes a GRE prep course.  I will use Kaplan as an example, but there are others.

• In person course $1,299 or in person plus $1,699 (link)
• Life online $999 or live online plus $1,399 (link)
• Tutor $2,499 (link)

By the time you count in missed work hours for test taking and preparation, transportation to the test (mine was 1-hour from my college), and various other expenses, a student could easily be out a significant amount of money.   It is also an additional $27 to have your own GRE test scores sent to more than the 4 “free” schools that come with the test.

So is the GRE finding us the best students or simply the richest?

Educational Testing Services: a multinational operation, complete with for-profit subsidiaries

So who is ETS? They are a nonprofit 501(c)(3) created in 1947, located in Princeton, New Jersey. In an article from Business Week…

”Mention the Educational Testing Service, and most people think of the SAT, the Scholastic Aptitude Test that millions of high schoolers sweat over each year in hopes of lofty scores to help ease their way into colleges. But if ETS President Kurt M. Landgraf has his way, Americans will encounter the testing giant’s exams throughout grade school and right through their professional careers. Landgraf, a former DuPont executive brought to the organization in 2000 to give ETS a dose of business-world smarts, has a grand vision for the cerebral Princeton (N.J.) nonprofit. Worried that the backlash against college testing means a lackluster future for the SAT and other higher-ed ETS exams, Landgraf has been trying to diversify into two growth markets: tests and curriculum development for grade schools, where the federal No Child Left Behind Act has spurred national demand for testing, and the corporate market, where Landgraf sees potential growth in testing for managerial skills. By 2008, he hopes expansion in these two areas will more than double ETS’s $900 million anticipated 2004 revenue. “My job is to diversify ETS so we are no longer reliant on one or two major tests,” he says.

Does this sound like language applied to a nonprofit? From the New York Times…

It has quietly grown into a multinational operation, complete with for-profit subsidiaries, a reserve fund of $91 million, and revenue last year of $411 million… Its lush 360-acre property is dotted with low, tasteful brick buildings, tennis courts, a swimming pool, a private hotel and an impressive home where its president lives rent free… E.T.S. has come under fire not only for its failure to address increased incidents of cheating and fraud, but also for what its critics say is its transformation into a highly competitive business operation that is as much multinational monopoly as nonprofit institution, one capable of charging hefty fees, laying off workers and using sharp elbows in competing against rivals… ”E.T.S. is standing on the cusp of deciding whether it is an education institution or a commercial institution,” said Winton H. Manning, a former E.T.S. senior vice president who retired two years ago. ”I’m disappointed in the direction they have taken away from education and public service.”

In response to growing criticism of its monopoly, New York state passed the Educational Testing Act, a disclosure law which required ETS to make available certain test questions and graded answer sheets to students.

For all practical purposes, ETS has grown into a for-profit institution trading on its nonprofit status to create a monopoly (read the New York Times and Business Week articles for more alarming revelations). For example,

• As of the Spring of 2018, 24,141 graduate students were enrolled as graduate students in Louisana.
• Let’s assume that the applicant pool into Louisiana graduate programs was 2:1 (they are probably considerably more competitive), so 48,282 applicants took the GRE.
• At $160 per test (its actually $190 for non-US students), that generates $7,725,120 for just the current body of graduate students.
• If we assume the near 50,000 applicants represent the cumulative applicants over a typical 5 years Ph.D. program, i.e. represent multiple cohorts, then ETS generates conservatively $1,545,024 per year from Louisiana alone.

In 2016, ETS brought in $1,609,201,000 dollars in revenue.  You read that right. One. Point. Six. BILLION. Dollars.   Most of that revenue, over 97% comes directly from testing.    You can find the complete financials from 2016 backward at ProPublica.  ETS also markets through one of their subsidiaries, and for-profit, a test book at $33.89 (at Amazon). “We prepare the tests-let us help prepare you!”  It is unclear the total revenues these subsidiaries bring in.

The GRE does not predict success in graduate school

But what of the original question? To what extent does the GRE predict success in graduate school? A meta-analysis in 2001 by Kuncel et al. demonstrated that correlations between GRE scores and multiple metrics of graduate performance were low. Correlation with graduate GPA ranged from 0.34-0.36. With performance as evaluated by departmental faculty the correlation ranged from 0.35-0.42, time of degree completion ranged from -0.08-0.28, citation count ranged from 0.17-0.24, and degree attainment from 0.11 to 0.20. While encouraging these correlations are positive (in most cases) and even accounting that GRE is supposed to be evaluated in the context of other materials (but often are not), these correlations are not that impressive. Do we need the GRE scores to evaluate applicants? Interesting, the same study also demonstrated that undergraduate GPA performed equally well as the GRE.

Even [ETS], warns that there’s only a tenuous connection between test scores and success in graduate school. According to the ETS report, “Toward a Description of Successful Graduate Students,” “The limitations of graduate school admissions tests in the face of the complexity of the graduate education process have long been recognized.” The report acknowledges that critical skills associated with scholarly and professional competence aren’t measured by the GRE… Perhaps more alarmingly, as with the SAT, high GRE test scores time and again tend to correlate with a student’s socioeconomic status, race, and gender. Research dating back decades from the University of Florida, Stanford, New York University, the University of Missouri, and  ETS has shown that the GRE underpredicts the success of minority students…ETS studies have also concluded the GRE particularly underpredicts for women over 25, who represent more than half of female test-takers.  [link]

Death to the GRE

In recap, the GRE is a financial burden to students, poorly predicts graduate student success, and has biases associated with socioeconomic status, race, and gender with profits going to a $1 billion revenue “nonprofit” company.  It is abundantly clear, we need to rid academia of the GRE.  Thankfully, I am not the only one to believe this.  Two physics professors in op-ed in Nature (open-access) called for the same.

This is a call to acknowledge that the typical weight given to GRE scores in admissions is disproportionate. If we diminish reliance on GRE and instead augment current admissions practices with proven markers of achievement, such as grit and diligence, we will make our PhD programmes more inclusive and will more efficiently identify applicants with potential for long-term success as researchers.

As well as the President of the American Astronomical Society,

 It’s a matter of great concern to all of us: failing to draw from the full pool of talent weakens our profession. It’s vitally important to train leaders who will help our profession achieve true equity and inclusion, and thus the strongest possible astronomical community.

In a bold move shortly after, the American Astronomical Society adopted a policy to encourage departments to make the GRE optional or to stop using it.  What has followed is a wave of schools and departments dropping GREs (link, link).  Joshua Hall, Director of Admissions at UNC (on Twitter @jdhallphd) maintains a list of all biology and biomedical programs who have dropped GREs from the applicant process.  

Your next step? If your department, school, and university still use the GRE, it is time to call the faculty together and kill the GRE.

UPDATE 1: Follow the #grexit hashtag on Twitter