SAT SCORES, COLLEGE, AND DATA MINING
By Ed Colet
A student's success in college is important at the individual level, and at the national level, it is important that minority students also be able to take advantage of the opportunities for further education. In this column I discuss some interesting work underway at the Educational Testing Service (ETS) to identify and to facilitate a college's selection of talented minority students. Given the noble intentions, I also discuss how a data mining approach may be more useful in matching students with colleges.
An article in the Wall Street Journal (8/31/1999, "New weights can alter SAT scores", page B1) describes an interesting project underway at ETS that is designed to better identify talented (especially minority) students on the basis of their SAT scores. A "strivers score" is a measure derived from a student's actual SAT score. If the difference between the student's actual score and an expected score is greater than 200 points, then the student is identified as a striver. The expected score is a statistical formula that incorporates 14 factors some of which are aspects about family, language, age, academics, school location, student body, mother's employment status, and race. ETS or a college that the student has applied to will calculate this striver's score but it is not intended to be reported to the student.
The rationale for this measure is apparently driven by deficiencies in using plain SAT test scores to evaluate students for admission. It is also intended to provide colleges with a better way to justify their selection of minority students. It has long been recognized that background intangibles are important in test performance: "A 1200 SAT score from a student in Beverly Hills means something totally different from a 1200 from a student in a school in South Central Los Angeles ..." quotes a college official. Thus there is a need to provide a background context with which to interpret a student's score in evaluating their application.
The use of a statistical formula to compute an expected test score does two things. One, is it puts some of the intangibles affecting test performance into a quantifiable context. Second, the difference between actual and predicted scores identifies high striving students. Presumably this can help admissions directors interpret student performance and potential.
Needless to say, the ETS efforts are sure to be controversial because of the way the sensitive issues of race and ethnicity are handled. The effect of the strivers score would be to help Black and Hispanic minorities because the factors in the model that are associated with these minority groups will tend to drive down the value of an expected score (e.g. inner city school location, English as a second language, etc). Therefore, it is "easier" for a minority student to become identified as a striver.
Based on past data, for a given group of identified strivers, if race is included as a predicting factor, then a greater percentage of this group are Black or Hispanic than if race is not considered. For Asians and Whites the pattern is reversed: i.e. if race is used in the predicted score, a smaller percentage of students identified as strivers were Asian or White. Thus, using race and ethnicity to identify strivers will have a distinctly favorable effect on Black and Hispanic students.
The approach identifies high achieving minority students and thus is well intentioned, but may also be controversial because of the way scores are computed. Unless one has reason to suspect otherwise, one should expect that there should be an equivalent proportion of students from each ethnicity that are likely to be identified as strivers - it would be prejudicial to think otherwise. But by defining a striver using a fixed threshold of 200 points above an expected score, this is not likely to be the case. A hypothetical example: if an ethnic subgroup has an expected score of 1500 by virtue of extremely favorable background circumstances then by definition no one in this subgroup can be a striver given that the maximum SAT score is 1600. If instead, a striver should possibly be defined as a student scoring more than 2 standard deviations above their expected subgroup mean, then they can be readily identified.
The ideal situation is to ensure that administered SAT tests are completely unbiased. This in fact had been the intent of the SAT - to have a single number that admissions officers could use to fairly compare students with each other. Relying solely upon high school grade point averages was problematic because different high schools and even different teachers had different standards for grades, making it impossible for admissions officers to compare students fairly. The SAT was also designed to measure "aptitude" - and therefore wouldn't contain questions that required prior knowledge, but tapped into some latent (innate?) academic ability. Alas, it appears that the goal of developing an unbiased test that taps latent abilities regardless of background seems to be unrealistic, hence the need for adjustments and derivations to the score.
A practical alternative is to utilize a data mining approach. The striver score is clearly designed to identify outliers and in turn, it is implicitly assumed that these strivers will succeed in college. But the same student/striver may do well in one college and not another. Finding the right match can be difficult, but is important for all concerned (student, college, parents). Data mining software can readily identify outliers, and possibly also the factors underlying a student's likelihood for success in college. Because there is a lot of information about a student's background and a tremendous information about a particular college's characteristics (size, distance from home, climate, tuition costs, proportion of students in fraternities/sororities, national ranking, etc. etc.), then it should be possible to use data mining to predict whether or not the student and the college represent a good match and one in which all parties will be satisfied with the experience.
Ed Colet is the Acting Director of Research at Virtual Gold Inc., responsible for developing analytical methods for data mining and for investigating human factors and usability issues of business intelligence systems. At present, he is in the final stage of completing a doctoral dissertation in the Cognition and Perception program at New York University's Department of Psychology. Ed has also worked for IBM Research at the T.J. Watson Research Center. At IBM, Ed was a member of the group that developed Advanced Scout, the data mining application for NBA teams. His research interests focus on statistical methods and human factors.
For more information, see http://www.virtualgold.com.