1 1 2 UNITED STATES DISTRICT COURT EASTERN DISTRICT OF MICHIGAN 3 SOUTHERN DIVISION 4 5 BARBARA GRUTTER, For herself and all others 6 Similarly situated, 7 Plaintiff, 8 v. Civil Action 97-CV-75928 9 LEE BOLLINGER, JEFFREY LEHMAN, DENNIS SHIELDS, and REGENTS OF 10 THE UNIVERSITY OF MICHIGAN, 11 Defendants. _________________________________________/ 12 13 BENCH TRIAL - VOLUME 8 14 TUESDAY, FEBRUARY 6TH, 2001 15 16 BEFORE THE HONORABLE BERNARD FRIEDMAN United States District Judge 17 Theodore Levin United States Courthouse 231 West Lafayette Boulevard, Room 238 18 Detroit, Michigan 19 - - - 20 Appearances: 21 Kirk O. Kolbo, Esq., 22 R. Lawrence Purdy, Esq., 23 On behalf of the Plaintiff, 24 25 APPEARANCES (CONTINUED): John Payton, Esq., Craig Goldblatt, Esq., On behalf of the Defendants Bollinger, et al, - - - George B. Washington, Esq. Miranda K. S. Massie, Esq. On behalf of Intervening Defendants. Joan L. Morgan, Official Court Reporter Proceedings recorded by mechanical stenography. Transcript produced by computer-aided transcription. I N D E X - - - WINTESS: PAGE: MOTIONS 4 MARTIN SHAPIRO Direct Examination by Ms. Massie 17 Cross-Examination by Mr. Purdy 57 JAY ROSNER Direct Examination (Cont) by Ms. Massie 89 Cross-Examination by Mr. Payton 127 Cross-Examination by Mr. Kolbo 133 Redirect Examination by Ms. Massie 153 CONCEPCION ESCOBAR Direct Examination by Ms. Masley 158 E X H I B I T S RECEIVED Trial Exhibit Number 170 25 GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 4 1 Detroit, Michigan 2 Tuesday, February 6th, 2001 3 9:00 a.m. 4 - - - 5 THE COURT: How is everybody? Relaxed and ready 6 to go. I can see you weren't too relaxed, as I've been 7 reading since 6:00 o'clock this morning, and last night also. 8 Let's start I guess with the Professor Steele matter. 9 Let's get that resolved and go from there. 10 Good morning. 11 MS. MASSIE: Good morning, Judge Friedman. 12 Judge, here's our view of the situation with 13 Professor Steele. Professor Steele's testimony is absolutely 14 essential to our ability to make our case to you about 15 standardized test scores, and academic performance. It's 16 always been essential to us. 17 In fact, when we first filed the motion to 18 intervene, the day -- the day after that, or the day after 19 that, within a couple of days there were three people we 20 contacted about being expert witnesses. He was one of those 21 first three people. The other two were Gary Orfield whom you 22 have already heard from, and Walter Allen whom you'll be 23 hearing from tomorrow. He is at the heart of the questions 24 that you set for trial, and the defendants pointed out in 25 their response when the plaintiffs were trying to get GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 5 1 Professor Steele's testimony excluded there's nothing 2 duplicitous about his testimony. Nobody has said what he 3 would say if he were to testify. And nobody can say. He is a 4 national expert of unparallel qualifications. 5 THE COURT: You didn't take his deposition? 6 MS. MASSIE: His deposition was taken before the 7 Sixth Circuit granting intervention, and we didn't think there 8 was going to be any need to take his deposition because until 9 it was announced on the 19th or that Friday that Professor 10 Steele was not going to be called even though he was on the 11 will-call list, we fully expected to be able to ask him some 12 questions here before you. 13 THE COURT: Okay. 14 MS. MASSIE: In short, Judge, we would have listed 15 him as a witness if the University had not, and under local 16 rules we're entitled to call him. The only question is where 17 that's by deposition or whether you have a right -- whether we 18 can make a video tape and bring it back here. We shut 19 everything down, go make a video tape deposition, de bene esse 20 deposition, and bring it back here and play it for you or 21 whether you have the ability to ask him questions about 22 whatever's on your mind and your concerns. 23 That, Judge, is the only question that we face here. 24 MR. PAYTON: Your Honor, I guess I just want to 25 describe how we got to where we are, then where we are. We GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 6 1 retain Professor Steele almost three years ago, 1998, to be an 2 expert in this case. We have used his expert report in 3 connection with all the summary judgment motions including the 4 last round. We still use his summary judgment expert report. 5 I think it's very important. That's why we put it in. We did 6 place him on our original witness list from whenever it was a 7 couple of years ago, and on our current witness list, and on 8 the witness list that we prepared after the summary judgment. 9 But as things progressed we decided that it was unnecessary 10 for us to call him. We let everybody know on the 19th which 11 is the end of the first week of trial. On the 24h, which was 12 the last day of the second week of trial, Ms. Massie told me 13 she intended to subpoena him for a video deposition. I wrote 14 to her about that, and said that a video deposition in last 15 week would have been quite disruptive to our efforts to 16 prepare for this trial and that in any event discovery was 17 closed. 18 She then -- we talked. She then dropped the video 19 deposition, and on February 1st, wrote me and attached a 20 subpoena for a trial presentation. We got a trial subpoena on 21 February 1st, last Thursday. Last Friday, February 2nd, I 22 wrote her and attached a motion to quash the trial subpoena. 23 I had in response to several queries from her about several of 24 our witnesses let the intervenors talk to a number of our 25 witnesses because they wanted to use them in this case. So GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 7 1 Professor Foner, Professor Sugrue, Professor Lempert, and 2 Professor Steele, we let them contact. They contacted 3 Professor Steele. After they contacted Professor Steele, I 4 spoke with Professor Steele. He -- we now represent him in 5 this last matter, and after talking to him, and after him 6 talking to the intervenors, he has decided that he doesn't 7 want to be a witness. That's why we filed a motion to quash. 8 We represent him. 9 The subpoena that was issued, a trial subpoena 10 simply can't reach him for him to testify in this trial. They 11 started out with a video deposition, they dropped that. They 12 issued a trial subpoena. I think the trial subpoena is 13 clearly is not valid being issued from the Northern District 14 of California at all. They can say that they checked the 15 wrong box. They checked the box that said, trial -- you know 16 to appear at trial. So I think that's where we are. You 17 know, you can't reach him. 18 I have an additional report to make to everyone. I 19 have -- this has satellite litigation in the Northern District 20 of California where we filed the motion to quash. That was 21 filed on Friday in opposition and a reply filed yesterday. I 22 talked to the judge to whom that was assigned, a law clerk. 23 The judge to whom it's assigned is Judge Illston, and the law 24 clerk told me last night that Judge Illston who has the motion 25 to quash is a little in the dark about what just is being GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 8 1 argued and where and what is going on and what is the context 2 here. And, therefore, is waiting to see what this court does. 3 Was not -- thought there was also a motion to quash in this 4 court, thought you had the identical motion. And asked that I 5 tell everyone that until she sorts just what is where and what 6 this court has done with whatever court has she's just 7 standing back. But I think where things are, it's a trial 8 subpoena, you can't -- it's not valid -- 9 MS. MASSIE: Judge Friedman, could I be heard 10 briefly? 11 THE COURT: You sure can, but it's going to be one 12 second. I want to see what the plaintiff's position is. 13 You'll have the last word. 14 MR. PURDY: Your Honor, we take no position in the 15 battle between the university and the intervenors on this. 16 But I do want to raise one issue. We are concerned about the 17 effort to conduct a trial testimony by video -- 18 tela-conferencing unless -- in other words, we would object to 19 that unless the intervenors assured that all of the trial 20 exhibits and all the deposition and the exhibits to Doctor 21 Steele's deposition were made available. I have no idea 22 whether they have made any arrangements to do that in the 23 event they were to get leave to call him via 24 tela-conferencing. But that would pose a problem if he didn't 25 have those materials. I just want to raise a practical issue. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 9 1 But as far as the dispute between the university and the 2 intervenors, we take no position. Obviously, our position, 3 our feeling is that his testimony simply isn't relevant. 4 We're not taking an issue about his expertise or anything of 5 that nature. So that's all. 6 MS. MASSIE: And we will make arrangements as to the 7 practicalities that Mr. Purdy is raising if the Court allows 8 us to go forward. 9 Just very briefly, Judge Friedman, essentially the 10 University waived its objection to having the testimony taken 11 by video conference. And that's what the motion to quash 12 means in the Northern District of California is really about, 13 and thats' what this dispute is really about. 14 Mr. Payton told me that he had the same objection 15 to video deposition de bene esse to be introduced at trial, to 16 trial testimony, or deposition testimony presided over by the 17 Court by video conference, the same action to those two 18 things, as he did to Professor Steele appearing in person to 19 testify. He waived his objection to the video conference nex 20 of the testimony we're proposing. I'm not sure we checked the 21 wrong box. The Federal rules don't yet have provisions 22 specifically directed at video conferencing. That's 23 specifically why we raised it. We thought it would be 24 preferable not to just make a tape and bring it here, given 25 that would mean stopping everything and given that you GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 10 1 wouldn't have the opportunity to ask Professor Steele the 2 questions that you might have. And that's what this is really 3 about. It's about whether can we do this by video conferencing 4 or not. I don't think there's any question that we can go out 5 to California, take a video deposition of Professor Steele, 6 and play it for the Court. 7 THE COURT: Okay. Thank you. 8 A couple of things. Number one, is we have a witness 9 here that does not want to voluntarily come. He does not want 10 to testify at this trial at this point. He is represented by 11 counsel, in the form of Mr. Payton who has made that very 12 clear, to the Court, and to everybody here. The suggestion of 13 video conferencing is one I think I brought up at the 14 beginning of this trial, and I brought it up in the context of 15 saving dollars for the parties, and trial dollars, and in the 16 context of where we had a voluntary witness, the witness who 17 is going to get on a plan and voluntarily come to Michigan 18 rather making them involuntarily get on a plane. I thought we 19 save costs for everybody. I thought it was a good idea. I'm 20 an advocate of it, and an advocate of saving trial dollars 21 whenever we can. But I think here it's not appropriate for a 22 couple reasons. Number one is we have a witness that's 23 outside the jurisdiction of this court. I could not subpoena 24 that witness. I could not order that witness to come, more 25 than one hundred miles and so forth. He was made as a will- GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 11 1 call witness by the defendants in this matter, but what that 2 is it's a -- the Pre-Trial Statement to be prepared for what's 3 to be anticipated at a trial. It's not hard and fast thing. 4 It doesn't mean that the party who puts him on as a will call 5 decides that it's not in their best interest or doesn't comply 6 with the rules in terms of relevance, whatever the reason is, 7 that doesn't mean that they have to call that particular 8 person. And that's what happened here. On the 19th of 9 January it became very clear that Professor Steele was not 10 going to be called. And as I indicated I have no ability to 11 compel him to be here. A subpoena for video testimony is only 12 for voluntary witnesses. There's nothing in the rules that 13 provide for it or anything of that nature. 14 So I believe at this point that I can't compel him 15 to be here. In terms of any type of deposition the 16 intervenors had months and months of opportunity to take his 17 deposition. They chose not to for the reasons articulated on 18 the record, but that's not the fault of the defendants in this 19 matter. It's not the fault of the plaintiffs. It's not the 20 fault of the intervenors, but we're in the middle of a trial, 21 and we have a witness who chooses not to make himself 22 available to testify. I have no subpoena power. Therefore 23 the Court -- there's nothing that I can do in relation to 24 providing his testimony here at trial. 25 As to the subpoena in California, I guess California GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 12 1 is going to have to deal with that. It's their subpoena. I 2 suggest that the subpoena whether it be -- whatever reason was 3 for video deposition that I'm not going to permit it at this 4 point because as I say we've got a witness who doesn't come 5 within the spirit of what I thought would be good economy, and 6 therefore, it would be moot out there anyhow the way I see it. 7 So that would be my ruling. 8 MS. MASSIE: Judge, can I say two things? 9 THE COURT: Of course. 10 MS. MASSIE: First, the reason we didn't list 11 Professor Steele as a witness in the Pre-trial Order was 12 because he was listed as a will-call witness and the local 13 rules give us the right to call him without further notice -- 14 THE COURT: Let me just indicate one thing, I didn't 15 sign that order -- I don't want to get into the technical part 16 of it, but I don't want -- I think the lawyering in this case 17 has been outstanding, I'm not criticizing anyone. However, 18 that order was not signed until I think the day after the 19 trial started. 20 Anyhow, go on. 21 MS. MASSIE: Well -- and I don't understand what 22 bearing that has. We were relying on the fact that he was 23 listed as a will-call witness. If he had been listed as a 24 may-call witness we would have done something about that too, 25 to preserve our right to get his testimony in this live record GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 13 1 of this trial. There is nobody who can explain some of the 2 aspects of how race and racism effect academic performance and 3 standardized test scores in the same way Professor Steele can. 4 It is absolutely essential to any intelligent understanding of 5 the questions you set for this trial. And we must then take 6 his video deposition. We didn't take it because Mr. -- it had 7 already been taken by -- 8 THE COURT: I'm nothing going to reargue this. 9 You've argued this already. He chooses not to testify here 10 today. You didn't take it. It was a discovery matter that 11 you chose. At that point there was no will-call or anything 12 of that nature during discovery in this matter because the 13 order had not been signed. There wasn't even a Pre-Trial 14 Order at that time. 15 And, again, I'm not faulting you. I think you're a 16 good lawyer. I think everybody is case are good lawyers. It's 17 not a question of good lawyering or bad lawyering. Professor 18 Steele chooses not to testify. I have no jurisdiction to 19 bring him here. Discovery is completed. The matter is 20 closed. There's nothing else I can do. 21 MS. MASSIE: It's not a discovery deposition. It's 22 a de bene esse deposition, to preserve his testimony for 23 trial. 24 THE COURT: But we're here in trial, and your case 25 is on. And we're going to continue with trial, period. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 14 1 I mean, that's as easy as it is. We're going every day. We 2 have a schedule, and we're going to try to do that schedule. 3 MS. MASSIE: But don't we have to be allowed to send 4 somebody out there simultaneously to take his deposition to 5 play it for you. We can't present our case to you adequately 6 without the man's testimony. And one other thing, Judge, I 7 did speak to him before Mr. Payton instructed me I wasn't to 8 speak with him any more, I did speak with him, with Mr. 9 Payton's knowledge, and he did not say at all that he didn't 10 want to testify for us. In fact, that's inconsistent with our 11 discussion. 12 THE COURT: He has a lawyer and his lawyer has made 13 that representation. And I have the highest regard for his 14 lawyer and as I do for you. And I'm sure wouldn't tell me 15 something that isn't correct, and I'm sure that -- in fact, 16 I'm positive you wouldn't, and I'm positive Mr. Payton 17 wouldn't either. That's his client's position at this point. 18 And as I indicated there's nothing else that I can; 19 do. We've known that he wasn't going to since the 19th of 20 January. There's been no secret. In fact, it's probably been 21 before that, but I think that's when it was finally announced. 22 MS. MASSIE: It wasn't before that. 23 THE COURT: The matter is closed. We can argue that 24 all day, but I've made my ruling, and we have to move on. 25 MS. MASSIE: Well, this is completely unfair, and GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 15 1 you're making it impossible for us to make our case. 2 THE COURT: I don't think it is completely unfair. 3 MS. MASSIE: It is. 4 THE COURT: I think it complies with what the law is 5 and the rules. 6 The next motion that we have is motion -- well, we 7 can do this when we don't have a witness, to clarify the 8 record. We'll got to that sometime probably near the end of 9 the day. 10 MS. MASSIE: Call Professor Martin Shapiro. 11 MR. KOLBO: I'm sorry, I have one other matter with 12 the Court. It's a scheduling issue, and point of information. 13 I wanted to bring the Court up-to-date on what we anticipate 14 for our rebuttal case, and scheduling and so forth, and I have 15 a request to make to the Court. I have talked to counsel 16 about this. We anticipate calling at this point two rebuttal 17 witnesses. First of all, we're going to recall briefly Kinley 18 Larntz, to the stand to respond to some of the testimony of 19 Stephen Raudenbush. And secondly, we anticipate calling 20 Professor Gail Heriot who is a professor at the University of 21 San Diego Law School. She's got an expert report on file. 22 THE COURT: I've read it. 23 MR. KOLBO: And she'll be called first thing Monday 24 morning we anticipate based on what we think the schedule will 25 be at this point. Dr. Larntz has asked me to see if I can't GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 16 1 find to accommodate him to get his testimony on this week. To 2 make a long story short, the only day that really works this 3 week, unfortunately, would be Saturday. And I wanted to raise 4 that as a possibility with the Court, as I know that the Court 5 had expressed at some time that there was a possibility -- 6 THE COURT: If it's good for everybody else, I would 7 be more than happy to be available Saturday morning, if 8 everyone else has no objections. 9 MR. KOLBO: And I have talked with counsel. What 10 we've talked about for various reasons is to suggest that we 11 start early like at 8:30 on Saturday if that's okay. 12 THE COURT: It's fine with me. 13 MR. KOLBO: I think it will be very short, your 14 Honor. I think that our Direct will be less than an hour. 15 And I would be surprised if the Cross would be that long. So I 16 think we're about a half a day. 17 THE COURT: If everybody agrees, I have no problems 18 with it. If no one has any objection, Saturday morning, 8:30, 19 would be acceptable. Let me just check with my staff. Does 20 anybody have any problems? Steven? 21 THE LAW CLERK: No problem. 22 MR. KOLBO: I know it's not the most popular 23 request. 24 THE COURT: We'll make arrangements. As long as I 25 know, I'll make arrangements with the Marshal's Service, and GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 17 1 we'll do it Saturday morning. 2 MR. KOLBO: Thank you, your Honor. 3 THE COURT: Before you let him go, let me just check 4 with the Marshals to make sure -- the Marshals won't give us a 5 hard time, it's with GSA who owns the building to make sure 6 that we can have a little bit of heat in here Saturday 7 morning. Sometimes they give us a hard time on that. Should 8 it be a problem, we'll work something out though. 9 MR. KOLBO: Thank you, your Honor. 10 THE COURT: Mr. Rosner, you're going to recall 11 after the Professor -- is it Professor or is a Doctor -- 12 MS. MASSIE: Professor. 13 I apologize, Professor Shapiro will go first, and 14 then Mr. Rosner. 15 THE COURT: No problem. Professor, you can stand 16 right over here, we'll swear you in. 17 M A R T I N S H A P I R O , 18 being first duly sworn by the Court to tell the truth, was examined 19 and testified upon his oath as follows: 20 DIRECT EXAMINATION 21 BY MS. MASSIE: 22 Q Hi, Professor Shapiro. 23 A Good morning. 24 Q What's your job? 25 A Professor of Psychology at Emory University in Atlanta. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 18 1 THE COURT: Oh, what a nice place. 2 THE WITNESS: Thank you. 3 THE COURT: One of my nieces, in fact, two of my 4 nieces went there. We had an intern last summer that went 5 there. So I had an opportunity to visit the campus. It sure 6 is beautiful. 7 THE WITNESS: Thank you. I've enjoyed being there. 8 A long time now. 9 THE COURT: A long time. 10 BY MS. MASSIE: 11 Q How long have you been there? 12 A I came there in 1964. That's a long time. 13 Q As a professor of psychology? 14 A I came -- actually I was an associate professor of 15 psychology, and I guess four years later was a professor of 16 psychology. 17 Q And four years later you were awarded tenure? 18 A No, actually I came in '64. I had left the previous 19 position which I had tenure so I got tenure back in 1965. 20 Q Tell us, just briefly, about your employment before you 21 got to Emory if you would? 22 A Okay. I'll start the education I guess. I have a 23 bachelor's degree from Yale University with a major in 24 psychology. A PhD degree from Indiana University with 25 concentrations in psychology and mathematics. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 19 1 After I received my PhD I stayed a year at Indiana 2 as the United States Public Health Post-Doctoral Research 3 Fellow. Then I went to The University of Houston as an 4 assistant professor of psychology. And I became an associate 5 professor of psychology at the University of Houston, and then 6 a visiting associate professor at Baylor Med School. 7 I then moved to Emory as an associate professor of 8 psychology. And except for a sabbatical period at The 9 University of College London I've been at Emory the entire 10 time. 11 Q What areas have you focused on? 12 A Well, a long time ago I did laboratory work with animals 13 in learning and conditioning. But since -- after that I did a 14 fair amount of research in learning in children. And then I've 15 mostly concentrated in test and measurement. During the entire 16 period I've always been at -- on the mathematical statistical 17 side of psychology. So at the moment I would say statistics 18 and test and measurement. 19 Q What is your teaching? What are your teaching 20 responsibilities? What courses do you usually teach? 21 A I teach two courses, one called psychology and law, and 22 the other course is psychological test and measurement. 23 Q Have you offered expert testimony in the past? 24 A Yes. 25 Q Tell us the areas in which you have offered -- bearing GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 20 1 mind we're going to be focusing on testing, educational 2 discrimination in tests? 3 A Well, let me get rid of the others first. 4 And irrelevant area that is irrelevant to this trial 5 that I've done a lot of testifying in is regarding the 6 probabilities associated with DNA, matching primarily in 7 criminal cases some in paternity cases. Again, it's really 8 statistics and probability but it's a different question. A 9 lot of cases in employment discrimination. Again, really the 10 same issues, that is, the issues of selection and the validity 11 of the selection procedures. In fact, the issues are 12 identical, that is does a selection procedure have disparate 13 impact and then is the selection procedure valid in a 14 psychometric sense. some cases dealing with other kinds of 15 statistical issues like targeting by state police in highway 16 stops has racial targeting. But the remainder of the cases 17 have dealt with educational testing, educational selection. 18 Again, the issues being the adverse impact of a selection 19 procedure and the psychometric validity or invalidity of the 20 selection procedure. 21 Q Tell us about some of those cases? 22 A Well, the first one was a case brought by Georgia Legal 23 Services Corporation in which we obtained the Office of Civil 24 Rights, that is, the U. S. Department of Education Office of 25 Civil Rights computers tapes on all of the public school GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 21 1 districts in the state of Georgia. And suit was brought against 2 11 of the districts after we had analyzed the tapes and 3 selected the districts with respect to ability tracking, that 4 is, their use of testing to assign students differentially the 5 classrooms. And the use of testing to assign students to 6 special education. 7 I don't want to make it too long a story but 8 typically in many of the southern districts, of course, 9 Georgia is one of the southern states, the school systems did 10 not use ability tracking until after they were desegregated in 11 1971. So the issue was really a statistical analysis dealing 12 with whether -- within school placement had served as a 13 substitute for jurist segregation that is, where they were 14 segregating within the schools. And that case was in the 15 Southern District of Georgia, Savannah. 16 There was another case in Savannah dealing with the 17 use of an exit exam, that is, after the school system, Tagnal 18 County, Georgia, had desegregated, they then instituted an 19 exit, that is, you can only get your diploma if you passed the 20 exit exam which was which was the California Achievement Test 21 that they used. 22 Q Let me stop you for one second. The first case you just 23 mentioned the tracking case is that George State Conference? 24 A The Conference of Branches -- I'm sorry. The Georgia 25 Conference of Branches of the NAACP versus the state of GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 22 1 Georgia. 2 Q And the other one you were just telling us about? 3 A It's styled as Johnson versus Sikes. 4 There is a slightly motive and really a school case 5 but it's a licensing test -- licensing case. 6 I guess the next case was Golden Rule Insurance 7 Company versus Mathias. Mathias was the commissioner of 8 insurance of the state of Illinois. And the educational -- 9 THE COURT: Are these cases that he's testified in? 10 MS. MASSIE: Yes. 11 THE COURT: Do we need all the history of the cases? 12 MS. MASSIE: I think this one will be helpful. We 13 won't go into all of them. 14 THE COURT: Go on. 15 BY MS. MASSIE: 16 A And the Educational Testing Service was joined as a 17 co-defendant in that case. And that's in a sense why it's 18 relevant. And that case got settled. There was formed an 19 advisory committee which oversaw the settlement for a period of 20 five years, and then there were reports for another five years. 21 So that case involved a lot of interaction with the Educational 22 Testing Service after the case as well as before. 23 The other school case -- and I'll try to speed it 24 up, I'm sorry. One was sure Sharif versus the Board of 25 Education of New York which was a challenge to New York GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 23 1 state's use of the SAT as a selection device for the awarding 2 of state -- college scholarships. And the case dealt with the 3 validity of the SAT for that purpose. 4 Another case was the People Care versus Rockford 5 Illinois case. Another case was the City of Little Rock 6 School District versus the Pulaski County School District. It 7 really was a challenge by the City of Little Rock School 8 District but the County was creating a white flight by the 9 testing procedures and the placement of student procedures 10 used by the Pulaski County School System. Pulaski surrounds 11 the City of Little Rock. 12 Then there is the case of Hopwood versus the State 13 of Texas dealing with the affirmative -- a challenge to the 14 affirmative action programs used by the University of Texas 15 Law School. There is also Wooden versus the Board of Regents 16 of the University of Georgia which was a challenge to the 17 University of Georgia affirmative action procedure in the 18 undergraduate college of the University of Georgia Athens. 19 That may exhaust, I'm not sure. 20 Q Some of those cases don't -- not all of them dealt with 21 questions about tests, test bias, test validity? 22 A Yes, they all do in one way or another. Some of them are 23 very specific to university admissions. 24 Q I'm going to try two things at once. If I can get you to 25 turn to Tab 170, in the binder that's in front of you. If you GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 24 1 could take a look at that and tell us if it is the report of 2 your expert testimony that you filed in this case. 3 A Yes. 4 MR. PURDY: Excuse me, your Honor, I'm sorry to 5 interrupt. I apologize I didn't do it earlier, but just so 6 it's clear, we have our continuing objection. 7 THE COURT: Yes. You can do it whenever you like, 8 but your objection is continuing. 9 . MR. PURDY: That's fine. 10 THE COURT: It's a new day and a new witness. 11 MR. PURDY: I apologize. 12 THE COURT: No problem. 13 MS. MASSIE: You don't have to stand up and say that 14 every time. We can concede that you object to the relevance 15 of everything to do race and racism in this case. 16 THE COURT: Well, you know, there's no speeches. 17 He's made his objection, and let's go on. 18 BY MS. MASSIE: 19 Q Doctor Shapiro, that is your expert report? 20 A Yes. 21 MS. MASSIE: I'd like to move the admission of the 22 report and simultaneously ask that Professor Shapiro be 23 certified as an expert on psychometrics, generally questions 24 of test validity, test construction, test assessment. 25 THE COURT: Any objection? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 25 1 MR. PURDY: Only subject -- 2 THE COURT: Continuing objection. 3 MR. NIEHOFF: No objection. 4 THE COURT: The report will be received and he will 5 be able to testify as an expert in that area. 6 MS. MASSIE: Thank you, Judge Friedman. 7 (Trial Exhibit Number 170 received into evidence.) 8 BY MS. MASSIE: 9 Q Professor Shapiro, in your report, your earlier work on 10 this case you wrote that there are four basic parts to the 11 business of putting a standardized admissions tests. I'm going 12 to focus your attention on the first two parts that you 13 identify for your testimony today. But if you could please say 14 what the four parts are and say a little bit about them for 15 everybody's benefit, as an introduction. 16 A The first step I discuss is the definition of the content 17 domain of the test. That is, very simply, the decision as t 18 what will be tested, that is what kinds of material will be on 19 the test. Just to use an example, on the LSAT there's a verbal 20 section, there's a quantitive section. Somebody made that 21 decision. There is no content regarding social science. 22 There's no content regarding art or music. I mean, those are 23 the kinds of decisions that are made optretory when the test 24 is first designed. The second step is the actual writing and 25 selection of the items on the test. That is, once you decide GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 26 1 what the content area is you then have to write the test 2 questions and you have to test or pretest the examination 3 questions. And that is in some sense a very well, as I tried 4 to explain a very critical and important step in the process. 5 And really determines the nature and consequences of the test. 6 I assume we'll go into that in more detail. 7 The first step is an attempt to validate the test, 8 that is, an attempt to compare the results on the test to some 9 criterion, that is, to simply calculate the equation that 10 predicts some criterion from the test score itself. It may be 11 the test score is used in conjunction with other information, 12 for example, we all know that in the case of admissions 13 typically the predictive equation uses both the test score and 14 the prior grade point average. If it's college admission, 15 it's the high school grade point average. If it's 16 professional school it's college grade point average. 17 The fourth step of -- which frequently is not done 18 actually, but the fourth step is to look backward at the 19 predictive equation and ask the question did it actually 20 predict, that is, after the students are admitted into school 21 and have had a year or more of grades to go back and to see 22 whether the predictive equation was accurate, that is, the 23 degree to which it actually did predict grades. 24 Q How many times did test companies, test makes go through 25 this process that you've discussed, this four-part process of GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 27 1 producing a test? 2 A Well, although the content domain has some tendency to 3 have its own inertia and stay rather constant, the process 4 itself is a continuing process. I mean you continually have to 5 come up with new forms of the exam. You have to -- new 6 versions, new questions. So you constantly have to test new 7 experimental items. 8 You then have to constantly look at last year's 9 students and the derive an equation from which you will now 10 try to predict performance by next year's students. And then 11 hopefully, although I say frequently is not done, you have to 12 look at this year's students one year later and say did the 13 regression equation, did the predictive equation that you 14 dragged from last year really work this year. So it's 15 continual. I mean, it's a never ending process. It's what 16 keeps selling tests. 17 Q I want to back you up for just a second. Why do you need 18 new test forms? Why do you need new versions of the test? 19 A Well, obviously -- people would memorize the test 20 questions and -- well, they would become a commodity of market. 21 I mean, you could sell test questions, but furthermore, if that 22 happens then obviously the test results would have no 23 particular meaning other than who could manage to acquire old 24 test scores or their equivalent. 25 Q Why do they have to test the new questions? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 28 1 A Well, several reasons, but primarily you cannot really 2 predict whether a question is a good question. And that 3 statement contains a lot of pieces. You may design a question 4 that is misleading and not realize it, and that would not 5 become obvious until you pre-tested the question. And you look 6 subjectively at the -- I'm going to use a technical term, the 7 distracters, the wrong choices were chosen. And you might look 8 at the wrong choices and see well, people must have 9 misinterpreted this question to have picked this choice, and I 10 now see what was wrong with the question. So we either revise 11 the question or throw it out or whatever. 12 There is more to it than that simple effect. There 13 is the desire to have a high degree of homogeneity within the 14 test, that is, the test makers strive to have the results on 15 the test items be comparable from one item to another. They 16 want people who get a high score on the test overall, on the 17 total test, to also get -- also have the higher probability of 18 getting this particular item correct. That is somehow the 19 idea is that the test is measuring a single trait, a single 20 attribute and so all the items ought to correlate with each 21 other. But in somebody who gets a high test score after 22 getting all the items correct, and somebody gets a low test 23 score, ought to be getting all items wrong. Well, of course 24 it doesn't come out perfectly like that, but it should tend in 25 that direction. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 29 1 Q You say in your report as I recall that new test forms 2 have to be equated in addition to the statistical limitation 3 constraints you've described, new test forms have to be equated 4 to old ones. Can you say what you mean by that? 5 A That's a rather real complex process. What you strive to 6 do typically is to have the -- I'll use the term that I don't 7 really think is accurate, but it's used. You want to have the 8 same level of difficulty from test form to test form. Now, by 9 difficulty you have no way of measuring difficulty other than 10 how many people get it right or wrong, okay. But using that as 11 the definition of difficulty rather circular definition, but a 12 definition, you want to have your successive test forms of 13 equal difficulty so that a particular test score on the new 14 form is comparable to a test score on an earlier form because 15 when the students apply they have taken different test forms. 16 And the you want to have some assurance that the test forms are 17 comparable. 18 So without pre-testing, you would have no way of 19 knowing the difficulty level of each time. The difficulty 20 level ends up being more than a hundred people that got it 21 wrong or wrong. 22 So in order to have test equating, and in order to 23 have test homogeneity, internal consistency, the items must be 24 pre-tested. There is no alternative. 25 Q And the test equation requirement if I understand you the GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 30 1 same thing to an admissions officer several years ago? Is that 2 a fair summary or not? 3 A Yes, and the way it's done is there -- although there are 4 different variations of it, at the basic level at some point 5 you must have common items, from form-to-form, because it is 6 possible that the test takers change over time. And it's only 7 by having items that are common to more than one form that 8 you're able to maintain the correction, that is, you're able to 9 maintain the -- any change in how you scale the exam, that is, 10 how you interpret the number of items correct. 11 So here you have a test form which consists of items 12 that been pre-tested on earlier forms, and then this test form 13 is taken by the same students who are taking these new 14 pre-test items. These new pre-test items go into a new form, 15 with newer set of pre-text items then every step along the way 16 you have some items that had appeared in a previous form 17 either as actual items or as pre-test item. So now you can 18 link the forms to each other and equate them. 19 Q So the test kind of become its own measuring stick? 20 A Oh, yes. The whole thing -- 21 THE COURT: Yes. From now, you just have answer the 22 questions. 23 THE WITNESS: I'm sorry. 24 THE COURT: You have a very good lawyer questioning 25 you and she'll draw out whatever she needs. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 31 1 THE WITNESS: Yes. 2 BY MS. MASSIE: 3 Q But Professor Shapiro, you should develop the points that 4 you think need developing technically -- 5 THE COURT: I think that's the attorney's rule, to 6 develop points that you believe are relevant. Otherwise, we 7 could be here for decades. 8 BY MS. MASSIE: 9 Q We'll try to work together, Professor Shapiro. 10 Let me turn your attention to the content domain 11 question. 12 A All right. 13 Q How does the domain of a test in terms of its contents 14 get determined? 15 A Very often the test is covering actual academic subject 16 areas. That is -- 17 THE COURT: Do you have expertise to the domain. Do 18 you write exams? Do you help select the questions, the areas, 19 things of that nature? 20 THE WITNESS: Well, I have previously with respect 21 to exams. I mean, I don't that as my full-timed occupation. 22 THE COURT: Is that why you're here today to 23 testify? I thought it was a different area. 24 THE WITNESS: Well, it's part of it. 25 THE COURT: Okay, go on. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 32 1 BY MS. MASSIE: 2 Q Who determines the content of the test? 3 A Well, the test manufacturer. And on some exams the 4 content of the test is given to some extent by the content by 5 the of what you're trying predict. For example, freshmen in 6 college have to take calculus typically. Calculus requires the 7 knowledge of algebra and geometry so not surprisingly the SAT 8 covers algebra and geometry. It doesn't cover calculus because 9 calculus you take as a freshman in college. 10 Other exams like the SLAT, on the other hand, have 11 no real prerequisites to the subject matter that is going to 12 be taught. But if you don't learn law as an undergraduate as a 13 prerequisite as a basis for learning law in law school. So 14 the content of the LSAT is not subject matter driven. It is, 15 in fact, more of an aptitude or construct driven test. It's 16 timed to measure something abstract rather than specific 17 knowledge like the at SLAT to some extend does. 18 Q How does the way of doing that get decided on? 19 A Oh, It's a valued judgment by test constructor. 20 MR. PURDY: Excuse me, your Honor, I have to object 21 just on foundation grounds. If we're going to go into the 22 LSAT or the SAT unless there's some foundation laid that he 23 knows how they go about developing their questions, then I 24 think it lacks foundation. 25 THE COURT: I don't think they got into that GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 33 1 specifically yet, but your objection is well taken. Right now 2 I think the question is general, how does somebody develop it. 3 BY MS. MASSIE: 4 Q Tell us what you know about the test development 5 procedures that go into the making of the SAT and the SLAT, if 6 you would. 7 A Well, the test manufacturers do go put out technical 8 manuals or descriptions of their procedures. The general set 9 of rules regarding the construction of tests is a document 10 referred as to the standards for psychological and educational 11 testing which is put out by the American Psychological 12 Association, and three3 of the educational associations -- or 13 two of them. I don't remember exactly which ones in education. 14 But these specified standards for test construction that are 15 generally accepted throughout the industry and the standards 16 require documentation so they require at test makers to state 17 what their procedures are and what their standards are. And 18 they typically will tell you the distribution of items, 19 difficulties on the test forms. They will tell you the leve of 20 correlation that they require between individual items and the 21 test as a whole. 22 I mean these are not trade secrets. These are -- 23 this is information that has to be displayed in order to 24 satisfy the professional standards. 25 Q And have you looked at the literature that produced about GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 34 1 the making of the make of the SAT and the LSAT? 2 A Yeah. Well, all of the admissions tests have some degree 3 of documentation. The SAT and the ACT have more available than 4 the others, but they all have some, enough to be able to 5 ascertain the procedure that's used. 6 Q Are the test development procedures and question 7 selection procedures that you started to talk about, and will 8 be talking about are those pretty much parallel across the SAT, 9 the LSAT, and GMET, the other standardized admissions tests? 10 A Yes. For example, the LSAT was first originally designed 11 and constructed by the Educational Testing Service using the 12 same general format and procedures that are used in the SAT. 13 It was later weaned from the Educational Testing Service but 14 it's not changed in its technical specifications. 15 Q With that let's go back to the question of how the 16 content of the test gest defined. I'm specifically interested 17 in asking whether it ever changes? 18 A Yes, there is a fairly recent interesting example of that 19 which really grew out of the case Sharif versus the New York 20 Board of Education. It is easily shown that the performance of 21 women on the SLAT below the performance of men. And the 22 argument was presented and actually the trial court in the case 23 recognized that argument as being correct, that what that 24 really means is that SAT can not be a valid predictor or across 25 sex, across men and women because, in fact, women get higher GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 35 1 grades in secondary school and get higher grades in college 2 than men do. So if men are performing better on the SAT than 3 women but women perform better in both high school and college 4 well, the SAT is neither predicting college nor postdicting igh 5 school. therefore, the test is by definition invalid. 6 What was done is in order to ameliorate the 7 difference with respect to awarding of scholarships like the 8 national merit scholarship the Office of Education negotiated 9 an agreement with the Educational Testing Service in which a 10 writing section was added to the PSAT which is used for the 11 scholarship slip awards, in order to reduce the gap between 12 and male and female performance. I mean, there was change in 13 the content that was driven by the intended or unintended, 14 doesn't matter, consequences of the PSAT. So it was really 15 such a political solution, not a psychometric solution. 16 Q When you say the consequences of the PSAT, spell that out 17 for us. 18 A Well, if you look at the awarding of National Merit 19 Scholarships, like the awarding of the New York State 20 Scholarships, the majority of the scholarships go to males. I 21 forget the exact number, but it's quite different. I mean, it's 22 more than sixty-forty, for example. And logically, of course, 23 that makes no sense because again women perform better in high 24 school and perform better in college, and yet men are getting 25 the majority of the college scholarships, the National Merit GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 36 1 Scholarships. 2 So the decision was made to try to reduce that 3 difference and the way it was done was to change the contents 4 of the test. 5 Q What was the impact of that intervention of the context 6 of the test, if you know? 7 A It did reduce the gap, and I don't remember the exact 8 numbers. It did not reduce it to fifty/fifty, but it do not 9 eliminate the gap. If you take -- if you consider grades as a 10 valid criterion, of course, the results really should be in 11 favor of women. That is, more women should get the 12 scholarships than men, and that's not true. More men still get 13 them. There is still a difference in male and female 14 performance on the SAT and PSAT. 15 Q Generally speaking, who ends up making these decisions 16 about the content? 17 A Well, it's the decision of the test maker. And the 18 example I just gave demonstrates that, you know, the test maker 19 in response to whatever pressures are put on the test maker 20 either by government or by the users of the test in this case 21 the colleges. The colleges really didn't do that; although 22 some colleges over time did eliminate their reliance on the 23 SAT. I mean, in a sense, that's pressure also as you begin to 24 lose market. 25 Q Could you make standardized admissions tests more GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 37 1 predictive by changing the content? 2 A Very likely. The correlations between the admissions 3 test and actual performance in the first year of school are 4 frequently not very good. In fact -- although they differ from 5 school to school, it's not hard to find universities in which 6 the predictability of the admissions test is very poor. And so 7 it is certainly reasonable to think that if you broaden the 8 content of the test or change the content of the test that 9 would have an effect and many cases the only effect would be to 10 go up because the predictability is really very bad. Although 11 again, it differs from school to school. 12 Q What do you mean broaden the contents of the test? 13 A Well, if you look at the content and see that it only 14 covers a limited of skills or knowledge areas you could broaden 15 it in the sense of covering a larger spread of constant areas, 16 that is, it could well be, although it is speculation, it could 17 ell be, that adding social science or actually science -- I 18 mean, science really isn't on the SAT either -- I mean asking 19 questions about actual science, if you added social science and 20 science it would not be surprising that it might predict 21 first-year performance even better. 22 Q If it might -- if you could increase prediction by 23 tapping into a broader range of skills and knowledge areas why 24 isn't that done? 25 A Well, I think the biggest reason is simply the inertia GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 38 1 but every time you change the contents of the test you loose 2 the ability to equate with prior forms. The test has now 3 changed, and so students who are applying, who took the test 4 three years ago are giving you a different measures than 5 students who are applying this year. Now -- I mean, it's not 6 insurmountable, and periodically the tests do get changed, but 7 it does create an administrative problem, and that's part of 8 the reason for the inertia. So, I mean, in some sense it 9 involves you're saying well, the test wasn't as good before as 10 it could have been. I mean, it's a funny admission of the 11 quality of the prior test. So there's a great tendency not to 12 do it. And the content remains relative stable even though 13 there really is no quantifiable statistical evidence to support 14 it remaining constant over time. 15 Q Let's turn to item selection. Tell us how new test items 16 come into being? 17 A Of course, it's basic level in committees or groups or 18 individuals who are contracted to write items. For example, the 19 Educational Testing Service has big brochure on item writing, 20 that they give new people who have not been involved in it 21 before. And the item writing at some level involves the 22 simplest things like can you use answers like none of the 23 above, or one or two versus three and four, or do you use 24 negation or -- all that kind of thing. At another level item 25 writing is -- becomes a statistical question of how does this GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 39 1 item compare to other items. That, is what, is the 2 relationship between performance on this item performance on 3 the remainder of the test, the other items. As you get into 4 this whole issue that in general falls within the rubric of 5 test reliability, that is, the ability of the test to give you 6 a constant answer or relatively constant answer, not just 7 form-to-form but item-to-item. 8 Q How does an item to the point where it's being pre-tested 9 which as I understand is what you're getting at now How does an 10 item get to that point? 11 A Well, that point is simply writing the time and going 12 through some committee review or logical review. And then 13 those items are placed on different test forms. that is, you 14 need to pre-test lot of items so that the way you do it is if 15 you have, let's say, four hundred new items you might put -- 16 you could put ten of them on this many tests that handed out. 17 You could put another ten on another number of tests that are 18 handed out. So you get these smaller samples of individuals who 19 each only are doing ten of these questions but over the whole 20 testing group that day that test administration, you're doing 21 all these four or five hundred test items. 22 Then at the end of the you've got the ability to 23 calculate whether the probability of getting this item, what 24 is the probability of getting this new item correct compared 25 to this other old item, or set of old items. So the ultimate GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 40 1 selection is made on the basis of the item performance across 2 these individual groups of test takers. That's how you 3 generate enough pre-tested items. 4 Q So every item that ends up being scored is pre-tested; is 5 that true? 6 A That's correct. It's put all in a prior form and 7 pre-tested. Now, when I say it's true, it is possible that you 8 might make some slight change in the item and assume that one 9 changes the statistical characteristics. But even that's 10 unusual. 11 Q What information do the test companies get from the 12 pre-testing? 13 A Well, originally the information that was gotten is 14 actually -- let's say it this way: There are actually test 15 strips that are made up. There's a statistical table that is 16 made up for each item. And you can look up the item in its 17 statistics. And originally those statistics were the percent 18 correct for that item. And that would be the average test 19 score of every person who picked Choice A, that is their 20 average performance on the rest of the test. The average test 21 score for a group of people who chose alternative B, okay, 22 that's on the test strip. And then also on the test is some 23 estimate of the item difficulty calculated on the basis of the 24 percent correct on that item versus the percent correct on the 25 other item, that is, some comparable measure, some standardized GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 41 1 corrected difficulty. 2 And then there's a section of the item strip that 3 has to do with the correlation between that item and the rest 4 of the test. And frequently that's done in terms of looking 5 at the lowest performing third of all the test takers, what's 6 the correlation between their performance and the performance 7 on this item looking at the middle third, looking at the top 8 third, and say you get this distribution of correlation. 9 That's the way it was originally done. Later on, as 10 really as a consequence of the Golden Rule versus Mathias 11 lawsuit, those test strips to contain information on 12 statistics that were broken down by racial and sex groups, 13 that is, the statistics would be shown separately for male and 14 female. The statistics would be shown separately for the 15 major racial groups. And then there were also developed 16 techniques creating statistics which purported to show the 17 relative difference between the racial groups in performance 18 on that particular item, that is, a method called differential 19 item functioning was developed by the Educational Testing 20 Service, purportedly to show which items have bigger racial or 21 sex differences than the overall test items, which items stood 22 out as being more impacting on a particular group than the 23 item -- the remainder of the items. 24 Q What do you mean more impacting? 25 A Here's the technique. We take everybody who got a hundred GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 42 1 and fifty on the exam, just to pick a score, it doesn't matter. 2 We'll assume -- I don't agree with this -- but I'm saying it 3 this way -- we'll assume that everybody who gets a hundred and 4 fifty is of equal ability. Now, we'll look at this particular 5 item and we'll ask what was the percent correct for the women 6 and what was the percent correct for the men. Given that we 7 presume that the men and women we're comparing are equally able 8 because they all got a hundred and fifty, we'll ask is the 9 difference between percent correct male versus female on this 10 item greater than we would expect by chance as we'll run a 11 statistical test on it to see whether this item is giving you a 12 sex difference even though we're comparing people who are 13 otherwise equal in test performance. 14 Q Well, how does that work if the test has a different 15 impact on women, for example, than men, black test takers than 16 white ones? 17 A As I say, I don't agree but that method, that method 18 that's called DIF, differential item functioning, actually does 19 look at the job because if you look at the -- 20 THE COURT: You don't agree, but there are other 21 experts that have obviously tested it use it as a standard to 22 some extent? 23 THE WITNESS: No, I wouldn't say it that way, your 24 Honor. The testing companies, specifically the Educational 25 Testing Service uses it. I agree that it does pick the most GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 43 1 extreme items. I do not agree that it's sufficiently 2 sensitive to pick items that have less than most extreme 3 differential -- 4 THE COURT: But you said whatever it is, Educational 5 Testing, uses it. You also said they have some kind of 6 protocol or something that's distributed in the industry, and 7 academically and so forth. They must have devised this system 8 somehow. 9 THE WITNESS: Educational Testing Service, 10 statistician, Paul Holland, devised it as a result of the 11 litigation with the Golden Rule Insurance Company as an 12 alternative technique for detecting bias items. 13 THE COURT: You suggested a better technique? 14 THE WITNESS: Yes, I think there are better 15 techniques. I then argue that then in the extreme, that is, in 16 an extremely biased item you would detect it by using the DIF 17 Method. I'm simply saying there are items that are biased but 18 not as extremely biased, that the DIF Method is insensitive 19 to. 20 THE COURT: But other than the DIF Method, there's 21 another method they could use in or do that. 22 THE WITNESS: Yes, they do, but they don't. 23 THE COURT: They chose not do. 24 THE WITNESS: Yes. 25 THE COURT: Do you know why they chose not? Of your GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 44 1 own knowledge, not what you've heard, or -- I mean, have you 2 ever approached them, or ever discuss it with them? 3 THE WITNESS: Yes. I was involved in the committee 4 that formulated the Golden Rule settlement with the 5 Educational Testing Service, and I would then -- Paul Holland 6 and I were both involved in the advisory committee that 7 oversaw it. And it was developed as he and I interacted. So I 8 am familiar with it. 9 One of their arguments, I understand. Let -- 10 THE COURT: You don't have to get into it. Would it 11 be fair to say you have a disagreement with them in terms of 12 -- they believe one thing and you believe another? 13 THE WITNESS: Yes, that's fair. 14 THE COURT: You may move on. 15 BY MS. MASSIE: 16 Q What's the basis of the disagreement? 17 A Okay. If you take the LSAT, it is clear that at the very 18 top of the distribution that is, let's say, at the very highest 19 score -- 20 THE COURT: Why don't you explain -- you say -- you 21 say, "it is clear." Tell us if it clear to both sides or just 22 to you. 23 A It's clear to everyone. 24 THE COURT: Okay. 25 A That the distribution of LSAT scores is such that women GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 45 1 are not in the very highest scores. That is, if you look at 2 the top of the LSAT distribution and these distributions have 3 been published in the past so this is literally at -- 4 THE COURT: You're testifying under oath. It's 5 clear to everybody -- differentiate between what's clear to 6 you, and what's usually may be the difference between them. As 7 to the top of the women, everybody agrees is what you're 8 saying. 9 A The numbers clearly show that women are under-represented 10 in the highest, the very, very highest LSAT scores, which means 11 that when you say let's hold the ability constant and you set 12 that constant value for your -- this one group at the very 13 highest score, you can't ask that question because there are, 14 in essence, no women in that group, or if you do it by race. 15 What you get looking at the distribution is although you say 16 you're holding people equal by ability at the very highest 17 levels you can't a statistically significance difference 18 between the performance of Group A and Group B because there 19 are not enough people out of Group B even in the group to get a 20 reliable estimate of the item, the particular item difficulty. 21 So what you have is a method that is measuring relative 22 differences between items, but has no standard for overall test 23 bias. In fact, it incorporates overall test bias by taking 24 that total test score as your measure of ability. 25 Clearly if you use the DIF Method and had an GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 46 1 external validate measure of ability, the method would make 2 sense. But if the measure of ability is the performance 3 internally on the test itself, then obviously the method 4 incorporates the overall average level of test bias, and 5 you're simply evaluating an item against that average level of 6 test bias. 7 Q So to be clear there are groups of people who score lower 8 on standardized admissions tests, and let's focus on the LSAT, 9 women are one such group on average; is that true? 10 A Yes, although with women once you get off the top of the 11 distribution it rapidly becomes consistent looking. So if 12 you're looking at the number of men and women in the 70th 13 percentile you're not to going find a sex difference. If 14 you're looking at the 99th percentile, you would. 15 Q In the aggregate, on average, there is a score gap 16 between women and men on the LSAT? 17 A I don't know if it's statistically significant on the 18 average. In some sense, it's an irrelevant question and 19 because people at the bottom of the distribution don't get 20 admitted. So the real issue is what happens in the upper 21 reaches of the distribution, And yet in the upper reaches of 22 the distribution there's a difference. 23 Q Let's turn to racial groups. Are there aggregate 24 differences in test scores to your knowledge by race on the 25 LSAT? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 47 1 A Yes, there are. 2 Q Describe them. 3 A Well, it varies from school-to-school. I'm don't think 4 you can come up with a general description, but if you look at 5 the overall distributions -- it would look as though you have 6 one valve-shaped curve for whites, and let's say another valve 7 shaped curve for African-Americans, and it's displaced by a 8 pretty good number of points. I don't have an exact number, but 9 I mean it's not clearly statistically significance difference. 10 I mean, you can see it. It's two different distributions. I've 11 seen a lot of these kind of data. There's no dispute about 12 that. 13 Q I want to tie this back to what you were saying -- 14 THE COURT: When you say no dispute, does the 15 Educational Service, do they dispute that? 16 THE WITNESS: No, no one does. I mean, when you 17 look these tests scores it's clear that you get very large 18 differences due do economics. I mean you see that in the SAT 19 where students have fill out a questionnaire before they take 20 form, and you can just plot the distribution for everyone 21 whose income is in this bracket, or the next bracket. And 22 they give out the samples, the data, each year of about a 23 hundred thousand randomly selected test papers. You can buy 24 the CD rom for a hundred and fifty dollars and look at it 25 until your heart is content, by economics, by race, by GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 48 1 urban-rural, by state, by religion. I mean, there's huge 2 differences from religious group to religious group. And it's 3 just in the data base. I mean you can look at it. 4 BY MS. MASSIE: 5 Q Going back previously what you were saying about 6 differential item functioning, how is it that method of 7 attempting to get at bias which is now I assume how ETS frames 8 it, deals with the average gap you've just described by race? 9 A It doesn't. 10 Q Why is that? 11 A Because it's comparing only people who have been equated 12 on totals test score. So if you look at these two 13 distributions when you're looking at the top grades you're 14 containing almost no people at all from this other racial 15 group. And then finally when you get down into a lower range 16 are you comparing let's say blacks to white, but that is 17 assuming that those students black and white are equal in 18 ability just because they got the same test score, but, in 19 fact, you know that's not true. 20 Q Why? 21 A Because the test items are selected to conform with, to 22 correlate with, overall performance. So, let's say, let's take 23 two things that are true. Let's say, we've got African- 24 American scoring as whole lower than whites. Now,let's assume 25 that we get an item in which African-Americans actually do GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 49 1 better than whites. Well, when you calculate the correlation 2 between that item and total test score, that correlation, in 3 fact, would be negative. The students who do worse on the test, 4 the African-American students, are doing better on that item. 5 The students who do best on the test, the white students, are 6 doing worse on the item, you get a negative correlation, that 7 item gets thrown out. In fact, the only items that will get 8 selected are the items that for future use are the items that 9 preserve the difference that existed in the earlier test form, 10 because they're the only items that are consistent the earlier 11 test score. They are the only items that have a high positive 12 correlation. 13 THE COURT: But the same exact principle would apply 14 for economics, for rural versus urban -- 15 THE WITNESS: That is right. 16 THE COURT: For religion. 17 THE WITNESS: That is correct. 18 THE COURT: So it's not just race. 19 THE WITNESS: That's correct. 20 BY MS. MASSIE: 21 Q Does racial Impact of the different questions is made 22 known by the pre-testing procedures; correct? 23 A Absolutely. As I said before, it is now standard 24 practice to calculate the statistics for racial and sex groups 25 separately. The data are all there, they're laid out in front GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 50 1 of you. 2 THE COURT: They do it for economics too? 3 THE WITNESS: No. 4 THE COURT: They don't do it for rural or. 5 THE WITNESS: No. 6 THE COURT: Or north versus south. 7 THE WITNESS: No. 8 THE COURT: But you would predict it would be the -- 9 THE WITNESS: Absolutely. 10 A The test is designed to preserve the status quo. That's 11 the effect of it. Whoever did before is now going to be 12 carried along by picking items that create a similar group of 13 winners again. 14 BY MS. MASSIE: 15 Q Is there anything external that's used to say whether a 16 question should be thrown out or not? 17 A No. There is an example that is cited by the Educational 18 Testing Service. In 1932, Carl Bringham who originated the a 19 LSAT test did a study in which he compared some antonyms 20 performance or some antonyms questions to actual English course 21 grades six months later. As far as I can tell that is the only 22 time anyone has compared test performance item-by-item to some 23 external criterion, and that was in 1932. 24 THE COURT: But those that may use these exams for 25 whatever purpose are aware of the same thing that you are GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 51 1 aware of or may be if they wish to make -- avail themselves to 2 that information and, therefore, if they're the decision 3 makers they can take that into consideration, in making what 4 use they may use in these exams. 5 THE WITNESS: Your Honor, I agree with you and I'm 6 not trying to sit here and say that the exams are evil. I'm 7 saying that if you understand what the test means, and what 8 the test scores mean then they do provide you with additional 9 information. 10 THE COURT: As long as you understand what they 11 mean. They don't take them at raw scores. 12 THE WITNESS: That is correct. 13 THE COURT: You take into consideration that they 14 may have some bias in terms of race, but they may have some 15 bias in terms of urban versus rural. They may have some 16 religious bias -- 17 THE WITNESS: But you can't look at these predictive 18 equations based on test score and grade point average and use 19 them in a mechanical way ignoring all the other attributes of 20 each additional applicant and come up with a reasonable and 21 valid answer. 22 THE COURT: Now, I understand your position. 23 Thanks. 24 BY MS. MASSIE: 25 Q What's the impact on black and minority test takers of GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 52 1 the test development procedures that you've described? 2 A Well, the impact is enormous. The difference in 3 performance among racial groups is large enough that if you 4 have a highly selected institution and you rely exclusively on 5 this predicted first-year GPA you are going to exclude all the 6 members of various minority groups. 7 THE COURT: But only minority groups, but also 8 groups that may be the majority, that are of low economic area 9 -- 10 THE WITNESS: That is correct, your Honor. 11 The biggest study I've ever seen was done at UCLA where they 12 actually looked at the economic status of the admitted 13 students, and seventy percent -- this is a state school, after 14 all -- but seventy percent of the students admitted to the law 15 school came from families with yearly incomes of excess of two 16 hundred thousand dollars. That is an extraordinary statistic. 17 They then instituted a program in the admissions procedure 18 trying to get greater economic diversity in their student body 19 by considering other information, by reading the applications 20 more thoroughly, by trying to evaluate them less -- relying 21 less on the predicted first year GPA. And they were able to 22 ameliorate it to some extent. But, yeah, the effect is 23 enormous. 24 BY MS. MASSIE: 25 Q Other than the one instance you've described from 193, GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 53 1 are you aware of any time when the test development companies, 2 the test making companies have tried to use something external 3 to define ability as oppose to simply the results that they've 4 already achieved on a test? 5 A There are two answers to the question. There is an 6 instance on an old version of the LSAT in which they looked at 7 the correlation among the subparts of the exam. The exam had 8 more different subparts, that is, had more homogeneity in it, 9 back then. And they found a reasonably high correlation, and 10 they took that as providing some validity evidence for the 11 exam, but still it was internal, it was the subparts of the 12 exam compared to each other. 13 There are also always done as a matter of course the 14 correlation between the exam and the GPA, that is, the prior 15 grade point average, and that is sometimes taken as some 16 evidence for the validity of the exam. 17 It's a kind of funny argument because, of course, if 18 the correlation were very high you wouldn't need the exam. 19 You've got to be careful using that argument. But in fact 20 they're spared that because the correlation isn't very often 21 very high. Sometimes it's in the middle range, sometimes it's 22 in, you know, you're picking up maybe ten, twenty percent of 23 the variability. That is, there is some relationship between 24 the exam and the GPA. But typically it's very low, in fact. 25 There is an -- sorry to belabor this -- there is a GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 54 1 study done by SLAT or the LSAS on schools that had large 2 enough -- law schools that had large enough minority 3 enrollment. I think the cutoff was fifty minority students in 4 several categories, that is, African-American, or Hispanic. 5 In fact, they broke Hispanic down into Latinos also. But 6 looking at those schools that have relatively large numbers of 7 people in those groups, and they published a study showing the 8 correlations between LSAT and undergraduate GPA within each of 9 the three racial groups for those particular schools are 10 identified by code numbers. And it is interesting in many of 11 the schools the correlation within the minority racial groups 12 between undergraduate GPA and their LSAT scores is very low. 13 In fact, I mean down near no correlation at all. In fact, 14 there are a couple of places where the correlation is actually 15 negative, that is, the students who got higher grades, in 16 fact, got lower SLAT scores. This was published in 1992 by 17 the law schools. LSAT scores there was public 1992 by law 18 school, now the applicant service. 19 Q Doctor Shapiro, what is your view as the way out of this 20 circularity and maintenance of bias that you've described? 21 A Well, I think the Court asked me a question that I gave 22 the answer to before. There is no way out except to look at 23 the other information regarding each applicant and to evaluate 24 the LSAT score and the undergraduate GPA, that is, the 25 predicted first-year average in light of all those other GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 55 1 considerations. The -- to argue otherwise is really to argue 2 that you ought to take the predicted first-year GPA and just it 3 use as an absolute cut, that is, everybody above some level 4 gets in, everybody below it doesn't get in. But that's never 5 bene true. I mean, nobody ever was foolish enough to use that 6 way. In fact, when you at the practices they frequently are 7 about the admissions committee considers other attributes of 8 the student, what is this student's LSAT for somebody at that 9 economic level, for somebody of that race, for somebody from 10 that rural area. I mean, this is what's done and that's what 11 has to be done. 12 Q Do you know of any interventions into the contents of the 13 text like the one you've described for the PSAT, national merit 14 scholarships, on behalf of minority students? 15 A No, it's never been tried. 16 Q Is there any way other than speaking specifically of race 17 and the racially biasing effect of the LSAT, is there any way 18 other than taking race into account, or taking racism into 19 account in admissions that you can counter-act that? 20 A No. Actually if you think of other minorities groups you 21 even get clearer examples of that, I mean other than 22 African-Americans. If you take Hispanics where you've got a 23 large number of people who are bilingual, or for whom English 24 is not their first language, those test scores, and even the 25 GPAs must be -- I mean to make sense, must be evaluated in GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 56 1 terms of that disadvantage. I mean -- in fact, the New York 2 State Commission, the committee that I was on, was looking at 3 English as the first language as well as race and sex. I mean, 4 those are huge variables. Being bilingual is a great 5 disadvantage in these English language multiple choice exams. 6 And, of course, in the real world, being bilingual is certainly 7 and advantage, but not if you're taking the SAT or the SLAT. 8 So I mean, there's no way to use that information 9 without using all these other characteristics of the student. 10 Q Just one last question, Professor Shapiro, in your view 11 does the aggregate difference in scores by race on the LSAT 12 indicate anything about the ability of blacks and other 13 minority LSAT takers to study or to practice law? 14 A I know no evidence that suggest that, no. 15 MS. MASSIE: That's all I have. 16 MR. NIEHOFF: Your Honor, if we could just have a 17 few minutes. 18 THE COURT: I was going to take a break at eleven 19 because I have some sentencings at eleven. Do you want to 20 take a break now? 21 MR. NIEHOFF: If we could. 22 THE COURT: Okay. We will break for lunch around 23 12:30. 24 We will stand in recess. 25 (Court recessed, 10:50 a.m.) GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 57 1 (Court reconvened, 11:30 a.m.) 2 THE COURT: Okay. Ready? 3 MR. NIEHOFF: Your Honor, we don't have any 4 questions. 5 THE COURT: Plaintiff? 6 MR. PURDY: Yes, your Honor. 7 CROSS-EXAMINATION 8 BY MR. PURDY: 9 Q Good morning, Professor Shapiro. 10 A Good morning. 11 Q My name is Larry Purdy. I'm an attorney for the 12 plaintiff. And we had not had a chance to meet until this 13 morning. That is the first time we've actually got a chance to 14 shake hands and say hello to one another. Just a couple of 15 questions, really. 16 Counsel at the end of her examination asked you 17 about whether or not there have been any interventions on the 18 SLAT on behalf of minority students to help eliminate the bias 19 that you've discussed and I believe your answer was none; is 20 that correct? 21 A Yes. 22 Q You said it's never been tried? 23 A Yes, I think that's what I said. 24 Q Why haven't you tried it? 25 MS. MASSIE: Objection. Vague, your Honor. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 58 1 BY MR. PURDY: 2 Q Have you made an effort to try to make a recommendation 3 to help eliminate the bias on the SLAT as you understand it in 4 terms of its impact on minority applicants in particular? 5 A No, I never had the pretension of thinking that, that I 6 would have an influence on the construction on the LSAT. 7 Q You mentioned earlier to us I thought in connection with 8 the Golden Rule case -- incidentally the Golden Rule 9 case involved an insurance exam? 10 A That is correct. 11 Q As I understood it, and maybe you can correct me if I'm 12 wrong, Professor Shapiro, I thought you indicated you had dealt 13 with an ETS representative on the committee that oversaw that 14 settlement? Was I misunderstanding your testimony? 15 A No, the settlement provided that both ETS and Golden Rule 16 would have representatives on the advisory committee that 17 monitored and oversaw the settlement. 18 Q My question is in terms -- so you had contact in terms of 19 at least the Golden Rule settlement with representatives from 20 ETS; is that correct? 21 A Yes, but before the settlement. I mean, there were 22 months of negotiations. ETS was represented by Wilmer, Cutler 23 and Pickering. Golden Rule was represented by counsel. The 24 state of Illinois was represented by the Attorney General's 25 Office. ETS and Golden Rule brought their statisticians to the GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 59 1 negotiations. And this went on for months of a very protracted 2 period of time. And then after the settlement, the settlement 3 required that the data be given to the Social Security 4 Administration to be broken down by race and analyzed by race. 5 And we had to cooperate in arranging that. And then the over-- 6 the supervisory committee contained several people from EST. 7 And also ETS presented to the committee the item-by-item 8 statistical results. The committee actually reviewed items for 9 content and then reviewed the items for their statistical 10 results, and approved or disapproved the test forms constructed 11 by EST. I mean it was very intrusive, the process. 12 Q You mentioned that Wilmer, Cutler and Pickering were 13 representing ETS in this matter. You understand that Wilmer, 14 Cutler and Pickering are the lawyers representing the 15 University in this case? 16 A Yes. 17 Q Do you know what position ETS was taking in the Golden 18 Rule case with regard to whether or not there was racial bias 19 in the test? 20 A It was unknown, actually. The State of Illinois in the 21 application for taking the exam did not include a question 22 regarding race. The underlying suspicion came because the 23 adjoining state, Wisconsin, used the same exam and did, in 24 fact, inquire as to the race of the test takers. 25 I don't think ETS took a position with respect to GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 60 1 Illinois. And, in fact, there were no data as I've just said 2 until after settlement when the ETS computer files which 3 identified test takers by Social Security number were given to 4 the Social Security Administration to do an item analysis that 5 is the very same analysis that the test makers do. But it 6 wasn't until that point after the settlement that there were 7 any hard data concerning racial impact. 8 Q I apologize and maybe my question wasn't clear. Do you 9 know whether or not at any time in the litigation prior to 10 entering into a settlement ETS took a position one way or the 11 other as to whether or not the test they had developed was 12 biased against any minority group? 13 A I don't think they actually a position. The litigation 14 up to that point had been that they had gotten themselves 15 dismissed ETS, because they argued they were not a proper 16 party. They simply constructed the test, they didn't use it. 17 The trial court accepted that argument. They were dismissed as 18 a party. Golden Rule then appealed to the Illinois appellate 19 court and got a reversal, the appellate court saying that 20 because they had counseled the state in the manner that they 21 did that their action was, in fact, state action, and they were 22 brought back in as a party. At that point the settlement 23 negotiations began. I don't there was any actual claim or 24 disclaim by ETS up to that point. 25 Q Do you know what position ETS takes today of whether or GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 61 1 not the SAT, for example, is biased against minority test 2 takers? 3 A Oh, I think ETS said that because of its methods for 4 reviewing items subjectively for discriminatory content, and 5 its use of the DIF, Differential Item Functioning Method, that 6 it has attempted to insure test fairness, that is, to remove 7 unfairness from the test. Unfairness being a somewhat 8 different concept than bias, okay? 9 I do not that Educational Testing Service denied 10 that there is differential item -- I'm sorry, that there is 11 differential predictive validity to the test. That is, 12 specifically with respect to women that the test predicts 13 differently than it does for men. I don't think they deny 14 that. In fact, I believe Doctor Cole who was the president of 15 ETS for a while acknowledged that after the Sharif case. 16 I think you are correct in your implication that 17 they then say -- they don't advertise this is a bias test. 18 You are correct in that. 19 Q Do you know Wayne Comara (sp)? 20 A No. 21 Q Do you know whether or not Mr. Comara who serves in a 22 senior capacity with ETS, was retained by the University in the 23 undergraduate case, for example? 24 A I don't know anything about it. 25 Q I gather you have not been provided with the deposition GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 62 1 of Mr. Comara who was a University expert in the undergraduate 2 case to read what he may have said about whether or not the SAT 3 tests or standardized tests are biased against minorities? 4 A No, I have not read his testimony. 5 Q You made a comment that ETS takes a position, and I want 6 to understand this clearly, that ETS predicts female test 7 takers' performance differently than it does males -- 8 A No, I didn't say that provide those predictions. What 9 I'm saying is that they do not deny that the data show that 10 there's differential predicted validity for males and females. 11 The -- I'm going to say the APA standards, the American 12 Psychological Association standards for educational and 13 psychological testing -- psychological and educational testing, 14 I'm sorry, provide that whenever there is a difference in group 15 performance that the test maker is obligated to investigate 16 whether or not there is differential predicted validity. And 17 ETS has done this, and it is confirmed that there is. 18 Q Does the -- let's go back just for a second. What is 19 your position as to whether or not the LSAT or the SAT either 20 one predict the performance of minority students? 21 A It varies from institution-to-institution. If you look 22 at the data published by LSAS where they actually give the 23 correlation coefficient the differences are enormous from 24 institution-to-institution. So I don't think you can come up 25 with a general answer to that. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 63 1 Q Have you read any of the materials that have been 2 produced in this case by other experts for the University such 3 as Derek Bok and William Boland? 4 A No, I have not read their testimony. 5 Q You don't know what their position is as to whether or 6 not the SAT or LSAT may or may not be biased against certain 7 minority groups? 8 A I mean I'm familiar with them only in the general context 9 of the book which they wrote. I don't know what they said 10 about the test, specifically. 11 Q Do you know whether or not they a position in the book 12 that you just referenced that the tests, the standardized tests 13 are not culturally biased against minorities? 14 A No, I don't know one way or the other. I have not 15 actually read their book. 16 Q Have you read the law review article by former Dean of 17 the law school, Terrance Sandlow, who is now a current 18 professor of the law school? Have you ever read his law review 19 article published in May of 1999 that touched on the subject of 20 whether or not standardized tests were culturally biased 21 against minorities? 22 A No, I have not. 23 Q So you don't know what position Professor Sandlow, former 24 Dean Sandlow, would take on that; do you? 25 A No. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 64 1 Q Are you familiar with the -- let me go back. 2 You mentioned four areas in your report -- 3 incidentally, did you draft, did you actually draft the report 4 that has been marked and entered into evidence? 5 A Did I write -- 6 Q That was your work product? 7 A Yes, I sat at the computer. 8 Q Do you recall making revisions to it at some point? 9 A Well, all along -- and I added to it as time went on. . 10 This is specifically true with respect to the fact that there 11 was -- or I was led to understand that there was a deadline for 12 the submitting the report and then at some point that deadline 13 was eased, or changed, or what have you and then I added some 14 and revised it some. 15 Q You outlined four areas and I'm going to -- let me 16 briefly mention them and then I want to ask you a single 17 question about each one, hopefully a single question. You 18 mention content domain. You mention item selection. You 19 mention the attempt to validate the test which in your report 20 is called the best fitting regression equation -- 21 A That's correct. 22 Q And then you mention looking backward at the predicted 23 equations, in other words determining whether or not the use of 24 A That's correct. 25 Q Have you ever made any suggestions to -- let's start GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 65 1 with: Who is responsible for developing the LSAT test today? 2 A I don't know the actual name of the group, but it's the 3 -- there are various organizations that have very similar names 4 and they all start with LSA, and it's that group. 5 Q Have you ever undertaken to find out who it is that is 6 currently responsible for preparing the test each year that 7 it's given to prospective applicants to law schools? 8 A You mean an actual identity, no. 9 Q Well, then my question may be simple. I gather it's safe 10 to say you have never written anything to the developer of the 11 test concerning your suggestions about how to improve the 12 content domain to remove the bias you believe is present; is 13 that correct? 14 A As I said before I have not been in direct contact with 15 -- I have not gratuitously offered them my advice. 16 Q Professor Shapiro, don't you believe your advice would be 17 helpful to them? 18 A I don't think there is anything unique about what I have 19 said. I think everything I've said is obvious to any cycle 20 nutrition and, therefore, I have nothing new to tell them that 21 they don't already know. I mean, this process has been going on 22 since around 1930, and there are no mysteries as to what's 23 done. We all agree as to what how the array tests are 24 constructed, and we all know the consequences of it. I don't 25 think I have anything brilliant to add. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 66 1 Q Do you believe that the developer of the LSAT and the ETS 2 with regard to the SAT that they are intentionally keeping bias 3 against certain groups that you've described -- understanding 4 they disagree with you in that regard -- but do you believe 5 they're intentionally keeping bias against certain groups or 6 genders in the test? 7 A It's a hard question. There is no question that the 8 original intent of the LSAT was a bias attempt. I'm sorry, the 9 -- I misspoke, the SAT. I'm sorry. There is no question that 10 the original version of the SAT was developed by Carl Bingham 11 for a bias purpose. 12 I'll elaborate. The problem was very simple. There 13 were too many Jews in the Ivy League undergraduate classes. 14 And there was an attempt to reduce that number by coming up 15 with an exam that could be used as well as high school GPA 16 because the high school GPA was -- as a criterion was 17 permitting a very large number of Jewish applicants to be 18 admitted. The Jewish student population at Columbia in the 19 undergraduate colleges is forty percent. And so a test was 20 developed that was modeled after the curriculum of the prep 21 schools which I think the idea was -- the idea as it was said, 22 was -- it's difficult for me to say it because it's so absurd, 23 the idea being is that you would be able to admit fewer 24 over-achievers who were just, you know, good students in 25 public high schools and it exceeded and the number of Jewish GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 67 1 students in Columbia College was reduced to twenty percent 2 instead of forty percent. That was the beginning of the SAT. 3 Carl Bingham designed the item analysis technique. It says 4 Carl Bingham did this in the ETS technical handbook. There's 5 no mystery about it. Carl Bingham has a written a book on 6 intelligence of showing in the 1920 by his argument that 7 Nordics were more intelligent than other races. He later 8 recanted it. I don't want to demonize him. But that is the 9 beginning of the SAT and that is the beginning of item 10 analysis. 11 Now whether somebody is sitting in there today with 12 that same attitude, I don't want to think that. But I don't 13 see anybody changing it either. 14 It's a very long answer to your question, but I 15 don't want to call people a bigot. At the same time I don't 16 want to tell you that isn't how it started because that is how 17 it started. 18 Q Do you believe that the SAT test today is biased against 19 Jewish students? 20 A I think that would be very difficult to maintain if you 21 looked at the percentage of students at Ivy League schools who 22 are Jewish. 23 Q But my only question is: So the bias was removed in some 24 form or fashion -- 25 A I -- GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 68 1 Q Let me just finish -- let me back up. 2 A Maybe that's why Jewish students started to go to prep 3 schools. 4 Q My question is you began by saying that the SAT was an 5 attempt to -- an intentional intent to bias Jewish applicants 6 to college. 7 A It was an attempt to create a content that they would 8 perform worse than another group of students. There's no 9 question about that. 10 Q But that's no longer true today as you just said that, if 11 you look at the -- 12 A You know, I -- I don't think -- I can't imagine people 13 sitting around and saying, you know, who can we be bias against 14 today. But they have maintained the same past -- you know for 15 all practical purposes and the same item selection procedure. 16 Q Let me go back to the item. That's the second of your 17 four categories. I gather it's safe to say -- and I don't mean 18 to belabor this -- but you've never made any suggestions to ETS 19 or the group that is currently responsible for developing the 20 LSAT as to how to improve item selection to remove the bias 21 that you've discussed; correct? 22 A That's right. We started out -- I mean my venture in 23 this started out in the late 1970s, almost 1980. There was a 24 big flurry of activity. Changes were made. ETS instituted 25 differential item functioning. They did several things that GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 69 1 were more sensitivity to the racial and sexual issues like the 2 change in the PSAT, and so forth. And things moved along. I 3 moved on to other things. We all stayed busy. 4 And then recently there has been this concerted 5 attack on affirmative action which has the effect of if it 6 succeeds of requiring people to make admissions decisions 7 based purely on numbers. So what was not a problem so much 8 before because you could counter-act the effect of the 9 selection process by considering other aspect of the 10 applications now is a very serious problem. 11 So you are right. I mean, I was sitting there kind 12 of dormant, like everybody else through all this time. But 13 you have now re-raised the issue. That is, the attack on 14 affirmative action now makes all of this relevant and 15 important again. 16 Q Let me ask you this: Would you recommend doing away with 17 the LSAT? 18 A I've never been in favor of getting rid of any 19 information that has any useful purpose. And it's not -- it's 20 not the devil. If you understand what it is and use it 21 appropriately it's all right. I'm not on a crusade to get rid 22 of information. 23 Q Are you in favor of applying different standards in terms 24 of evaluating the LSAT simply based on ones race or ethnicity? 25 Knowing nothing more -- nothing else about a student are you in GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 70 1 favor of using different standards to evaluate standardized 2 test scores based on ones race or ethnicity? 3 A Of course, race and ethnicity is not only race and 4 ethnicity. I mean that's like saying merely or only or 5 exclusively. I don't think I've ever said that. Race and 6 ethnicity in our society like sex and other characteristics are 7 characteristics that affect what we do and how we perform. 8 They're not irrelevant. The alternative is to look at the 9 predicted first-year GPA numerically and ignore everything 10 else. And I think that's stupid. It's not only stupid, it 11 prenicious. But, you know, to look at the other things, yes. I 12 think you ought to look at all of them. 13 Q In fact, you made clear and I believe we can all agree 14 there is no school that admits students simply on the basis of 15 those two numbers; isn't that true? At least not any that 16 you're aware of or that I'm aware of. 17 A That's right. I don't know of anyone who is quite that 18 stupid. 19 Q But my question is if it's a test which you believe has 20 disparate impacts on various groups, be it women, be it on 21 certain minority groups, or certain other groups, certain 22 religious groups -- 23 A It's not a question do I believe. It does. 24 Q Then why don't you suggest a way to change to change it, 25 to try to eliminate that? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 71 1 A Well, I think I am. I think by saying you should 2 evaluate the outcome of have the test score, just like you 3 should look at a student's GPA and evaluate it in terms of what 4 that student is. Was that student working through college, was 5 that student, you know, being an athlete? All of these things 6 matter. 7 Q Professor Shapiro, isn't that true for every student 8 regardless of his or her race or ethnicity? 9 A Of course. 10 Q Have you read Professor Lempert's report in this case, 11 where he has developed an index and analyzed its predictive 12 value for the law students at Michigan? 13 A I have read his report. I don't know exactly what your 14 -- the way you're characterizing -- what it is you're asking 15 me. I have read his report. 16 Q Do you agree that the use of the LSAT and the 17 undergraduate grade point average helps to predict the overall 18 academic performance of students once they enter the Law School 19 of Michigan? 20 A You mean, does it correlate with first-year law school 21 grades? 22 Q No, sir. 23 A I know that. 24 Q The entire law school grade point average after three 25 years? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 72 1 A Oh, three-year law school grades. In some schools it 2 does, and, you know the correlation is typically lower than 3 with first year. Some schools it doesn't. 4 Q Do you know you and I don't want get into a long 5 discussion because we're going to hear from Professor Lempert 6 later in this case, have you read his report to see what is 7 that he says the predictive nature of the LSAT and the 8 undergraduate grade point average with regard to all students 9 including minority students who enter the Law School of 10 Michigan? 11 A I have read his report. I don't remember the details 12 about it. I mean, you would have to ask me something very 13 specific for me to answer. 14 Q You don't recall whether or not it's Professor Lembert's 15 view that the undergraduate -- that the index, the combination 16 of the LSAT and the undergraduate grade point average is a 17 strong predictor of law school performance academically? 18 A Could you tell me what you mean by strong? That's a 19 subjective term. 20 Q Tell me what you think would be strong. You're a 21 statistician so -- 22 A But you see, that's just the problem. There is no 23 absolute standard for validity. Validity unlike its use in the 24 language validity in a statistical sense is a continuum. I 25 mean, it's just a value for -- you know, minus one to plus one. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 73 1 There isn't a standard. 2 Q Would you disagree with Professor Lempert if he says that 3 the law school admission test and the undergraduate grade point 4 average correlates stronger with law school grades? Would you 5 disagree with that? 6 A It's meaningless unless you tell me what the value of the 7 correlation co-efficient is. 8 Q Well, let's ask: have you ever read the law school's 9 admissions policy, in this case, the Michigan admissions 10 policy? 11 A I think so, yes. 12 Q Do you recall when you read it for the first time? 13 A It's been within the last year or so. 14 Q Was it before you prepared your report? 15 A Yes, I assume. 16 Q Do you recall that the law school admissions policy talks 17 about the index being a combination of the LSAT and the and the 18 grade point -- 19 A Yeah, but -- 20 Q Let me just finish. 21 A Oh, I'm sorry. I apologize. 22 Q We're trying to make it easier for the court reporter. 23 THE WITNESS: I apologize to you also. 24 BY MR. PURDY: 25 Do you recall that the law school admissions policy talks about GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 74 1 the index explaining twenty-seven percent of the variance of 2 first-year grade point averages? 3 A No, that's a number I recall, the twenty-seven percent. 4 When you say the index of course to me in technical jargon 5 that's the linear regression equation. Yeah, I think that is a 6 valid -- the number that's used twenty-seven percent. 7 Q Is that in your a view as a cycle nutrition a strong 8 correlation? 9 A That is the number that's comparable to a lot of -- I'm 10 going to use a technical term r squared, that is the square of 11 the of the correlation coefficient. That is comparable to a lot 12 of values that you see for various admissions tests in various 13 universities. Whether you want to say that predicting 14 twenty-seven percent of the variance at the cost of huge 15 disparate impact, it is -- is large or good. I mean, one could 16 say that. I don't agree. I think that is a huge social cost 17 to pay for twenty-seven percent of the variance. 18 Q Would you recommend -- strike that. 19 Do you know what weight the University of Michigan 20 gives to the law school -- the LSAT and the undergraduate 21 grade point average in terms of its admissions decisions? Do 22 you know what weight? I assume you don't, but I'm asking, do 23 you know? 24 A You mean weight in the individual committee member how he 25 or she uses that? Of course not. GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 75 1 Q Regardless of how they use it, if it has a -- if weighing 2 either of those factors has a disparate impact in your opinion 3 would you suggest that they use -- apply less weight to those 4 particular indices? 5 A I think we're playing with words. If by using -- if 6 applying less weight you mean moderating them by the use of 7 other student characteristics to make your decision, yes, I 8 agree they should. If by less weight you mean changing the 9 regression equation, no. I mean that is the least square's 10 best fitting regression equation. I mean, I'm saying you 11 shouldn't take it so seriously. 12 Q Why couldn't you -- in your view why couldn't you just 13 eliminate the LSAT from the admissions process at Michigan and 14 still be able to admit a fine class, a talented class of 15 students? 16 MR. NIEHOFF: Well, your Honor, I'm going to object 17 to the extent that the question presumes a foundation of 18 knowledge about what happens at Michigan. If Mr. Purdy wants 19 to ask the question generally, I think he can do so. 20 THE COURT: Why don't you ask it generally? 21 BY MR. PURDY: 22 Q Generally. 23 A Well, again, you see the problem is that the empirical 24 evidence differs from school-to-school, group-to-group. So 25 it's hard to give a general answer. But if you're asking me GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 76 1 should you have a one varied equation, that is, how to predict 2 GPA from let's say, predict law school GPA from college GPA, 3 no. I mean, that just leaves you in the same place except with 4 one variable instead of two variables. 5 The real issue is do you want to be bound by a 6 number that is arrived at statistically, that is, 7 probabilistically, and that accounts for twenty percent of the 8 variance. It wouldn't solve it by reducing the equation to one 9 variable and accounting for even less of the variance, maybe 10 twenty percent instead of twenty-seven percent. It still 11 would leave you with the same question, are you going to make 12 a lock-step decision based on a probability. But in fact, you 13 have all this other information from each applicant if you 14 just read the application. 15 Q Would you agree that all of that other information just 16 like the SLAT and the grade point should be evaluated for each 17 individual irregardless of his or her race? 18 A Yes. 19 Q Let me just ask you finally can you tell us what are the 20 reasons why a white student or an Asian-American applicant 21 might score poorly on the LSAT? 22 A In the first place to the best of my knowledge except for 23 a very limited of nationalities within the group, Asian 24 applicants, don't score poorly on the tests. 25 Q I'm asking what would be the reasons why you might find a GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 77 1 white or Asian-American who would score poorly on the LSAT? 2 A An individual person? 3 Q Yes, sir. An individual. These are individuals that 4 take these tests, are they not? 5 A They sure are. 6 Q So why -- what are the reasons why a white or 7 Asian-American applicant would score poorly on an LSAT? What 8 reasons would you give us for why that would happen? 9 A I don't -- I hope you don't take this flippedly. I got a 10 PhD with concentration in psychology and one of the two, more 11 than -- I'm sorry -- yeah, more than forty years ago. And the 12 one thing I've learned in psychology is I surely am not 13 presumptuous enough to try to understand why any one individual 14 person does anything in their life must less a score on a 15 particular exam, on a particular day. I mean, that's beyond my 16 ability. 17 Q Professor Shapiro, if -- whatever the reasons would be 18 which might explain why a white or Asian-American applicant 19 scoring poorly could those same reasons also be the explanation 20 for why a member of an other racial or ethnic group might score 21 poorly? Won't you agree with that? 22 A Well, you make it sound like people are fungible. I 23 don't want to answer it that way. But I don't know what you 24 mean by that. If I don't interpret your question -- 25 Q Well, if you take -- let's take -- I think the Court GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 78 1 earlier today was talking about the impact of being rural 2 versus urban, being a socioeconomic different class, and, of 3 course, all of these things cross racial lines, you understand 4 that; correct? 5 A Well, in different proportions. I mean, they're -- 6 Q Here's my question. Let's assume that you have some 7 person who do not get the opportunity for good schooling from K 8 through 12. And on top of that let's that they ran with the 9 crowd. Let's use -- we'll use a white rural student, ran with 10 the crowd, that wasn't interested in academic performance, and 11 when it came time to take the SAT or the LSAT, that showed very 12 little interest in performing well, and, in fact, when they 13 took the test they didn't perform well. Now, that could be an 14 explanation for why they would do poorly. 15 A I don't know of any such person quite frankly who took 16 the LSAT. I mean you have a non -- an academically 17 non-performance person who has no academic interests whose 18 socialization is that academics is unimportant and you've got 19 him taking the LSAT. 20 Q Fair enough. 21 A I don't know how to answer that. 22 Q Let me back up. And let's remove because I was probably 23 unfair to that student. Let's assume that he or she was a 24 victim of poor opportunities in terms of their undergrad -- K 25 through 12 education. And let's assume that even though GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 79 1 they're very interested, they worked hard, they didn't get 2 prepared as well as they should have for the test for they 3 perform poorly even though they wanted to prepare well. Can't 4 that be an explanation for why they did poorly? Poor 5 schooling, poor preparation? 6 A But you see any explanation is possible. So I mean -- 7 yeah, anything is possible. There are no impossibilities as 8 human beings. 9 Q And isn't that true for every test taker irrespective of 10 race or ethnicity? 11 A Yes. I'm mean -- not said to read one groups' 12 applications and not read the others. I don't understand the 13 question. 14 Q All right. And you do understand that are minority 15 applicants -- minority applicants whom you have placed in a 16 group where there's this tremendous gap upon average; right? 17 A Yes. 18 Q We talked about that earlier this morning. 19 A Yes. 20 Q There are minority applicants who perform very well on 21 this test; are there not? 22 A What minority. 23 Q Any minority. 24 A Well, that's not true. There are some groups for which 25 there are very, very few, a very small percentage of GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 80 1 individuals getting in the higher percentiles. 2 Q But there are -- in every group. Are you suggesting 3 there is a group where no minority performs at the highest 4 levels on these tests? 5 A The proportion of percentage can be extremely low. I 6 don't want to say no because if we back off of it for a moment 7 we -- there are -- there's even a question of whose a member of 8 a minority group. We get into all kinds of definitional 9 questions. But there are some groups for which the proportion 10 of people getting very high scores is very, very small. It's 11 not zero, but it's very small. 12 Q What is the relevance of what you were just talking about 13 the definition of a minority? What are you referring to? 14 A Well, I mean you're asking me a question in which you say 15 no, none. Okay. So in order to answer an absolute zero 16 question like that we're going to have to first decide who's in 17 that group. Of course, you know the matter is that the persons 18 who are in that group are those who say they are in the group. 19 So, I mean, it's not a good question is all I'm saying because 20 it implies a precision that we don't really have. 21 Q Well, with all due respect I'm not sure I used the word 22 and I apologize if I did. I think I said isn't it true that 23 there are members of every group that perform at the highest 24 levels on these tests. Now, are you taking the position that 25 there are none? GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 81 1 A I'm telling you the number is very, very small. 2 Q And I want to go back, why does the definition of who's 3 in these groups matter to you? 4 A Well, I just thought it matter -- it doesn't matter to 5 me. I thought it matter to you in the way you were asking the 6 question. You know, I don't need to define who's a member of 7 what group. 8 Q The final -- let me ask you if I could, If I could direct 9 to the last page of your report. I think it's 170. 10 A Okay. 11 Q Okay. You say -- let me just read it into the record, 12 and certainly correct me if I'm wrong. 13 Barring the discontinuance of reliance of the 14 regression equation, using admission test score and previous 15 grade point average for admissions decisions, it is reasonable 16 to view affirmative action as the only available antidote to 17 test development procedures which have reduced 18 African-American enrollment to zero or near-zero levels and 19 have substantially reduced the enrollment of other minority 20 groups." 21 A You read it correctly. 22 Q Now there I gather that you are suggesting however that 23 there be a discontinuance on this regression equation, are you 24 not? 25 A No. In a sense of using it I am saying it in the sense GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 82 1 of relying on it as a true valid predictor. So -- 2 Q Let me just ask -- 3 A I mean we're quibbling about the use of a word 4 "reliance." 5 Q No, no, I'm just reading your language which you said 6 barring the discontinuance of reliance on the regression 7 equation it's reasonable to view of affirmative action, I mean 8 you know in this case, of course, what we're talking about is 9 an explicit consideration of race for certain groups. You 10 understand that, do you not? 11 A No, I don't understand that. 12 Q You don't understand that the consideration of race is 13 considered for three particular groups in the University of 14 Michigan Law School? 15 A I don't understand what you mean by the consideration of 16 race. 17 Q Race is a factor that's considered the admissions -- 18 A Among other factors, yes, I understand that. 19 Q For three groups, not all groups, you understand that? 20 A Which -- 21 MR. NIEHOFF: Objection, your Honor. There is no 22 foundation that he actually knows about the policy. 23 THE COURT: He's asking. 24 MR. PURDY: 25 Q He said he read the files. You told us you read the GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 83 1 policy. 2 A I did read the policy and my answer I'm trying to 3 reconstruct in my own mind who -- which three you're referring 4 to. 5 Q African-Americans, Hispanics, and Native Americans. 6 A Okay. 7 Q You understand that race is factor considered for those 8 three groups in the admissions policy? 9 A Yeah, that makes sense. 10 Q You say "barring the discontinuance of reliance on the 11 regression equation " you go on, "it's reasonable to use 12 affirmative action as the only available antidote so that 13 certain groups such as African-Americans, and others aren't 14 reduced." 15 That's what you say, correct? 16 A Yeah, reliance meaning this -- where I say "barring the 17 discontinuance of reliance, I'm talking about using it in a 18 lock-step fashion. Let me just say one thing about that. As I 19 understand attacks on affirmative action in general and in this 20 case as well, it essentially is the argument that I've got a 21 higher predicted GPA than this person who was admitted and I 22 wasn't admitted. And that's what I'm talking about here when I 23 say reliance. It is as though that predicted equation, that 24 regression equation is the measure of quality. 25 Q Let me just correct you about this case. That is not the GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 84 1 position have the plaintiffs in this case. The position of the 2 plaintiffs in this case is that whatever factors are considered 3 in other words, for example, if the University wishes to 4 completely eliminate consideration of the SLAT, that's fine 5 with the plaintiffs in this case. If the University wants to 6 consider eliminating entirely the use of the undergraduate GPA, 7 that's fine with the plaintiffs in this case. If the 8 University wants to consider eliminating letters of 9 recommendation or consideration of outside activities, all of 10 those are fine with the plaintiffs in this case. The only 11 question is this case is whatever consideration, whatever 12 factors are going to be considered, ought to be fairly 13 considered irregardless of race. So -- just so you understand 14 now, this case, all right? Now, we're tracking? Do you 15 understand where I'm coming from? 16 A I understand what you're saying. I have trouble telling 17 the distinction between you said and what I said. 18 Q There is a distinction and we can argue about that. My 19 question is: If you assume the University is going to continue 20 to rely as they have in the past on this index are you 21 suggesting then that affirmative action, i.e, the 22 consideration, the explicit consideration of race must go on 23 forever? 24 A Forever? That's a long time. 25 Q Well, assume the university is going to continue to rely GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 85 1 on the index in the fashion that it currently does, can you 2 tell us as an expert how long you think affirmative action will 3 be needed as an antidote to preclude the problems that you set 4 forth in your report? 5 A That's easy. When the selection procedure stops having 6 an adverse impact then you can stop searching for alternative 7 selection procedures with less adverse impact. 3. 8 Q Do you have any -- 9 A I have no trouble with that. 10 Q Can you tell us in your role as an expert how long that 11 might be before that would occur? 12 A I don't think it's time bound I think it's data bound, 13 whenever it happens. That is, my understanding, you know, is 14 very simple. A procedure has adverse impact. The question then 15 is, is it a valid selection procedure. The question then is 16 there an alternative less discriminatory valid selection 17 procedure. 18 Q Do you believe there is a less discriminatory valid 19 procedure? 20 A This I think -- you know, very simply using the 21 regression equation together with other information is more 22 valid, certainly not less valid, and certainly has less 23 discriminatory impact. 24 Q Would you be able to use that regression equation in a 25 different manner and exclude the consideration of race and GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 86 1 accomplish what you are after? 2 A I don't know what you mean by the exclude the 3 consideration of race. 4 Q Let's assume you remove the question on the application 5 form so that no one knows when an application comes, the only 6 change you've made is you removed the box that identifies a 7 person's race or ethnicity. If you change the manner in which 8 you relied upon the regression equation, in a way that you 9 would feel comfortable with, would that eliminate the 10 consideration of race in the process? 11 A You know that's fairytale. I can look at a student's 12 name, I can look at what school they went to, I can look at 13 what high school they went to, I can look at the zip code and 14 their address, and I can with fairly good certainty tell you 15 what their race is. You cannot remove as a piece of 16 information in the United States. So to say rely on everything 17 else but, and take it off the application, is a dream. It's 18 just unrealistic. 19 Q Is there in your opinion no way to readjust the manner in 20 which you used the regression equation in order to eliminate 21 the need for consider race in the admissions process Or does it 22 even matter based on what you just said? 23 24 25 GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 87 1 A Well, if you got people making the decision, they're 2 going to act like people. And they make the decision based on 3 the information they either actually see or inaccurately 4 suppose. I mean, this is not machine making this decision. 5 Thank goodness it isn't. And you are not going to remove that 6 information and presumably you're not going to keep people from 7 using it however they individually use it, whether there's a 8 policy or not. 9 MR. PURDY: That's all, your Honor. 10 THE COURT: Anything? You may step down. Thank you 11 for coming. 12 Okay. We'll stand in recess until 2:15 p.m. 13 (Court in recess, 12:45 p.m.) 14 15 16 17 18 19 20 21 22 23 24 25 GRUTTER -v- BOLLINGER, ET. AL. BENCH TRIAL - VOLUME 8 TUESDAY, FEBRUARY 6TH, 2001 88 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 GRUTTER -v- BOLLINGER, ET. AL. 88 1 Afternoon Session 2 -- --- -- 3 THE COURT: Okay. Next witness. 4 MS. MASSIE: Call Jay Rosner. 5 THE COURT: Mr. Rosner, you're still under oath. 6 MS. MASSIE: Judge, I do apologize. I'm slightly 7 behind the ball here because we got a bit behind on the 8 security check. I need to pass some stuff out to you guys. 9 THE COURT: No problem. Take your time. 10 MS. MASSIE: For everyone's information, these are 11 proposed Exhibits 201 through 210. Those were handed out 12 before, but we thought it would be prudent to make some 13 extra copies. There is also a demonstrative exhibit that 14 we're going to try to get admitted. 15 And I think the people in the gallery, copies should 16 be coming. 17 THE COURT: You have changed attorneys. 18 MS. MASSIE: Yes. Sorry, Judge, and I had raised 19 that with Mr. Payton and Mr. Kolbo. 20 THE COURT: I don't mind. That's fine. I was 21 waiting for Mr. Washington. 22 MR. WASHINGTON: I'm taking it easy today, Judge. 23 MS. MASSIE: I think we have reached the state of 24 resonance, and I apologize to everybody for the holdup. 25 Hello, Mr. Rosner. 89 1 THE WITNESS: Hello. 2 MS. MASSIE: I would like to start, if it's okay 3 with you, Judge, by reading very, very briefly from the tail 4 end of Mr. Rosner's testimony last time just to make it 5 relate, because I think we're going to pick up exactly where 6 we left off. 7 THE COURT: Go on. 8 MS. MASSIE: This is you testifying, Mr. Rosner. 9 "I hear from educators all the 10 time, the student is so wonderful. A 11 minority student who in class --" 12 This is page 208 from Volume 7 of the trial 13 transcript, for everyone. 14 "I hear from educators all the time, 15 the student is so wonderful. A minority 16 student who in class has got the papers, 17 responds really well, and has mediocre 18 test scores. How can that be? The answer 19 is very common and very understandable. 20 Different skills." 21 And then the Court interjected: "For everybody, 22 not just minorities." 23 And you responded to the implied question: 24 "For everybody, although the way the 25 skills are measured is different and 90 1 we will talk about that when we come to 2 do some specific test questions." 3 DIRECT EXAMINATION (Continued) 4 BY MS. MASSIE: 5 Q So I would like to turn now to your testimony on 6 specific test questions and I should start by asking you 7 whether people in different racial groups have different 8 outcomes as racial groups on different standardized test 9 questions. 10 THE COURT: First you better ask him and lay a 11 foundation, because his expertise -- I mean, I let him kind 12 of go on a lot last time, and in fact, I thought about it 13 and thought about it and probably let him go on too much. 14 From now on we're going to answer the questions. 15 And I haven't heard any -- why don't you lay a 16 foundation if he has expertise in that area. 17 MS. MASSIE: I'm happy to do that for this specific 18 purpose. 19 BY MS. MASSIE: 20 Q Have you studied the ways in which SAT's and LSAT's 21 and other similar standardized admissions tests have an 22 impact on minority test takers? 23 A Yes. 24 THE COURT: How have you studied them? 25 THE WITNESS: In a number of ways, Your Honor. 91 1 THE COURT: First of all, have you had any training, 2 statistical training or anything of that nature? 3 THE WITNESS: I don't have any formal statistical 4 training, no. 5 THE COURT: Go on. How have you studied it? 6 THE WITNESS: I have studied this topic from working 7 myself with many dozens of SAT test forms and many dozens of 8 LSAT test forms, looking at those forms, analyzing them, and 9 then working with majority students and minority students on 10 individual test questions in the context of test 11 preparation, and I have done many, many hours of that. 12 I have also had discussions, particularly in recent 13 years, with minority students before they took the test, 14 while they were preparing for the test, and then I have had 15 discussions with minority students after they have their 16 test scores. So it's a continuum of information involving 17 the test questions themselves, the interaction of minority 18 students with the test questions, and then the results that 19 minority students get when they get the test scores. 20 THE COURT: Did you compile this data, did you write 21 it down, did you do a paper on it? 22 THE WITNESS: I have written -- I have written a 23 couple of articles on these topics. I speak regularly to 24 organizations about these topics. In terms of, I think 25 perhaps what you're asking is, have I done a statistical 92 1 analysis of this -- 2 THE COURT: That's not what I'm asking. 3 THE WITNESS: Then I misunderstood. 4 THE COURT: Is there a paper I can read that you 5 have done on this? Is there anything other than your 6 anecdotal discussions with people? 7 THE WITNESS: Probably the -- there is a test that 8 we will talk about that I have compiled that, in the context 9 of this testimony, that I just compiled a couple of weeks 10 ago and that's, I think, the only thing. 11 THE COURT: Just compiled a couple weeks ago? 12 THE WITNESS: Yes. 13 THE COURT: Go on. Let see where he goes and see if 14 it helps in the qualification, and I think he is qualified 15 to do certain things, but let's see. Go on. 16 MR. KOLBO: Your Honor, if I can just make it clear, 17 we have an objection on foundation. We don't -- we believe 18 this witness hasn't been qualified as an expert on test 19 design and psychometrics and -- 20 THE COURT: I agree with you, and that's what I 21 thought we were getting into. 22 In terms of him teaching the class, in terms of who 23 comes to his class, he is obviously well qualified for that, 24 he has been on campuses, he goes out, part of his job was 25 to start a whole division out in Seattle in terms of getting 93 1 students, but in terms of the other kinds of areas I haven't 2 heard anything other than he has the same degree that I 3 have. 4 MS. MASSIE: But he doesn't have to, as an expert, 5 to be qualified as an expert, he doesn't have to have 6 specialized academic training. I think we agree about that. 7 THE COURT: I absolutely agree, but he has to have 8 more than anecdotal kinds of information if he is going to 9 testify as an expert. 10 Anyhow, let's go. Let me -- we heard, we have heard 11 from experts, so we know what they are, including 12 Dr. Shapiro who testified before, who by training, by every 13 kind of imaginable thing has the expertise to testify in 14 testing areas and statistics and so forth. Go on. 15 The only reason I mention it, I'm looking at your 16 exhibits and -- 17 MS. MASSIE: I think Mr. Rosner's testimony will 18 build on Professor Shapiro's. 19 THE COURT: It may build on it, but he's got to have 20 some expertise other than the fact that he is a lawyer and 21 he works for a company that gives classes on testing. 22 MS. MASSIE: Let me try, continue trying to 23 establish that. 24 THE COURT: Sure. 25 BY MS. MASSIE: 94 1 Q You have trained teachers? 2 A I have trained test preparation instructors, yes, to 3 teach SAT courses and LSAT courses and other courses. 4 Q And you have made a study of test items and their 5 impacts on minority students as a part of your job 6 responsibilities? 7 A I have, over the last five years, not in the more 8 formal sense that the Judge was talking about in terms of 9 writing a journal article, but it's my job to try to 10 understand how minorities respond to and react to and can 11 answer better individual test questions. That is a central 12 part of my job. 13 THE COURT: Central part of your job now that you 14 have become the Director of this Institute, and you became 15 the Director of this Institute when? 16 THE WITNESS: In 1995, Your Honor. 17 Not -- actually, it predates that, because it was 18 also part of my job when I was teaching predominantly white 19 students in Seattle in the late '80's to understand how 20 those students in a test preparation context would better 21 answer individual test questions. That's what a test 22 preparation person does. 23 And my focus has been on minority students since 24 '95, in that regard. 25 BY MS. MASSIE: 95 1 Q And Mr. Rosner, you seek through the Foundation to 2 make an intervention in testing bias and the test score gap; 3 correct? 4 A Yes. 5 Q And that means seeking to understand it; correct? 6 A Yes. Understanding the test score gap and the 7 components that go into it and the questions that feed into 8 it is part of trying to reduce that gap in my work with 9 minority students. 10 Q And that means trying to understand what contributing 11 causes are to the test score gap; correct? 12 A Yes, as best I can, yes. 13 Q And that means being able to make clear on the basis 14 of hard information to minority students when you speak with 15 them that the test score gap is not a product of their lack 16 of qualifications or lack of capacity; correct? 17 A Yes, that's right. 18 Q Tell us how you know that there is a test score gap. 19 A Well, anyone who looks at the average results of tests 20 like the SAT or the LSAT sees that there are very disparate 21 impacts on under-represented minority populations, 22 particularly African Americans and Latinos. There is also 23 gaps between men and women, as Professor Shapiro pointed 24 out, but the focus of my work has been primarily on African 25 Americans students, secondarily on Latinos students. 96 1 Q And do you agree with Professor Shapiro that the 2 characteristics of this gap are broadly similar across the 3 different standardized admissions tests widely used in the 4 United States? 5 A Yes, the gaps, if we focus on the SAT and the LSAT, 6 for example, the gaps are persistent. They are consistent 7 and they operate the same way, and have, between whites and 8 blacks, for example, as they have for the last ten or twenty 9 years. They are also consistent on the other most common 10 admissions tests, like the GRE, et cetera. 11 Q I would like to turn your attention to the question of 12 -- to the matter, to avoid the word I was about to use -- 13 of question selection, item selection, because I think 14 you'll be able to concretize for the Court some of the 15 points that Professor Shapiro was making earlier today. 16 A Yes. I have been concerned about item selection for 17 years, and have acquired some data recently that I think 18 illuminates concerns that I have had for a long time about 19 how test items are selected. 20 Q And why have you been concerned about it? 21 A I recall an article that I wrote in 1993 that focused 22 on test question selection, and in that article I used the 23 distinction between males and females, but in thinking about 24 these topics after that article it became apparent to me 25 that the much more significant test score gaps are between, 97 1 for example, whites and blacks. I think your question was 2 why, and the answer is, it relates to my work. 3 THE COURT: From now on, please answer the question. 4 THE WITNESS: Yes. 5 BY MS. MASSIE: 6 Q Mr. Rosner, if I could direct you to Exhibit 202. 7 MS. MASSIE: So everyone knows, we expect to 8 introduce Exhibit 201 through a subsequent witness, that's 9 why we're skipping it. 10 BY MS. MASSIE: 11 Q Tell us what's going on with this exhibit, if you 12 would. 13 A Sure. Very briefly, I wrote Black or White Preference 14 Question at the top. That's something that I just raised 15 for an issue. This is an antonym question, an SAT antonym 16 question. 17 You'll notice the instructions, chose the opposite, 18 and what the test taker is supposed to do is to chose the 19 word that is the best answer for a word that is opposite in 20 meaning to the word, reneg, and very briefly, you'll see 21 that the answer C is in italics. That was in italics in the 22 San Francisco Chronicle, which was the source of this 23 particular information. 24 And so the correct answer is C, but that's not what 25 is really significant about this question. What is 98 1 significant is what I have in smaller print on the bottom, 2 and that is the results are that African Americans do 3 significantly better defining the antonym than do whites. 4 That's a quote from the San Francisco Chronicle, citing ETS. 5 And because African Americans do answer this question 6 correctly at higher rates than whites, I call this a black 7 preference question, and a white preference question has the 8 opposite meaning, that -- 9 THE COURT: How would you know this, other than you 10 saw it in the San Francisco Chronicle? 11 THE WITNESS: Well, that's the point, Your Honor, 12 you would not. 13 THE COURT: What special knowledge do you have 14 concerning this other than you saw it in the San Francisco 15 Chronicle? 16 THE WITNESS: The special knowledge -- 17 THE COURT: Have you done any testing of your own, 18 done anything like that? 19 THE WITNESS: I have talked to minority students. 20 THE COURT: My question is, have you done any 21 testing? 22 THE WITNESS: No. 23 THE COURT: Your source is the San Francisco 24 Chronicle? 25 THE WITNESS: Yes, whose source is Educational 99 1 Testing Services. 2 THE COURT: Did you go to them and talk to them 3 about it? 4 THE WITNESS: No, I didn't. 5 THE COURT: All right. 6 BY MS. MASSIE: 7 Q I see that this question was rejected for use on the 8 SAT? 9 A Yes, this was a question that was rejected for use on 10 the SAT. 11 THE COURT: How do you know that? 12 THE WITNESS: Again, that's -- that was quoted in 13 the San Francisco article citing ETS. The purpose of the 14 article -- 15 THE COURT: So your source is the San Francisco 16 Chronicle? 17 THE WITNESS: Yes. 18 THE COURT: You didn't go back to them and say, hey, 19 I have to testify in court. Did you reject it? Why did you 20 reject it? Is there a reason you rejected it? Did you use 21 it later? 22 Did you do any of those things? 23 THE WITNESS: No. 24 THE COURT: Yes or no? 25 THE WITNESS: No, I didn't. 100 1 THE COURT: You may continue. 2 BY MS. MASSIE: 3 Q And does this question -- just looking at this 4 question on the face of it, would you imagine that there 5 would be a significant gap in performance? 6 A This question looks like a typical SAT question. I 7 have seen many hundreds of antonym questions. There would 8 be no way that looking at this question -- I would like to 9 find a way, but there is no way that I could tell that this 10 is a black preference question by looking at the face of the 11 question. 12 Q And Mr. Rosner, what's the source of the gap in 13 performance on this question? 14 A The source of the gap -- well, this question is 15 clearly capturing something about race, because racial 16 groups answer it in disparate fashions. Beyond that, 17 despite my expertise and working with lots of minority 18 students, I can't be more specific as to exactly what about 19 race this is capturing. 20 Q It's capturing something that we don't yet know how to 21 describe or define? 22 A Something that I can't yet describe. 23 THE COURT: The only reason you know that is because 24 you read it in the San Francisco Chronicle? 25 THE WITNESS: That's right, Your Honor. 101 1 BY MS. MASSIE: 2 Q Why is it -- is it easy, Mr. Rosner, to get pretested, 3 rejected questions, questions that are pretested and then 4 not use on a scored test? 5 A No, it isn't. 6 And in fact, Your Honor, in partial answer to an 7 earlier question, I have asked Law Services for these 8 kinds -- this kind of data information, and they say there 9 is none publicly available. I haven't asked ETS about SATs. 10 I asked Law Services about LSAT questions. 11 THE COURT: But you know that Dr. Shapiro testified 12 that he was able to get some of that information through 13 litigation, not through -- 14 THE WITNESS: Yes, and he is -- 15 THE COURT: That's all. 16 THE WITNESS: Yes. 17 THE COURT: Go ahead. 18 BY MS. MASSIE: 19 Q Let me ask you to look at Exhibit 203. 20 Tell us about that question briefly, if you would. 21 A Okay. Very briefly, this is a sentence completion 22 question. It's another SAT question. The objective here is 23 to pick from among the five answers the pairs of words that 24 best fit into the meaning of the sentence with the two 25 blanks. Again, I note at the bottom the correct answer is 102 1 C, and again that's not what is most significant about this 2 question. 3 Most significant about this question as it pertains 4 to this case is that eight percent more African American 5 than whites answered this question correctly, again, a quote 6 from the Wall Street Journal, which again cites Educational 7 Testing Services. 8 And the disposition of this question, as the last 9 question, was this is a black preference -- again, my label 10 entitled a black preference question that was also rejected 11 for use on the SAT after pretesting. They told us in the 12 article that the pretesting was done in 1998. 13 Q Looking at this question, Mr. Rosner, does it seem 14 like a typical SAT question? 15 A Same as the last question, this looks to me like a 16 typical sentence completion question. I have seen hundreds 17 of these on the SAT. Yes. 18 Q And do you have any greater ability to explain in this 19 instance how the question captures something about race in 20 our society? 21 A No, same answer as to the last question. 22 Q Okay. Let me direct you to Exhibit 204. Tell us what 23 this is about, if you would. 24 A Yes. This is a math question, a math SAT question. 25 It's an algebra question, because it uses X as a variable, 103 1 and the question asks about the square root of a number. 2 The correct answer here is C, and again, that's not what is 3 really important here. 4 What's really important is the results which I 5 quoted from the Wall Street Journal, that seven percent more 6 African Americans than whites answered this question 7 correctly. It's a black preference question. It also is 8 rejected for use on the SAT after pretesting, as I have 9 written on the bottom there. 10 Q Looks like a normal question. 11 A Looks like any one of hundreds, many hundreds of 12 algebraic questions that I have seen on the SAT. 13 Q And do you have any idea what it is about this 14 question that -- 15 A Same answer, it's capturing something about race 16 generally, but I can't be specific despite my experience as 17 to what it's capturing. 18 By the way, for students in the room, if I can just 19 say -- 20 THE COURT: We're not here -- we're just here to 21 answer the questions. 22 THE WITNESS: Okay. I'm sorry, Your Honor. 23 THE COURT: But since you piqued their interest, you 24 can tell them now. The rule is -- you're a lawyer, and you 25 know the rules, but go on, since you've piqued mine, too. 104 1 THE WITNESS: Okay. One sentence. 2 If X equals one half, you can get the correct 3 answer. If you don't see that, there is a little trick 4 built into the question. You just won't get the correct 5 answer. I didn't want students to be puzzling about this 6 for a couple of hours. 7 THE COURT: I agree with you, but it's -- 8 MS. MASSIE: I'm hoping that my inability to 9 understand what you just said doesn't predict failure in my 10 future. 11 THE COURT: You don't have to take the test again, 12 so you don't have to worry about it. 13 BY MS. MASSIE: 14 Q You have just shown us three rejected pretested 15 questions on which black test takers outperformed white test 16 takers? 17 A That's right. 18 Q And in general, those kinds of questions aren't 19 available, that's why you're using data from a newspaper, 20 true? 21 A Very few of these questions are available. None are 22 available for the LSAT, very few are available for the SAT. 23 THE COURT: They must be somewhat available. Two 24 newspapers were able to receive them and write pretty 25 lengthy stories, it looks like, but go on. 105 1 BY MS. MASSIE: 2 Q Have you seen, other than these questions, have you 3 seen a lot of rejected pretested questions? 4 A On the SAT, I have seen approximately twelve pretested 5 questions that were rejected and those were all questions in 6 these two newspaper articles. 7 THE COURT: Did you ever -- you know, Professor 8 Shapiro was kind enough to tell us they put out this paper 9 about every test. Have you ever read one of those? 10 THE WITNESS: No, but what he didn't say about that, 11 Your Honor, is that they don't have the statistics for each 12 individual question. They -- 13 THE COURT: Do you know that? 14 THE WITNESS: They have the statistics for the test, 15 as I understood what he was saying. 16 THE COURT: But you have never seen one, you have 17 never asked for one? They obviously were pubic. He was 18 able to get them. 19 THE WITNESS: Yes, I have never seen one, never 20 asked for one. 21 THE COURT: Never knew they existed before you heard 22 his testimony, did you, or did you? 23 THE WITNESS: Not -- I know they make -- they make 24 information available. I don't know that that specific 25 information was available to that extent. 106 1 MS. MASSIE: Well, Professor Shapiro didn't say they 2 had item specific information, right? 3 THE COURT: Well, I heard his testimony. I wanted 4 to know if he heard of them. I don't know what's in there. 5 I just know what Professor Shapiro testified, is all. 6 Go on. 7 BY MS. MASSIE: 8 Q Mr. Rosner, let me ask you this: Is it only black 9 preference questions, to use your phrase, black test takers 10 that outperform white test takers, that are rejected by ETS 11 for use on tests? 12 THE WITNESS: No. 13 MR. KOLBO: Objection again on foundation. 14 THE COURT: He is testifying what he read in the 15 newspaper, and with all due respect to the newspapers, they 16 are my favorite thing to read every morning and every night, 17 and I was just on an airplane and I couldn't get through an 18 airplane trip without four or five newspapers, but his 19 knowledge is limited to what he reads in the newspapers. He 20 said that he has never tried to get the information. I 21 don't -- we're sitting here reading the newspaper. 22 MS. MASSIE: Judge Friedman, he has made a study of 23 the sources, the character, the impact of the selection of 24 these kinds of questions. 25 THE COURT: I'll give you a little more latitude, 107 1 but he even just testified that he doesn't know why the one 2 group or other group would do it other than what he read in 3 the Wall Street Journal or that he read in the Chronicle. 4 MS. MASSIE: But that's because this society hasn't 5 finished understanding race, that's the whole thing. 6 THE COURT: The whole thing is, you're right, except 7 he is here as an expert. I mean, an expert doesn't get 8 their data out of the newspaper. 9 MS. MASSIE: But Judge, these questions look like 10 you, anybody reading these questions, would think there is 11 not a racial bias in that question. 12 THE COURT: I agree. 13 MS. MASSIE: There is no insult there. There is 14 nothing in it that's a reality distractor. 15 THE COURT: Absolutely. If you said, these 16 questions, do you think there is a bias, I would say I had 17 no idea, because, you know, I probably couldn't answer them, 18 but that's not the issue. The issue is that all he is doing 19 is he is telling us what a newspaper article had to say 20 about these and he can't -- he does not know why and he said 21 he doesn't know why. He has been very honest. 22 Hundreds of thousands of others ones, he said -- 23 MS. MASSIE: Judge Friedman, there aren't released 24 pretest items that are publicly available. 25 THE COURT: Okay. Go on. 108 1 BY MS. MASSIE: 2 Q Do they reject questions on which white test takers 3 outscored black test takers as well or do they only 4 reject those -- 5 THE COURT: Or do you know? 6 THE WITNESS: I do know. 7 THE COURT: Okay. 8 THE WITNESS: And they reject white preference 9 questions, also. 10 BY MS. MASSIE: 11 Q Tell us how you know that and please explain. 12 A Because of the dozen problems in the newspaper 13 articles, there were about seven or eight white preference 14 questions that were also rejected. 15 Q Do you have any other source of information on that 16 proposition? 17 A No, that is the entirety of information that I have, 18 that I have with regard to pretested rejected questions. 19 Q And how about on the scored sections, do they only 20 include white preference questions? 21 A Well, a later exhibit will demonstrate that. The 22 answer is, on the scored sections, the scored sections are 23 virtually entirely comprised of white preference questions. 24 Q Please tell us which exhibit you want us to take a 25 look at. 109 1 A The next one, labeled 205. 2 And this was the study that I did that I referred to 3 earlier, Your Honor. 4 Q Tell us what's going on on this sheet of paper. 5 A Sure. I was able to obtain data on four SAT's, 6 actually, through Professor Shapiro, who had this data. 7 These were four SAT's administered to all students in the 8 State of New York in 1988 and 1989, for test forms at four 9 administrations, and this data had specifics for individual 10 questions, and that's what we're talking about, and that's 11 what I'm looking for. 12 And so I looked at the 580 questions that comprised 13 the four SAT's and I simply counted the number of white 14 preference questions, the number of black preference 15 questions, because I had the data of the percent answering 16 correct, rate for whites and blacks. That was in the data 17 set which is appended to my supplemental report. And I was 18 able to count only one black preference question and 574 19 white preference questions. 20 You'll notice a category, if I may continue, you 21 will notice a category on the bottom, it says, no preference 22 questions. Those were questions that were answered at 23 exactly the same percentage rate by whites and blacks. 24 Q And how many students took the tests that are the 25 basis of this count? 110 1 A The four tests were taken by in excess of 200,000 2 students. This is a very, very substantial data sample. 3 Q 200,000 students each? 4 A No, combined, 200,000 students. Approximately 90,000 5 students took two of the four tests. Another test was taken 6 by 20,000 students, and the final test was taken by 11,000 7 students. 8 Q Did you do any other analyses of the impact of these 9 four test forms on minority students? 10 A Yes. That would be the next exhibit. 11 Q And tell us about this. 12 A Very briefly, this, I mentioned before I had the 13 percent correct. The data set included the percent correct 14 for each question of whites, the percent correct for each 15 question of African American students, and the percent 16 correct for each question for Hispanics. 17 So the last sheet I did a white/black comparison. 18 For this sheet I did a white/Hispanic comparison, and where, 19 as I said before, the white answering percentage was higher, 20 that was a white preference question. Where the Hispanic 21 answering correct percentage was higher, that was a Hispanic 22 preference question. 23 And the breakdown is that of the 580 questions, I 24 found 11 Hispanic preference questions and 566 white 25 preference questions, with only three neutral, which I call 111 1 no preference question. Again, answered rates of whites and 2 Hispanics were at the same percentage for those three 3 questions. And these were all questions selected for use on 4 actual SAT's in the prior exhibit. Those were all -- these 5 are the same 580 questions selected for use on actual SAT's 6 administered to students. 7 Q So in other words, they are not questions that were 8 rejected after pretesting? 9 A No, they were in fact questions that were selected 10 after pretesting. 11 Q Do you have those questions? 12 A I managed to get those questions, yes. 13 Q And you have some here, of course? 14 A That is, I managed to -- I had the answering correct 15 data only and then I managed to get the content of all of 16 those questions. 17 Q And were you able to take a look at that? 18 A Yes. Then I studied the content of those questions 19 with regard to the answering percentages. 20 Q You have some here? 21 A The next exhibit is one of those questions. 22 Q That's the one that says oblivious at the top? 23 A Yes. And it's an antonym question like we discussed 24 before. Would you like me to briefly go over what's on 25 here? 112 1 Q Please do. 2 A Okay. Again, the test taker is to chose the opposite. 3 The correct answer is C. And the results of this question, 4 which were in the data set, is that 27 percent more whites 5 than African Americans answered this question correctly. 6 This was on the SAT form that I note there in the -- toward 7 the bottom right-hand corner. 8 And the disposition here, this was one of the 574 9 white preference questions to appear on these four tests in 10 doing a white/black comparison, and this question happens to 11 be the verbal white preference question with the largest 12 percentage difference. This is a large -- not only a large 13 gap between whites and blacks, but the largest gap for the 14 verbal questions. 15 Q Is this question supposed to be, in the eyes of ETS, 16 an easy or a hard question or a medium question, if you 17 know, and how do you know? 18 A Yes, this was an easy question. I know that, one, 19 from its placement on the test, and two, because the correct 20 answering percentage for whites was 83 percent, which is an 21 easy question. 22 Q So -- 23 A At least for whites it was an easy question. 24 Q Say more about the question number, the question 25 placement, I'm sorry. 113 1 A That was question number four out of ten to fifteen 2 antonym questions, and the questions in the one, two, three, 3 four positions on the test are easier questions, typically, 4 answered by 70 -- answered correctly by 75 percent or more 5 of test takers. 6 Q Here, is there anything, looking at this, at this 7 question on its face that let's you -- enables to you 8 explain how it is that it has such a disparate effect on 9 test takers? 10 A No, nothing on its face. I would have to look at 11 statistics. 12 Q Let's turn to -- 13 THE COURT: Let me just ask one question. 14 I note that's 1988. If we were to believe Professor 15 Shapiro, he indicated something to the effect that after 16 each test they go over and they analyze each question in 17 terms of different things, one of them being, you know, to 18 improve upon it, I forget exactly his words, but he was very 19 eloquent. 20 Would twelve years make a big difference, or do you 21 know? 22 THE WITNESS: Not in my experience, having worked 23 with students and looked at the results. These things, 24 particularly the SAT is extremely constant from year to 25 year. 114 1 THE COURT: In terms of results, but you have never 2 looked at -- these are the only individual questions you 3 have ever looked at? 4 THE WITNESS: These are the only individual 5 questions, Your Honor, I have looked at where I have the 6 percentage answering rates for whites versus blacks. 7 THE COURT: Go on. 8 BY MS. MASSIE: 9 Q So we're on 208, am I right? 10 A Yes. 11 Q Tell us about this. What is this question about? 12 A This is a math question from the SAT. It's a 13 particular, unusual kind of question called quantitative 14 comparison, that's the question type, and you're given two 15 values. 16 If you look down in the middle of the page, you will 17 see Column A and Column B, and the task for the test taker 18 is to decide whether Column A is greater, Column B is 19 greater, whether the two quantities are equal or whether 20 there is not enough information to answer the question, and 21 the answer choices, A, B, C, D, reflect those different 22 decisions by the test taker. 23 Q Tell us what is of interest about this question for 24 the overall test score gap. 25 A Well, what's interesting, certainly the correct answer 115 1 is D, what's interesting about this question is that as I 2 note below, three percent more African Americans than whites 3 answered this question correctly. This was a difficult math 4 question. It is -- and if you note the question position, 5 the source is noted there, it was question number 25. 6 And the disposition of this question is that this 7 was the -- this was that one black preference question that 8 existed on all four tests, so if you're looking for black 9 preference questions on the test, this was it. 10 Q So this was the one out of 580? 11 A One out of 580, yes. 12 Q And again, for this question and the last one, is 13 there anything in it that would cause you to think that 14 there might be something quirky in the data or there might 15 be a big gap just looking at the question on its face? 16 A On its face, my answer is the same as before. It's 17 capturing something about race, but I couldn't be more 18 specific than that. 19 The answering rate, by the way, I wanted to -- the 20 answering, this is a difficult question. Whites answered 21 this question -- 14 percent of whites answered this question 22 correctly, 16 percent of blacks, and the difference being 23 three percent, it was because of rounding. 24 Q Is that their rounding or yours? 25 A That's their rounding. 116 1 Q ETS? 2 A Yes, this three percent was their data in a table, as 3 was 16 percent and 14 percent. 4 Q Let's look at your final question here. This is 5 Exhibit 209? 6 A Yes. This is a math question, a geometry question, 7 very common on the SAT. Again, this is a question from 8 those four released tests. The correct answer, as I note, 9 is D. The results here are again from the table, 32 percent 10 more whites than African Americans answered this question 11 correctly. It's a white preference question, as were all 12 the questions on the test other than the last one, except 13 for the neutral questions. 14 And at the bottom I note, this happens to be the 15 math white preference question with the largest percentage 16 difference of all the white preference math questions on the 17 four tests selected to appear on the SAT. 18 Q Do you think that the questions -- that the individual 19 questions you have talked about so far this afternoon, that 20 they have any capacity to tell us anything about the overall 21 verbal or math skills of the test takers by race or in 22 general? 23 MR. KOLBO: Object to foundation. 24 THE COURT: Sustained. 25 BY MS. MASSIE: 117 1 Q Do you think that the gaps that you have indicated 2 exist in terms of correct answering rates, gaps by race in 3 terms of correct answering rates to these questions, whether 4 they are the questions on which the black test takers 5 outscored the white test takers or vice versa, indicate 6 something about the overall verbal or mathematical abilities 7 and competencies of the test takers? 8 MR. KOLBO: Same objection. 9 THE COURT: Sustained. 10 BY MS. MASSIE: 11 Q Mr. Rosner, what do these numbers mean? 12 A Which numbers are you referring to? 13 Q The gaps. 14 MR. KOLBO: Well, Your Honor, I don't like to pop up 15 and down, but I think it's the same question and I have the 16 same objection. 17 THE COURT: Same ruling. 18 BY MS. MASSIE: 19 Q Based on your experience working with ranges of 20 students in terms of ranges of test scores, students of 21 different races over a period of years, having some sense of 22 how different people answer different questions and what we 23 can deduce from answers to different questions and whether 24 you get a question right or wrong, do you believe that these 25 gaps mean, for example, in the first several questions, that 118 1 black test takers as a whole have a nine percent better 2 vocabulary than white test takers? 3 MR. KOLBO: Objection on foundation. 4 THE COURT: You know, there is no foundation that he 5 has anything other than his own personal opinion. He is 6 here as an expert. If you want to ask him expert questions 7 within his realm, you can ask him. He is not -- 8 MS. MASSIE: He has worked with hundreds of 9 questions with test bias, different impacts and outcomes, he 10 has coached teachers, he has studied the test. I don't 11 understand how he could be more of an expert about the 12 relationship between -- 13 THE COURT: I do. I made my ruling. 14 MS. MASSIE: Okay. Well, I need to make my record. 15 We need to be able to develop this kind of question in order 16 to make our case. 17 THE COURT: Your objection is noted. 18 BY MS. MASSIE: 19 Q Is the rejection of questions on which black and other 20 minority test takers outperform white test takers the source 21 of the test score gap? 22 A Selection of questions, yes. The selection of 23 questions obviously from this data, which is consistent -- 24 this data confirms my experience over thirteen years that 25 the tests are comprised of white preference questions 119 1 virtually entirely. Actually, three of these four tests 2 were entirely white preference questions with a few neutrals 3 and then one test had one black preference question and 4 that's -- this confirms the experience that I have seen over 5 the years working with students on these questions and 6 seeing their scores at the end of the process. 7 Q And is it that all the questions on which white 8 students are outperformed by other students, is it that 9 those questions are weeded out, is that the only source of 10 the gap? 11 A No, the question selection process is not the only 12 source of the test score gap, but in my opinion, it's a 13 central key, critically important source of the test score 14 gap. 15 Q And in terms of the questions that are selected for 16 inclusion on the test, do they mostly fall in the same range 17 of disparate impact on minority students? 18 A No, and the next exhibit outlines that. 19 Q Which is to say, Exhibit 210; is that right? 20 A Yes, Exhibit 210. 21 Q Okay. Tell us about this exhibit, Mr. Rosner. 22 A Actually, we just said range, I was thinking of a 23 narrow band, and in fact, they do fall in a range. It's a 24 wide range. 25 So to clarify my previous answer, I specifically 120 1 looked at the ranges of differences in percentage answering 2 rates and they were large. This simply lists the ranges of 3 answering of differences in percentage answering rates when 4 I did the black/white comparison and when I did the 5 Hispanic/white comparison, and the first paragraph notes 6 that for the 574 white preference questions on the four 7 tests that I have been talking about, there was a range on 8 the white preference questions from one percent to 32 9 percent, and I note that 108 of the 580 questions had a 20 10 percent or more gap favoring whites. 11 By the same token, I took a look at the white/ 12 Hispanic comparison and that was also a large gap ranging 13 from one percent to 28 percent. Again, these are all 14 questions that are selected to appear on the test where 15 those who were selecting the questions know this data ahead 16 of time. That's what the pretesting process is all about. 17 Q And what would happen if you picked different 18 questions? 19 A You would get different results, and I actually did 20 that. 21 Q Tell us about that. 22 A I compiled a test using these 580 questions as my 23 test, as my question pool. 24 THE COURT: The 580 questions, the '88, '89 25 questions? 121 1 THE WITNESS: That's right. For 580 questions from 2 '88, '89, for which I had data for each question. 3 THE COURT: When did you do this? 4 THE WITNESS: I did this approximately two weeks 5 ago. I obtained the questions approximately a month ago, 6 having had the data a little earlier than that, I believe. 7 And so I compiled the test form and was able to 8 alter the results considerably, picking from their 9 questions, picking from selected SAT questions. 10 We have a demonstrative exhibit that outlines what I 11 found when I did that. 12 BY MS. MASSIE: 13 Q Is that the one that does not have a number that was 14 handed out for everybody's information? 15 A That does not have a number in the lower right-hand 16 corner. 17 Q Tell us about -- tell us about the test that you 18 constructed. 19 A Well, I called this, at the top, the Reduced White 20 Preference SAT Form, that's a name that I gave it. Again, I 21 state here it's created from the '88, '89 SAT question pool, 22 which was 580 questions, and then I compared how my new form 23 behaved and the kind of results that it generated with how 24 the prior forms behaved; in other words, the results that 25 they generated. 122 1 Q And what did you -- did you impose any constraints 2 on yourself? 3 A Yes. When I constructed the new SAT form I chose 4 145 questions, because that's what was on the SAT at that 5 time. It's since been reduced by a couple of questions. 6 And when I chose those 145 questions, I paid very 7 close attention to question category, question difficulty, 8 question type, so that I created a test that I was 9 comfortable with that looked -- that looked like any 10 other SAT, that looked in fact like the other four SATs. 11 THE COURT: So you have never had any training in 12 creating a test, putting them together, other than your 13 experience in teaching the SAT's and LSAT's? 14 THE WITNESS: We put tests together when we teach 15 the SATs and the LSATs, but no, not with this kind of 16 orientation. 17 THE COURT: And you realize there are people that 18 are trained in that area? 19 THE WITNESS: Yes, psychometricians are trained to 20 pick white preference questions here. 21 THE COURT: Go on. 22 THE WITNESS: And what I found is, if you look at 23 the -- it's a little clearer to explain if you look at the 24 bottom first here. You notice on the left-hand margin, test 25 form zero, test form one, test form two, and test form 123 1 seven. Those were the four different test forms with the 2 date that they were administered the test form. 3 And then what I did was I compiled for each of 4 the tests the average white correct answering percentage 5 for 145 questions on that test, and that's the first column, 6 and then the average black answering correct percentage for 7 the 145 questions on the test, that's the second column, and 8 then the difference, which is the third column. 9 So for example, test form zero, 48 percent was the 10 average white answering correct percentage for all the 11 questions; 36 percent, the average black answering correct 12 question, and 12 percent the average difference. 13 And then if you look down, you'll see I did the same 14 thing for the three other tests. Then I took an average 15 of those four tests, and that is at the very bottom. 16 The 53 percent was the average white answering 17 correct percentage, 40 percent average black answering 18 correct percentage, and the difference I carried out to one 19 more decimal place, which was 13.2 percent or 13 percent, 20 if that would -- if one deems 13 percent more appropriate. 21 Now, if you look at the top, here's what I found 22 when I compiled my new test form, and I knew that it would 23 be reduced, but I didn't know how much until I compiled it. 24 I had the white answered correct percentage was 25 significantly lower at 47 percent, the black answering 124 1 correct percentage remained about the same at 40 percent, 2 but the difference was significantly lower, and 13.2 percent 3 down to 7.8 percent represents a 40 percent reduction in 4 the black/white gap on the four tests, comparing the tests 5 I compiled and the average of the four tests that were for 6 which I had data. 7 Q And you again used questions in the same difficulty 8 range? 9 A I said before, I paid very close -- I didn't pick -- 10 I could have picked the 145 questions with the lowest gaps 11 and that would have produced an even lower percent, but 12 it was -- it was fair to construct what would have the 13 structure and appearance and difficulty level and test 14 question variety that a regular test form did, so that's 15 the process that I used. 16 THE COURT: You ran this by somebody like Professor 17 Shapiro to make sure you got this right and that's how they 18 do it in the industry? 19 THE WITNESS: I did not run it by Professor Shapiro. 20 I used my own experience of looking at dozens and dozens of 21 these tests and knowing how they are structured to match up 22 question type and percentage. 23 BY MS. MASSIE: 24 Q Are you trying to do it like they do it in the 25 industry? 125 1 A I was trying to do -- well, the way they do it in 2 the industry is they chose all white preference questions 3 with very significant ranges. I specifically was trying 4 to reduce those ranges. In any set of choices that I had, 5 I was trying to chose the question with the smallest 6 difference, black/white difference, consistent with general 7 difficulty level, question type, question style. 8 Q And what was your purpose in doing this? 9 A The purpose was to highlight and emphasize what a 10 dramatic difference can be made even if you're choosing all 11 white preference questions or virtually all white preference 12 questions, a dramatic difference can be made by paying very 13 close attention to the question gaps and in fact having as a 14 priority trying to minimize those gaps, even though you're 15 working with selected SAT questions. 16 Q So it was a demonstrative exercise? 17 A It was a demonstrative exercise. 18 Q What role do you detect that randomness plays in the 19 numbers that you have told us about today? 20 MR. KOLBO: Objection to foundation, Your Honor. 21 BY MS. MASSIE: 22 Q Mr. Rosner, do you have an opinion on whether these 23 processes are systematic or whether they are in fact 24 affected with a large level of randomness? 25 MR. KOLBO: Same objection. 126 1 THE COURT: Sustained. 2 BY MS. MASSIE: 3 Q Mr. Rosner, in your work with the tests and with 4 students over the years, your work studying the tests and 5 its effect on students, your work training teachers, your 6 work reading material on the tests, have you come to a 7 conclusion about whether the outcomes on the test are the 8 product of design or chance? 9 A Clearly the product of design. That was clear to 10 me before I had this data and I believe this data simply 11 confirms what I have experienced all along in my test 12 preparation work. 13 MR. KOLBO: Well, Your Honor, I'll just move to 14 strike. That was a yes or no question and I'll object. 15 THE COURT: I'll overrule. I'll give it as much 16 credibility as it deserves. 17 MS. MASSIE: There is no need to insult the witness. 18 THE COURT: I'm not insulting anybody. Credibility 19 is up to the Court and that's why I overruled his objection. 20 MS. MASSIE: You have been insulting the witness 21 the entire time he has been on the stand this afternoon. 22 BY MS. MASSIE: 23 Q Mr. Rosner, could you please summarize for us your 24 view of the relationship between question selection and 25 the overall test score gap by race on the LSAT? 127 1 A Sure. 2 MR. KOLBO: Object to foundation, Your Honor. 3 THE COURT: I'm sorry, I didn't hear the question. 4 BY MS. MASSIE: 5 Q Can you please summarize for us your opinion on the 6 relationship between question selection procedures and the 7 score gap, the test score gap by race on the LSAT? 8 A The question selection procedures on the LSAT are 9 identical to this question selection procedure that I have 10 been talking about for the past hour, and that is, white 11 preference questions are chosen for the LSAT in a great, 12 great predominance. 13 MS. MASSIE: Thank you, Mr. Rosner. 14 THE WITNESS: By the way, Your Honor, I don't feel 15 insulted by you. 16 THE COURT: Thank you. I didn't mean to insult you. 17 MR. PAYTON: Can I have just five minutes? 18 THE COURT: Sure, you can have five minutes. Do you 19 want to take a real break for five minutes? 20 MR. PAYTON: Yes. 21 THE COURT: We will take five minutes. Stand in 22 recess. 23 COURT CLERK: All rise. 24 (Recess taken at 2:42 p.m.) 25 (Back on the record at 2:53 p.m.) 128 1 COURT CLERK: All rise. 2 THE COURT: You may be seated. Okay. 3 CROSS EXAMINATION 4 BY MR. PAYTON: 5 Q Mr. Rosner, I actually just want to ask you some 6 questions about what you testified to when we were last 7 here, which is what your business is. 8 Your business, the Princeton Review, is about 9 improving test scores, SAT scores, LSAT scores for people 10 who are going to take the test, is that it? 11 A My business is the Princeton Review Foundation and the 12 purpose of both the Foundation and the Princeton Review is 13 as you stated, yes. 14 Q Let me just focus on what the Princeton Review then 15 does, because its purpose is just to increase test scores, 16 SAT, LSAT. We will come to the Foundation later. 17 A Okay. 18 Q Okay. And that's also the business of Kaplan; is that 19 right? 20 A That is one of their businesses, yes. 21 Q What's the full name of Kaplan? 22 A Stanley Kaplan Educational Centers, I believe is the 23 full name. 24 Q Okay. And I want to focus on just that part of the 25 business of Kaplan and of the Princeton Review that focuses 129 1 on increasing LSAT scores; okay? 2 A Okay. 3 Q Mr. Washington, I think two weeks ago, referred to 4 that as boosting LSAT scores? 5 A He may have. 6 Q Okay. You seem to react badly to that. Is that -- 7 A Well, improving, boosting. One can generate a number 8 of synonyms that would refer to it. 9 Q All right. So is it fair to say that Princeton Review 10 and Kaplan are the two major companies that improve or boost 11 LSAT scores? 12 A Yes. 13 Q And Kaplan is the largest company, Princeton's 14 smaller, but still a large company in this business? 15 A Yes, they are the two -- they are the only two 16 entities of substantial size who do this on a national 17 basis. 18 Q Okay. And am I right that the students that are 19 likely to go to either the Princeton Review or to Kaplan 20 are those students who are actually looking to apply to the 21 more selective law schools, Michigans, Harvards, Yales, not 22 all of them, but are they disproportionately represented 23 among those applicants? 24 MR. KOLBO: I'll object to the foundation, Your 25 Honor. 130 1 THE COURT: If he knows. 2 BY MR. PAYTON: 3 Q If you know. If you know where they apply. Is that 4 speculative? 5 A There is a range and I'm not sure whether -- there are 6 a considerable number and a considerable percentage applying 7 to highly selective institutions. I'm not sure that it's 8 disproportionate. 9 Q Okay. Now, I think you told us that some 70,000 10 students take the LSAT in any given year, that was a rough 11 number? 12 A Right. 13 Q And of the 70,000, some 45,000 are white, is that 14 correct? 15 A That sounds correct to me, yes. 16 Q And I think you said that between Kaplan and the 17 Princeton Review together, you serve about 25,000 of those 18 45,000 white students? 19 A Actually, what I -- the answer is yes, almost, because 20 there are some students who we serve who don't apply. 21 Q Yes, you said some who back out or for whatever 22 reason. 23 A Yes. 24 Q Very small number? 25 A Yes, a relatively small percentage of students who we 131 1 serve don't apply. 2 Q Okay. And I believe you also said that neither the 3 Princeton Review nor Kaplan has more than a token number 4 of black or Hispanic students in their courses? 5 A That's generally true, yes. 6 Q You said you were sad to say that the black or Latino 7 representation was just token numbers? 8 A Yes. 9 Q If any. 10 A That's correct. 11 Q So let me just add that up. These are in essence 12 booster courses for all-white test takers of the LSAT? 13 A Primarily and predominantly, yes. 14 Q And you testified that some studies had been done for 15 both Kaplan and Princeton that indicated that in fact the 16 courses do boost the LSAT scores. I think you said that 17 there was a study of Kaplan and that it boosted LSAT scores 18 by seven points. Do you remember that? 19 A That's correct, and that's appended to my expert 20 report in this matter. 21 Q And I believe you said that there was a similar study 22 done for Princeton, and that it indicated that Princeton 23 boosted LSAT scores by six to seven points? 24 A Yes. I think actually the figure was 6.7 points, but 25 yes. 132 1 Q That's 6.7, so both about seven? 2 A Both about seven. 3 Q And I think you then told us that a seven point boost 4 was quite significant, and I think you said that if it 5 were in the middle range, around 150, that would be a 6 27 percentile difference; is that right? 7 A That's correct, and the 27 percent jump related to a 8 score of 146, I believe, but yes, you get the higher jumps 9 in the middle of the range. 10 Q And I also think you said that this boost, seven point 11 boost in LSAT scores, didn't reflect in any way an increased 12 aptitude for law or law school? 13 A Not at all, as far as I could tell, based on my 14 experience. 15 Q So it's a synthetic boost? 16 A It's a synthetic boost compared to the real world. 17 It's a boost in terms of better understanding and better 18 responding to the test. 19 Q Now, do I then understand this correctly, that given 20 the dramatic boost, say 27 percentile dramatic boost in LSAT 21 scores, on average, for your all white, 25,000 white student 22 clientele in a given year, that you have literally no black 23 or Hispanic participants in your courses? 24 A Token participants. 25 Q You have token participants. Am I right, would you 133 1 agree that the Princeton Review and Kaplan are responsible 2 for much of the test score gap in LSAT scores between 3 African Americans and Hispanics and white students? 4 MR. KOLBO: Object to foundation, Your Honor. 5 THE COURT: Overruled. If he knows. 6 A Kaplan and the Princeton Review are responsible for a 7 portion of the test score gap, which is significant. There 8 was one part of your -- there was an assumption, though, 9 in your question that I need to address. It was a long 10 question. 11 You pointed out the 27 percent boost. You picked 12 the maximum boost -- 13 Q I understand that. 14 A -- along the range, and in the context of the question 15 there might have been an assumption that that was a typical 16 boost. Seven points is 27 percent maximum. It's less than 17 that elsewhere. 18 Q As you move up or down? 19 A Yes. 20 Q But if you take it in the middle range, which you 21 directed us to the middle range so you could see what the 22 maximum boost would be -- 23 A Right. 24 Q -- if you're in the middle range, it's a 27 percentile 25 boost? 134 1 A No question about that. I think as you made a general 2 set of assumptions in asking your question, though, you 3 rejected one assumption which maximized an effect rather 4 than took a typical effect. 5 MR. PAYTON: I take your point. Thank you very 6 much. 7 THE COURT: Plaintiff, any questions? 8 MR. KOLBO: A few questions, Your Honor. 9 CROSS EXAMINATION 10 BY MR. KOLBO: 11 Q Good afternoon, Mr. Rosner. My name is Kirk Kolbo and 12 I'm one of the lawyers representing the Plaintiff in this 13 case and I just had a few questions for you. 14 First of all, just a clarification. My 15 understanding was, and correct me if I'm wrong, that your 16 estimate is that Kaplan and Princeton Review serve combined 17 about 25,000 students total; is that correct? 18 A Total annually, yes. 19 THE COURT: That includes, I'm sorry, not only LSAT, 20 though. 21 THE WITNESS: No, that's just LSAT, Your Honor, 22 and the breakdowns were approximately 10,000 for Princeton 23 Review, approximately 15,000 for Kaplan, give or take, those 24 were my estimates. 25 BY MR. KOLBO: 135 1 Q And that's 25,000 students including all minority 2 students that attend these courses; correct? 3 A Yes. 4 Q Not just the white students? 5 A That's true. It includes the token number of minority 6 students. 7 Q But you also testified that there are about 45,000 8 white students all together nationally that take the LSAT 9 score; is that correct? 10 A That's the data I have seen from Law Services, yes. 11 Q So there are about 20,000 white students each year 12 that take the LSAT test without taking either the Princeton 13 Review or the Kaplan course; is that correct? 14 A That's correct. 15 Q And then there is another -- over and above that there 16 is another 15,000 of presumably different races that don't 17 take either of those two courses; is that correct? 18 A That's correct. Actually, it's -- if I can amend the 19 prior answer, it's slightly more than 20,000, because there 20 are some who take our course who don't apply, but you're in 21 the ballpark. 22 Q If my math is right, and it's been wrong before up 23 here in front of everybody, there are about twice as many 24 students who don't take either one of these preparation 25 courses as students that could take them; is that correct? 136 1 A Define -- 2 Q Total of 70,000, of those about 25,000 students take 3 one or two of these preparation courses? 4 A And would you repeat your question? 5 Q There is roughly, by my count, about 45,000 students 6 that don't take either one of these test preparation 7 courses; correct? 8 A That's right, however, in your hypothetical you're 9 missing the fact that there are regional courses of 10 considerable intensity that I would consider. We were 11 talking about just national courses. There are regional 12 courses of considerable intensity that I would deem to be a 13 fully equivalent or a near equivalent to a Princeton Review 14 or a Kaplan course, so actually, there are more than 25,000 15 students taking intensive courses, with an additional couple 16 of thousand taking a regional course or a local course 17 rather than a national course. 18 Q Okay. Well, still it's then probably no more than 19 30,000 students all together that are taking some kind of 20 a preparation course? 21 A If you're talking about intensive preparation courses, 22 that would be a reasonable estimate, perhaps slightly on the 23 high side. 24 Q And another 45,000 or so who aren't taking any of 25 these type of courses? 137 1 A Slightly more than 45,000, again, because not 2 everybody is applying. 3 Q And I'll let everybody else do the math on that. 4 Is it fair to say it's possible to do well on the 5 LSAT test without taking any of these preparation courses? 6 A Yes. 7 Q There are among that 45,000 plus number of students, 8 there are many of them that do well on the test; correct? 9 A There would be a considerable number who do well 10 on the test without taking an intensive test preparation 11 course, that's correct. 12 Q And one can actually study for the test, the LSAT 13 test, without taking one of these organized preparation 14 training courses; correct? 15 A Certainly. 16 Q You could spend, what is it, 40 or 50 bucks to buy one 17 of the books on the LSAT and study that way? 18 A Yes, you can buy released prior tests, you can buy 19 test preparation books in the bookstore, and one can study 20 on one's own for the test, that's correct. 21 THE COURT: They even have a CD-Rom, don't they? 22 THE WITNESS: CDR's, yes, Your Honor, and there are 23 on-line courses now, so there has been a change in the last 24 couple of years in terms of software and even a greater 25 change in the last year or so with on-line resources. 138 1 BY MR. KOLBO: 2 Q And am I correct that -- correct me if I'm wrong, but 3 is it your understanding that the testing organizations, 4 ETS and ACT, that's the other one? 5 A ACT produces their own high school exam and from time 6 to time has written questions for the LSAT. 7 Q My recollection from your prior testimony at trial 8 here is you're not sure who it is, which of these testing 9 organizations that's writing the LSAT these days; correct? 10 A I lost track of that years ago, yes. 11 Q But is it -- am I correct that it's the position of 12 these testing organizations that test preparation, taking 13 courses like Kaplan's and Princeton Review, does not 14 significantly improve test, actual test performance, is 15 that a fair summary of their position? 16 A That's a fair summary of the position of every test 17 manufacturer. The SAT, ACT, Law Services, the AAMC produces 18 the MCAT, yes. 19 Q And they have taken that position, as you understand 20 it, based on studies that they have had done; correct? 21 A They do their own studies, yes. Are those studies the 22 basis of their position, I think marketing is the basis of 23 their position, but they do have studies to support their 24 position, also. 25 Q Well, do you have any reason to believe that they are 139 1 making this claim in contradiction to the studies that they 2 themselves have produced or sponsored? 3 A Yes, I do. 4 Q And what's that based on? 5 A That's based on reading their studies, which I think 6 are almost humorous in the flaws and fallacies that exist. 7 Q Well, studies themselves at least on their face, they 8 support the ETS and the ACT position? 9 A ETS paid researchers to reach conclusions that support 10 ETS's marketing position, and that's the same for Law 11 Services and the same for every test company, yes. That 12 is -- that is as consistent as the test score gap. That 13 always happens. 14 Q And you disagree with the conclusions that they have 15 reached? 16 A I disagree with the methodology. 17 Q You're not a -- 18 A And I'm sorry, and I disagree, yes, and I disagree 19 with the conclusions. 20 Q Just to be clear, and I think this is clear, but 21 you're not formally trained in statistics? 22 A No, I'm not. 23 Q Or psychometrics? 24 A No, I'm not. 25 Q Or psychology? 140 1 A No, I'm not. 2 Q Or education? 3 A I was close to an educational certification to teach, 4 but never got it, so no, I'm not. 5 Q And I think I said this, but if I missed it, 6 psychology, you have no formal -- 7 A You didn't miss psychology, and I said that I was not. 8 Q I apologize for repeating. 9 A That's okay. 10 Q And as I understand it from reading your report, it's 11 your position that these testing organizations are motivated 12 in their position that these preparation courses are not 13 significantly helpful because it's profitable for these 14 testing organizations to maintain the perception out there 15 that it doesn't help to prepare for these tests; is that a 16 fair statement? 17 A Not entirely. 18 Q I didn't state it very well, so why don't you tell me 19 what you believe motivates these testing organizations to 20 make these marketing claims that you believe are based on 21 erroneous data. 22 A It's a very simple concept, and the concept is, if 23 you're putting a number out there, a test score which the 24 University of Michigan or any other school is going to rely 25 upon, there has to be a certain sense that that number is 141 1 strong and solid. If there is a sense created that that 2 number can be moved around depending on whether somebody 3 can afford $1,000 for a test preparation course, that number 4 has very little meaning. 5 So there is a very strong marketing and sales 6 imperative, but more personally, an image imperative for 7 the test companies to take a position, and poorly designed 8 studies that conclude that test preparation doesn't work, 9 because the inverse of that that they do work would cut to 10 the heart of reliance on this number, which in fact is not 11 consistent or firm and can be moved pretty dramatically 12 depending on interventions. 13 Q Are these testing organizations, are they for-profit 14 enterprises, as far as you know? 15 A ETS, the largest of the organizations we're talking 16 about, is a non-profit organization. 17 Q Well, part of it, as I understand, though, you're 18 suggesting is that the testing companies in order to 19 justify -- one of the reasons, one of the ways to justify 20 the continued existence and use of these tests is to 21 disparage the value of test preparation courses, is that 22 a fair statement? 23 A That's a fair statement and reflects their behavior 24 over many years. 25 Q Okay. Now, Princeton Review, not the Foundation, but 142 1 Princeton Review, is a for-profit company; is that correct? 2 A Princeton Review is a for-profit company, yes. 3 Q And it makes a lot of money by giving test preparation 4 courses around the country? 5 A I can state emphatically that it doesn't make enough 6 money to satisfy the Princeton Review folks, but it does 7 generate a profit most years in running test preparation 8 courses. 9 Q Okay. And would you agree that the Princeton Review 10 has an interest in persuading and convincing and marketing 11 to people that test preparation courses in fact are valuable 12 in increasing the actual test scores of test takers? 13 A Yes, as does Kaplan. 14 Q And Princeton Review has produced studies showing that 15 use of review courses improves their scores and the testing 16 companies have produced studies saying that in fact they 17 don't improve significantly in performance; correct? 18 A That's correct, but Mr. Kolbo, you have been saying 19 the Princeton Review, and I'm assuming in the last couple 20 of questions you're talking about our for-profit side, you 21 did say that, and the answers to the questions would be 22 different if you talked about the Princeton Review 23 Foundation. I just want to make that point. 24 Q Oh, sure. But who is paying for these studies that 25 support the Princeton Review Foundation on the value of 143 1 these tests -- I'm sorry -- value of test preparation, who 2 is paying for that, the Princeton Review Foundation? 3 A No, you said in your initial question, before you 4 corrected, it was the Princeton Review Foundation. It's 5 the Princeton Review is paying for the study that supports 6 the Princeton Review score gains. The Foundation plays no 7 role in that other -- that I know about it. 8 Q Just to be clear, the Princeton Review certainly has a 9 financial motive for persuading the public that there is a 10 value served by taking these prep, test preparation courses; 11 correct? 12 A Correct. 13 Q Do you have any idea what percentage of the students 14 who apply to the University of Michigan take a test 15 preparation course of some kind, whether it's Kaplan or 16 Princeton Review or one of these regional courses? 17 A No. 18 Q And you have no idea what percentage, I assume, from 19 your answer, in terms of breaking down the percentages of 20 African Americans, Asian Americans, white students, Hispanic 21 students, Native Americans, you have no idea what percentage 22 of those students in those groups applying for Michigan law 23 school take the preparation course; correct? 24 A I don't have specific information as to Michigan, but 25 when you -- as soon as you talk about students overall, as 144 1 soon as you start talking about black students and Latino 2 students, I know the rate at which those students take test 3 preparation courses, I testified several times they take 4 them at token rates, so one can make an inference as to what 5 would happen with those groups applying to Michigan, but I 6 don't have specific information. 7 Q And I just want to -- I think I understand this, but I 8 want to be clear. You have testified several times that 9 only token members of certain minorities take -- 10 A Yes. 11 Q -- Princeton Review courses. I assume these courses 12 are open to students of all races and ethnicities? 13 A Students are -- the courses are open to students of 14 all races. 15 If I can amend my last answer, no one, I believe, 16 would know percentages of students in various racial or 17 ethnic groups who have taken courses who are applying to 18 the University of Michigan. I stated in my direct testimony 19 two weeks ago that there are indeed motivations for students 20 not to reveal that information. 21 Q Well, someone can certainly ask, could they not, do a 22 survey? 23 A Oh, that's how -- you have just hit upon something we 24 talked about before. That's how the test companies do their 25 studies. They ask students whether you have taken a course, 145 1 and they assume the response that they are getting is 2 accurate, which is laughable. 3 Q You're not suggesting that it's impossible somehow 4 to commission a survey to find out how many individuals 5 applying to the University of Michigan typically each year 6 have taken a test preparation course? 7 A I assert that that's impossible. 8 Q You assert that it's impossible? 9 A Yes. 10 Q And that's not a question that could, for example, be 11 asked on an application? 12 A Oh, it can be asked. It's just, there is no reason to 13 assume that the answer would be honest or accurate, because 14 the student, as I said, has a motivation not to reveal that 15 information. 16 Q And so because of that -- 17 THE COURT: I don't understand. What would be the 18 motivation if they are doing blind surveys? We do blind 19 surveys all the time, send out a blind survey all the time. 20 And when I say blind, you don't have to put your name, no 21 numbers, no nothing. They send them out on Federal Judges 22 at least every, I forget, every four years. 23 THE WITNESS: Understood, Your Honor. The basis of 24 that answer is having discussions with perhaps 50 or 60 or 25 70 students over the last ten years who have said to me, 146 1 there is this question, have I taken a course, what do I 2 say. And I say, you know, you answer the question. I 3 can't tell you to answer it any way but honestly. And the 4 students say, well, they have no business of knowing, I'm 5 not going to tell them. 6 THE COURT: But that's the same thing if you get 7 any kind of survey, some people are honest and some people 8 aren't. 9 THE WITNESS: I think -- 10 THE COURT: You think law students are less honest? 11 THE WITNESS: Not to argue with you, Your Honor, 12 but I -- 13 THE COURT: I'm not arguing with you. As I say, we 14 all get surveys. 15 THE WITNESS: I think I'm pointing out a difference 16 in this kind of a survey. It would be more -- it would 17 even -- there would even be a greater difference if the 18 University of Michigan asked this. 19 THE COURT: What's so bad about saying, yes, I took 20 a bar review course or I took an LSAT course? 21 I don't want to get into it with you. That's okay. 22 That's okay. 23 THE WITNESS: The short answer to that, Your Honor, 24 is that there are lots of disparagement of these courses 25 coming from the test companies and students can get the 147 1 feeling very easily that I'm not maybe supposed to be doing 2 this. 3 BY MR. KOLBO: 4 Q Do you assume that because a student believes that an 5 answer to a question will be contrary to their interests 6 that they will give a false answer on an application? 7 A As a general principle, no, but in the specific 8 instance, I told you the kinds of experiences that I have 9 had where students indicated to me that they weren't going 10 to give this information, because they had a funny feeling 11 about it and they felt the entity asking the information 12 didn't have any right to know it. 13 Q Another question on preparation, and I'll move on 14 very briefly to something else. 15 We talked about how some students may take one of 16 these formal courses, like Princeton or Kaplan or one of 17 these regional courses and how others students might just 18 buy a book and study that way. Have you done any -- have 19 you conducted any studies to determine or are you aware of 20 any studies that assess the differences in performance 21 between those types of preparation? 22 A I think law -- if we talk about the LSAT, Law Services 23 does a study on the effect of preparation, like all the 24 other companies, and they do this survey data, and I 25 believe -- I believe they do have separate questions on, 148 1 for example, did you prepare using books, did you take a 2 course, so in the context of the methodology that they use, 3 they may have some conclusions as to that distinction, so. 4 Q You don't know what those are? 5 A I don't know what those are. 6 Q You have testified last time and this afternoon about 7 often referring to a number of questions, SAT questions; 8 right? 9 A Yes. Those were all SAT questions. I was unable to 10 obtain LSAT questions. 11 Q And these are SAT questions that are about ten or 12 eleven years old or so? 13 A Yes, twelve or -- in some cases, thirteen years old. 14 Q Now you say that you have testified that you have not 15 had access to LSAT questions; correct? 16 A Let me be clear. LSAT questions, the content of the 17 questions, is available. It always has been. I'm sorry, 18 since 1980 it has been. What is unavailable is the percent 19 answering rate breakdowns disaggregated by ethnicity, and 20 when I talk generally about unavailability, that's what that 21 data is that I'm referring to that is unavailable from Law 22 Services completely, and it's unavailable from ETS except 23 for these four tests. 24 And I checked with Dr. Shapiro. The data that he 25 was talking about, Your Honor, does not include this racial 149 1 breakdown data. So it's entirely unavailable other than 2 what we have talked about in the courtroom here with regard 3 to the SAT and the LSAT. 4 Q But for example, rejected questions, those are a 5 certain form of question that one can take a look at, 6 correct, you have done that with SAT questions? 7 A I have done that with the dozen or so questions that 8 have been made available to me through ETS, through the 9 newspaper. 10 Q And they simply have not been available to you from 11 the LSAT test; correct? 12 A Tried to get them. Was told that they are not 13 publicly available. 14 Q Presumably. 15 A Again, referring "they" to the statistics, not the 16 questions. 17 Q "They" being the testing organization; correct? 18 A Correct. 19 Q And you're not sure which one that is, even, ETS of or 20 its ACT? 21 A No, I'm sure what that is. That's Law Services. 22 Q Law Services, okay. 23 A And they administer the test. The other company 24 subcontracts test writing, test question writing. 25 Q Okay. So Law Services would have -- someone in 150 1 Law Services has this detailed breakdown on performance 2 among racial groups on these tests; correct? 3 A Yes, when they do their DIF analysis, they have them. 4 Q And Law Services has access to these rejected 5 questions that are rejected for various disparate impact 6 reasons; correct? 7 A If the test companies supply them with sets of 8 rejected questions, they have them, and I assume that 9 they have them, or certainly easy access to them. 10 Q And you were here this morning when Professor Shapiro 11 testified? 12 A Yes. 13 Q And explained that he has been able to see, get access 14 to some of these, at least SAT questions in the course of 15 litigation, correct, Illinois litigation, for example? 16 A Yes. Yes, I recall his testimony on that, but Mr. -- 17 Professor Shapiro informs me, and again, I checked on this 18 at the break, that other than these four tests, no data 19 like this is available for any SAT question. 20 Q Well, presumably, if a lawsuit was pending and one 21 of the parties served a subpoena on Law Services, there 22 would be somebody there who could answer questions based 23 on this data that's not available to you; correct? 24 A There would be somebody there who would refuse to 25 answer those questions, yes, but who would be capable of 151 1 answering the questions. 2 Q And who would be subject to a subpoena? 3 A Yes. 4 Q You're a lawyer; correct? 5 A Pardon? 6 Q You're a lawyer? 7 A Yes, I am. 8 Q Law Services could be compelled by a Court to make 9 certain disclosures? 10 A They could theoretically be compelled by a Court, yes. 11 Q That just hasn't happened in this case as far as you 12 know? 13 A That hasn't happened in any case with regard to the 14 SAT of which I'm aware or in any case with regard to the 15 LSAT of which I am aware. 16 Q Just to be clear on this, there is -- to your 17 knowledge, there has been no subpoena served on Law Services 18 in connection with this case, correct, for this data? 19 A I could respond to that, but it would be hearsay. I 20 have no direct knowledge. 21 Q Well, I have no knowledge of a subpoena, Mr. Rosner. 22 I have just a couple more questions for you 23 regarding the exhibits that you used. Each one of them, 24 starting with page -- or Exhibit 202 through 208, 209, at 25 the top there is a designation, black or white preference 152 1 question. That's your designation, correct, to call it a 2 black or white preference question? 3 A I'm sorry, which one were you referring to? 4 Q Well, you can take any one of these, starting with 5 Exhibit 202. 6 A Okay. Sure. 7 Q You prepared these exhibits? 8 A I prepared these exhibits, and yes, I tried to be 9 clear in my direct testimony that the label, Black or White 10 Preference Question was my label, and I defined it, I hoped, 11 to the understanding of everyone. 12 Q Okay. These aren't the newspapers that you're relying 13 on that are using this term? 14 A No, again, I wanted to emphasize that this was my 15 terminology. I hope I made that clear. 16 Q And you're responsible for the way these exhibits were 17 prepared? 18 A Yes, I'm responsible for the way -- I'm responsible 19 for the content of these exhibits. 202 through 209 were 20 prepared by me on my computer, with the exception of the 21 figure on 209, which was my handiwork with a ruler and 22 a pen. 23 Q Okay. Could you turn to Exhibit 204, please? 24 A Sure. 25 Q And I'm just going to ask a couple questions more out 153 1 of curiosity than anything else, but I'm just about done, 2 so I don't think I would be wasting much time. 3 I was just reading ing this, taking a look at 4 this question while you were testifying, and the -- as I 5 understand it, the correct answer is supposed to be D; is 6 that right? 7 A Yes, that's right. 8 Q And just to take a look at these questions, there is a 9 value for X that one would need to put in here to determine 10 what the correct answer is; correct? 11 A That's right. 12 Q Isn't the correct answer for X, correct value for 13 X the number three? 14 A Oh, you fell into the trap. 15 Q How's that? 16 A Shall I explain? 17 Q Sure. 18 A That's the way -- that's one of the traps that -- one 19 of the common traps that is built into questions. Both 20 black preference questions and white preference questions 21 have traps. The trap here is that X could be three or 22 negative three and that's why D is the correct answer. 23 Q So I got that question wrong? 24 A You're opening yourself up for a comment which I 25 won't make. 154 1 MR. KOLBO: Your Honor, I have nothing else. 2 THE COURT: Any further questions? 3 MS. MASSIE: Just a couple questions. 4 REDIRECT EXAMINATION 5 BY MS. MASSIE: 6 Q Mr. Rosner, have you in your teaching and your 7 supervision of teachers known people's scores to improve 8 by give or take seven points on the LSAT over the course 9 of a Princeton Review class? 10 A Yes, that confirms with my experience. I'm 11 particularly concerned about the impacts of white preference 12 tests and white preference questions to black students and 13 so I have paid very, very close attention to that over the 14 last several years. Seven points is a typical improvement 15 for black students, although the average improvement would 16 be maybe -- for black students maybe five, five and-a-half 17 points, something in that range, based on my anecdotal 18 observations. I haven't done a formal study on that. 19 Q When you say anecdotal observations, you have taught 20 hundreds of students and you have been attentive to the 21 degree to which their scores improve; is that correct? 22 A Yes. It's a large body of information, though 23 anecdotal. 24 Q And it confirms the more formal studies that have 25 been commissioned, maybe I shouldn't be saying this in the 155 1 plural, at least the one study that was commissioned by the 2 Princeton Review? 3 A There is a general confirmation, yes, with possibly a 4 relatively small difference, in my experience. 5 Q And in your experience, Professor Shapiro has 6 testified about the process of test equating so that a 7 score today means more or less the same as a score several 8 or even more years ago. Have you in the course of your 9 teaching noticed that there are -- that results remain 10 relatively constant? 11 A Yes, and test score gap is consistent. It's 12 consistent, and one of the reasons it is is because these 13 are white preference tests comprised of white preference 14 questions and that remains. It's clear to me that that 15 remains consistent from 1988 to today on both the SAT 16 and the LSAT. 17 Q You clarified that the four SAT's you spoke about 18 earlier in your direct were the only four SAT's that have 19 been made publicly available with percentages, answering 20 correct percentages of people answering correct by race. 21 A Yes. 22 Q There aren't any questions like that available for 23 the LSAT; right? 24 A Base on my discussions with Law Services, they tell me 25 that there are none publicly available, that none of that 156 1 data with regard to the LS -- with regard to LSAT questions 2 is publicly available there. 3 Q And we did try to get that information through legal 4 process in this litigation, didn't we? 5 A We discussed -- you and I discussed serving a subpoena 6 on Law Services, and I remember we were -- we had a 7 discussion on the question, the kinds of questions that 8 would be in that subpoena, and I was told by you that the 9 subpoena was served, although I never saw a copy of it, or 10 that's why I said it was hearsay that that kind of thing 11 happened. 12 Q And I also told you that it got stonewalled; right? 13 A Yes, you told me the response, which was no 14 information. 15 MS. MASSIE: Thanks. 16 THE COURT: You may step down. Thank you. 17 Appreciate you coming back to our nice weather. 18 THE WITNESS: Try to arrange some sunshine next 19 time. Thank you. 20 THE COURT: Because your weather sometimes isn't 21 much better than yours. 22 THE WITNESS: Sometimes. 23 THE COURT: Next witness, please. 24 MS. MASSIE: Jodi Masley will be examining our next 25 witness. 157 1 THE COURT: Before the day is out, let's kind of get 2 an idea of how many more witnesses. We can do it now or 3 later, I don't care, before you leave. 4 I'll tell you why, I have a jury that I have to pick 5 before next Thursday. I don't have to try the case, but I 6 have to pick a jury before -- I think it's next Thursday or 7 Wednesday, we will talk about that. 8 MS. MASSIE: Okay. We can talk about it either now 9 or later. Do you have a preference? 10 THE COURT: Right now, I don't care, since you're 11 standing there. 12 MS. MASSIE: We have two witnesses tomorrow. I have 13 spoken with Mr. Kolbo and Payton. We have two, Crystal 14 James and Walter Allen. Crystal James is a second year 15 student at the UCLA School of Law and Walter Allen is the 16 team leader for the team that conducted our campus climate 17 study. 18 On Thursday we have Eric Foner, Professor of History 19 from Columbia University, designated as a witness by the 20 University. We have Eugene Garcia, who is the Dean of the 21 Graduate School of Education at UC Berkley, and David White, 22 who is an expert on testing, some testing issues. 23 And on Friday, we have Rick Lempert, who you heard 24 from as a fact witness, but he is also an expert studying 25 the -- comparing the career trajectories and career 158 1 success of -- 2 THE COURT: I know who he is. 3 MS. MASSIE: You know him. And Frank Woo, Faith 4 Smith, and we are also working out which -- I have discussed 5 with Counsel several students we may call in addition. 6 THE COURT: Okay. Mr. Payton, did you want to say 7 something? 8 MR. PAYTON: At some point we should talk about 9 rebuttal and the rest of it, but we don't have to do it now. 10 MS. MASSIE: Let's do that later. 11 THE COURT: But let's do it before we leave, because 12 I want to pick this jury sometime next week. I wouldn't 13 start the case until we finished with this one, but I have 14 to at least pick a jury. The farthest they are coming from 15 is Milan, so -- 16 I'm sorry. Stand for one more second and I will 17 swear you in. Don't look so scared. 18 -- --- -- 19 C O N C E P C I O N E S C O B A R, 20 having been called as a witness herein, and after having 21 been first duly sworn to tell the truth, the whole truth 22 and nothing but the truth was examined and testified. 23 -- --- -- 24 THE COURT: You may be seated. Thank you. 25 If you could give us your full name, and starting 159 1 with the spelling. 2 THE WITNESS: Concepcion Escobar, E-s-c-o-b-a-r. 3 THE COURT: E-s-c-o -- 4 THE WITNESS: E-s-c-o-b-a-r. 5 THE COURT: Thank you. 6 DIRECT EXAMINATION 7 BY MS. MASLEY: 8 Q Ms. Escobar, could you tell us your address? 9 A 551 South State Street, Ann Arbor, Michigan, 48109. 10 Q And are you a student? 11 A Yes. 12 Q Where are you a student today? 13 A I'm at the University of Michigan Law School. 14 Q I'm going to ask you a series of questions just about 15 your background to start out. Where were you born? 16 A In Chicago. 17 Q Can you tell us something about your parents? 18 A My mother is an immigrant from Mexico and my father 19 is originally from Arizona and he is an Apache or was 20 Apache. He is deceased. 21 Q Okay. Is there anything else you would -- more you 22 would like to tell us about your parents? 23 A Well, they both came to Chicago when they were 24 thirteen years old. They have less than a first grade 25 education. When they came to Chicago, they came alone. 160 1 My father was abandoned by his parents when he was 2 eight years old and pretty much grew up on the street, and, 3 you know, at thirteen years old went to Chicago and started 4 working in factories. 5 My parents met when my mother was 19 and my father 6 was 24 and they dated for six months and then married after 7 one year. 8 Q And what does your mother do? 9 A She is a factory worker. 10 Q How long has she been a factory worker? 11 A For 38 years. 12 Q What kind of factory does she work in? 13 A The name of the company is Williams Electronics. They 14 make pinball machines and video games, but the company is 15 going out of business, so she is going to be losing her job 16 I think in April. 17 Q Where did you do your undergrad work? 18 A Amherst College. 19 Q What was your cumulative GPA when you left Amherst? 20 A 3.43. 21 Q And what was your LSAT score when you applied to 22 law skull? 23 A 158. 24 Q I'm going to ask you a series of questions and your 25 testimony in this case is very important and it's going 161 1 to go to two areas, the alleged double standard that the 2 Plaintiff asserts that exists between white and minority, 3 under-represented minority applicants, and also the bias 4 that is offset by affirmative action, and your testimony 5 is going to be very important to helping the Judge 6 understand your distinct experience at Amherst. 7 First, how did you get to Amherst? 8 A I was a student at Whitney Young Magnet High School, 9 which is -- it's widely recognized as the best public 10 high school in Chicago, but it's a segregated school, it's 11 mostly black, and, you know, about maybe ten percent white 12 students, some Asians and some Latinos. 13 I was a student there and I took the PSAT test, I 14 think at the beginning of my junior year, and I did well on 15 that test, so I started receiving mailings from colleges all 16 across the country, but I -- I had a part-time job and I was 17 taking a full load, so I didn't really have time to read all 18 of that stuff. And just one day at school I overheard one 19 of my English teachers talking to another teacher and he 20 happened to mention that Amherst was a very good school, so 21 I went home and I looked for the stuff from Amherst, and, 22 you know, I filled out the postcard and sent it in to get 23 more information. And after I did that, the college was -- 24 you know, they kept in touch with me, encouraging me to 25 apply. 162 1 Then at the start of my senior year of high school 2 there was a minority preview weekend that was in November. 3 I believe it was in November, October, November of my senior 4 year. And, you know, I had also been getting mailings from 5 Harvard University and there was going to be an information 6 session put on by Harvard students who went to different 7 high schools around the country to recruit. And the high 8 school that they were recruiting at is Latin High School, 9 which is also a segregated high school, but it's mostly 10 white, and it's known as the best private high school 11 in Chicago, and the richest high school in Chicago. 12 So I went that one day to the information session 13 at Latin High School, and actually, I wasn't too impressed 14 with what the students had to say about the school. It 15 didn't sound like a place that I would like. So -- but from 16 that meeting, some of the Latin High School students said 17 that they wanted to put together like a road trip for the 18 students to visit Harvard, and so I asked them if I could 19 go, but I wanted to visit Amherst for the minority preview 20 weekend. So we drove and they dropped me off and I was 21 at Amherst for, I think, three days, like Friday, Saturday, 22 Sunday. 23 And during that weekend I was hosted by two black 24 students. They were freshmen and, you know, everyone that 25 I came in -- almost everyone I came into contact with were 163 1 other high school seniors thinking about applying to Amherst 2 who were all of color. Then we went to a few sessions with 3 admissions counselors, I think professors, administrators, 4 other students of color. We went to a party every night 5 at the -- it was call the Charles Drew House, which is the 6 Black Culture House. And I just felt very comfortable there 7 and it was a small school and it was beautiful, so -- and we 8 had been encouraged to apply as early decision applicants. 9 So I did apply as an early decision applicant, and I was 10 accepted. 11 Q Did you experience tracking in your high school? 12 A Yes, I did. I was tracked really from kindergarten 13 through twelve grade. When I started school my grandmother 14 used to babysit us, my mom's mom, and so she only spoke 15 Spanish and so we only spoke Spanish. My father spoke 16 English, but he just wasn't around a lot, so we spoke some 17 English, but more Spanish. So I was put into the Spanish 18 language kindergarten. 19 And so I was in Spanish language classes up to 20 second grade. And I don't know, this is just a side note, 21 but I just remember feeling desperate to learn English, 22 because I just felt like the English classes were superior 23 to ours. And in second grade, I actually was -- I have a 24 sister who is eleven months older than me, we were in the 25 same classes, and around second grade I was doing her 164 1 homework and stuff, so I got transferred into a third grade 2 class that was in English. 3 And I was in some kind of gifted program where I was 4 pulled out of class for one or two periods a day and given 5 special instruction with other kids in the gifted program in 6 special things like creative writing and things like that. 7 And then in middle school, there were actually 8 honors and regular classes, so I was put in the honors class 9 for seventh and eighth grade. And for seventh and eighth 10 grade I had a teacher who came from a family of teachers and 11 one of her siblings taught at a magnet high school, and he 12 had given her applications to give to kids in my class so 13 that we could apply to magnet high schools. And so she only 14 chose four students in my whole class to give applications 15 to so they could take the entrance exams for the high 16 schools. So I was chosen and three other kids were chosen, 17 but I was the only one who got into Whitney Young. 18 Then at Whitney Young, there was also that system of 19 regulars and honors classes and based on your score on the 20 entrance exams of the high school, you were put in either 21 track, so I was put into the honors classes. 22 Q And what was the racial composition of honors classes? 23 A They were almost all white. I would say every class 24 had about 30 students. Each class had about 20 white 25 students, five white students -- I mean five Asian students, 165 1 and, you know, the rest Latino and black. 2 Q Apart from hearing, overhearing a conversation with 3 your teachers, did you have any other notion of Amherst 4 College? 5 A No. 6 Q And did you ever get any help from a counselor? 7 A Well, I went for help to a counselor. My high school 8 had four counselors and 2,000 students, so each counselor 9 was responsible for 500 students, and their job was, you 10 know, to help people with emotional problems, to help them 11 pick classes, to help them choose colleges, and when I had 12 already made the decision to apply to Amherst, I went to 13 the college counselor to ask her how I could request a 14 transcript and to ask for financial aid information about -- 15 to see if she knew about any scholarships or anything. And 16 she just directed me to a file cabinet outside her office. 17 And that was very different treatment from the treatment 18 that, you know, my white classmates, Asian classmates got. 19 Well, when she -- she said, oh, you know, you're 20 thinking about college? That's very good. What schools are 21 you thinking of? And so I told her Amherst College, and 22 she thought I said Elmhurst College, which was a college in 23 Illinois that's not very selective, and she said, oh, that's 24 great. And I said, no, Amherst College. And she said, oh. 25 Oh, yeah, I went to, you know, a function there once, and, 166 1 you know, all I remember are pearls and white gloves. 2 You'll never fit in. 3 And that was her advice to me. So she didn't tell 4 me about any scholarships or give me any useful information. 5 I had to find out all this stuff on my own. And I found out 6 later on like at graduation where they were giving out all 7 these scholarships and awards and things, I found out that 8 there were scholarships available from my high school that 9 she just didn't tell me about. And I think that the reason 10 she didn't tell me was because of my race, that -- I mean, 11 and it just didn't happen with her, it happened every time 12 somebody was tapped, you know, to compete in something or, 13 you know, to represent the school somewhere, it was just 14 assumed that the only kids who were in the top ten, ranked 15 in the top ten, which I was, were white, and in fact, they 16 were all white except for me. So that was my experience 17 with college counseling. 18 Q And before getting to Amherst, had you ever seen 19 another college? 20 A Yes. When I was a senior in high school, I don't 21 remember how, but I qualified for -- oh, I had been a 22 volunteer at the Field Museum of Natural History in Chicago 23 and I had worked with a chemist who was a professor at the 24 University of Chicago, and so through my work there he 25 recommended me for a program where high school students 167 1 could take college level courses at the University of 2 Illinois at Chicago, which is a university that was right 3 next to my high school, and it was created for community 4 college students in Chicago so that they could have a 5 four-year university to transfer to that was right in 6 the city that wasn't, you know, as selective as a school 7 like the University of Chicago. So, you know, I took a few 8 classes there so that I could transfer those for college 9 credit. 10 Q Did you make a comparison when you arrived at Amherst 11 between Amherst and that college? 12 A Definitely. The differences were really stark. 13 First, the UIC, the campus was an urban campus, so it was 14 just -- it was about ten buildings and, you know, there was 15 like a stadium, like a concert hall, but at -- I didn't 16 even know it belonged to the school, because it was used 17 for public concerts, but it was just, you know, the ten 18 buildings, and everybody commuted. It didn't have dorms. 19 And the people who went there were mostly from community 20 colleges or they were from, you know, poor high schools 21 or -- it was still mostly white, but, you know, the white 22 students there just to me didn't seem serious about college, 23 didn't seem serious about studying, so I just thought I 24 was at a really bad college, because I felt like I was the 25 smartest person in my classes and I was a high school 168 1 student. 2 Then, I mean, compared to Amherst, just physically, 3 you know, Amherst is a really -- it looks a stereotype of 4 an east coast school, you know, it has a lot of 19th century 5 buildings, a huge gym, this big hill that people use for, 6 you know, soccer and stuff. It has the most tennis courts 7 per capita of any school in the country, things like that. 8 So physically, it was very different. It was a lot smaller. 9 I don't know how many thousands of students UIC had, but I 10 know they had -- it was in the thousands. Amherst had only 11 1,600 students, and the average class size was thirteen. 12 Q Had you had experiences with being in a minority 13 before getting to Amherst? 14 A I had -- well, up to eighth grade I had gone to 15 my neighborhood school, and I had no notion of being a 16 so-called minority. I had never even heard that term. 17 When I went to high school, the high school was 18 mostly black, but I was in mostly all white classes, but I 19 didn't feel like a minority, because the school was all 20 black. Like, it was cool to be black, and there were all 21 kinds of black people. It wasn't -- you know, the high 22 school itself was right next to a ghetto, but the students 23 didn't come exclusively from that area, the students came 24 from all over the city, from all levels of socioeconomic 25 background, so there was a lot of diversity there among the 169 1 black students. 2 And then I had worked as a lifeguard at a beach in 3 Chicago. It was -- it's known as like the showcase beach in 4 Chicago. It's the largest. It stretches a mile, and it's 5 the beach that everybody wants to work at, because it's, you 6 know, it's so nice. Everybody else gets stuck at pools for 7 the whole summer, so everybody wants to work at the beach. 8 And, you know, they say that people with the best 9 scores on their lifeguard tests get sent there, but in fact, 10 the supervisors who work at this beach chose kids, you know, 11 who were on their swim teams during the year. Whether they 12 had the best scores or not, these kids tend to go there. 13 So the first year that I worked there, there were 14 50 white lifeguards and me, and that was my first experience 15 of being, you know, a minority with the majority population, 16 and it was a really, really bad experience. 17 Q Can you say why? 18 A It was the first time that I felt representative of 19 my race, yet I wasn't on an equal playing with, you know, 20 the people who would, you know, say things to me, which were 21 the supervisors, and they would say things like, oh, you're 22 my little Mexican senorita, and, you know, if I said 23 something back, they would say, oh, hot salsa, you know. 24 If any Latinos came to the beach it was a -- mostly 25 the patrons at the beach were mostly white, because it's 170 1 right in an area called the Gold Coast where rich people 2 live, and these rich people were mostly all white, so it's 3 kind of their beach, and when Latinos would come to the 4 beach the supervisors would call them the Spanish Armada. 5 So it was that kind of thing that would just make 6 me really angry, and when I would say something back to them 7 about, you know, calling me a senorita or trying to speak to 8 me in Spanish, I would get punished. And the punishment was 9 sitting for four hours in a boat without a break, getting a 10 half-hour lunch, and then sitting for another four hours in 11 the boat without a break, and then you had to help carry 12 the boats into the boat house. 13 And this was a standard punishment. I wasn't the 14 only one punished like that. I mean, people were punished 15 for things like if you didn't go to a party at night. They 16 would assign you to go to the party, if you didn't go, the 17 next day you were punished that way. But I got punished 18 more often, because I would say things to them when they 19 would, you know, use these slurs. 20 But when I was thinking about going to Amherst, I 21 didn't realize that it was a -- well, such a white school, 22 and when I did realize that it was such a white school, I 23 didn't think that it was going to be as bad as my experience 24 at the beach, because I just couldn't -- I mean, I thought 25 that these people at the beach that, you know, they weren't 171 1 people who did well in school, that they were just a bunch 2 of jocks, that, you know, that these weren't the kind of 3 people that I was going to meet at a place like Amherst 4 College. 5 Q And did you end up finding that that was true? 6 A No. Well, I think that people's -- well, most of the 7 students had come from homogeneous backgrounds. Most of the 8 students of color had come from segregated schools and most 9 of the white students had come from all white schools. So 10 no one had much experience with anybody that was really 11 outside of their race or -- you know, minorities went to 12 minority schools, not necessarily of their race, but no 13 one had a lot of experience with each other. 14 And, I mean, my social group became the women who 15 lived on my floor in my dorm, and I think there were about 16 30 women on the floor. Three were Asian, two were black, 17 there was me, and the rest were white on my floor. 18 And, you know, I mean, I thought initially that 19 everybody was very polite, but, you know, later on we had 20 conflicts where I just felt very uncomfortable. 21 Q What kind of conflicts did you have? 22 A Well, first there was -- it was the socioeconomic 23 difference. Like, I found out that I was the only person 24 on my whole floor who was getting any kind of financial aid. 25 The tuition at the school was $23,000 a year at the time, 172 1 and they were paying it in full and that was a total shock 2 to me. I just couldn't believe that there were people, you 3 know, in this country who could afford to pay that much 4 money in one year, you know, for four years for school. 5 Another conflict was that the college had a minority 6 preorientation program. I think it was two or three days. 7 And basically, they sent letters to every student that had 8 identified as a minority student, inviting them to come to 9 the school two or three days before, you know, the -- well, 10 before the white students came to campus. 11 And they had started this program because -- well, 12 really as a result of complaints and demonstrations by 13 previous students of color who felt like they had no support 14 at the college for dealing with an environment that was 15 suddenly nearly all white. 16 So I went to this preorientation weekend and we were 17 encouraged by the different facilitators of the different 18 sessions that they had to sign up for groups like the Black 19 Students Union and Asian American Students Association and 20 Latino Student Group and groups like that. You know, so we 21 all signed up. 22 And, you know, so at the time I was the only person 23 on my whole floor who had gone to that program, so like when 24 I moved in, I was -- I was the only one on my whole floor, 25 plus the resident counselor who was on my floor. And when 173 1 everybody else moved in, and, you know, they found out 2 that -- they didn't know before they came that there had 3 been this preorientation weekend, so when they showed up and 4 they found out that the school had had this for us, that 5 they had spent money on this program and that they hadn't 6 invited the white students, you know, the white students 7 expressed anger at that. 8 Q And what did you think of that? 9 A Well, intuitively I thought that there was something 10 wrong with their reaction, and that they didn't see anything 11 wrong with their reaction, because they were complaining 12 to me, and, you know, I tried to say that, you know, maybe 13 people in the past had been victims of racism on the part 14 of other students and that the school was, you know, just 15 trying to help us cope before anything happened and just 16 trying to express support, you know, trying to say that if 17 something like this did happen to anybody, that, you know, 18 they had people in the administration to go to that would 19 be sympathetic. 20 But my white friends didn't accept that explanation. 21 They just said that this was wrong and that even having 22 these groups, these cultural groups was wrong, because, you 23 know, it encouraged us to separate ourselves, that it gave 24 us an advantage. They claimed that these groups made it 25 easier for us socially because we had an instant group of 174 1 friends, that's how they put it, an instant group of friends 2 that they didn't have, that they had to try harder to make 3 friends. 4 Q Had you experienced racism in your education before 5 getting to Amherst? 6 A I had -- I mean, one thing that I can remember very 7 clearly was when I was in high school and I was taking a 8 psychology class and it was with this teacher who never 9 had us do much work, his cubicle was right outside of our 10 classroom, so usually he would stand outside and talk to 11 people for fifteen minutes while we just sat there and did 12 nothing, and then, you know, he would walk in and just, you 13 know, just talk to us about current events or whatever was 14 going on. 15 Every once in a while we had a real class, but one 16 day, you know, he walked in fifteen minutes late, it was a 17 40-minute class, and he said, today we're going to talk 18 about stereotypes. You know, don't you guys notice that 19 here in school that, you know, people think things about 20 other people based on their race, but, you know, they never 21 say it, but you know you think it. 22 And, you know, you know that there is not, you know, 23 one Thanksgiving that goes by that your families don't say 24 things about other races, and I mean, what do you guys think 25 about that, and what do you guys think of the fact that, you 175 1 know, this class is nearly all white, and it's an honors 2 class, and, you know, 90 percent of the students in this 3 school are black. Why do you think that is? 4 And that was the first time ever in my life that 5 this topic had been introduced in a class at all, really, 6 in a group of, you know, mixed race people. So people 7 initially were reluctant to participate, so he wrote on the 8 board, white, Hispanic, black, Asian, and Jews, and then he 9 just, you know, went to each group and had people throw out, 10 you know, stereotypes that they thought, you know, were 11 common about these groups. 12 And I wasn't participating at all. I felt very 13 uncomfortable with that discussion, especially because it 14 was a class where, you know, fifteen students were white, 15 three were Asian, I don't think there were any blacks in 16 that class, and I was the one Latino. 17 So the stereotypes about white people and about 18 Asians were all positive, and about Jews they -- nobody 19 wanted to say anything about Jews. There was at least one 20 Jewish kid in the class, and, you know, so the teacher asked 21 him straight out, you're Jewish, aren't you? You had a Bar 22 Mitzvah, didn't you? And he said, yeah. And he said, well, 23 what should we put for Jews? And the kid said, well, we eat 24 matzoh balls, and that was all that they could come up with 25 for Jews. 176 1 But then when they got to blacks, it was -- the list 2 was really derogatory and after every item, everybody was 3 just totally laughing and they said things about blacks 4 like, they like watermelon. They like fried chicken. They 5 like fruit flavored soda. They cheat the welfare system. 6 There was like about a five-minute discussion about the 7 proper way to -- what to write on the board to say that 8 black men had big penises, and then the teacher said, well, 9 let's just say well endowed, you know. And also that they 10 cheat the welfare system and that they use drugs. So it 11 was a big long list compared to all the other groups. 12 And then for the Latinos, it was a similar set of 13 negative characteristics, like they are illegal. They don't 14 speak English. They are lazy. They become maids. And I 15 don't remember what the other stuff was, but -- 16 Q How many years ago was that? 17 A It was in 1989. 18 Q How vividly do you remember that discussion? 19 A Like it was yesterday. 20 Well, the thing about that discussion was, it was 21 the first time in school that I had got really, really 22 angry. I had never been so mad, so angry that I couldn't 23 speak. And, I mean, I think he meant to go on, the teacher 24 meant to go on and discuss these, you know, but we only got 25 as far as making the lists and everybody laughing about the 177 1 lists and then the class was over. 2 And when the class was over, a student sitting next 3 to me said, your nose is bleeding. And I hadn't realized 4 that I had gotten a nose bleed, and it was from being so 5 angry. So, you know, I just pinched my nose and left the 6 class and my teacher saw that. And I just said, you know, 7 I have to go to the bathroom. He said, okay. And then the 8 next day, there was no -- we didn't finish the discussion. 9 That was the end of it. It was just this list of 10 stereotypes, everybody laughing, and nothing more. 11 Q Did you go to Amherst believing you would have any 12 difficulties because of your race? 13 A No. When I had gone for that preview weekend, I 14 think because I was hosted by black students and because, 15 you know, we went to parties at the Black Culture House and 16 because the group that I was socializing with were other 17 minority high school students, I saw the white students 18 around, but I didn't get the sense that it was, you know, 19 such a white school, such a majority white school. 20 And I thought that because, you know, these people 21 were educated, Amherst is ranked the number one small 22 college in the country, I thought that because these people 23 were educated that, you know, that they would be open 24 minded, that I wouldn't face any kind of discrimination, 25 that I wouldn't feel uncomfortable, that, you know, my 178 1 stereotype of college in general, the idea that I had, you 2 know, and been inspired by my whole life before actually 3 going to college was that it was an environment of 4 transcendentalists, basically, of Renaissance people, 5 people who wanted to learn about everything, people who 6 were interested in learning. 7 You know, one of the things I had said on the 8 application for Amherst was that they look at the whole 9 person and that they encourage the students to interact 10 with each other, because you have to remember that they 11 looked at the whole person and so that there was something 12 to -- you know, that everyone could contribute, and they 13 encouraged people to look for that in their classmates. 14 So I really expected that, I took that literally, 15 and I really didn't expect to have any problem at all. 16 Q Were you valued as a whole person? 17 A Not at all. I mean, one of my first experiences at 18 Amherst was, I think we had like the two or three days of 19 the minority orientation and then we had about four days, 20 you know, with the rest of the students of orientation, so 21 it was about a week all together. I think like on the very 22 first night that the white students came, there was like a 23 welcome, welcoming reception at the President's house and I 24 went to that, you know, with other people on my floor. 25 And we got there when there weren't many people 179 1 there, so we got to meet him and talk to him, and, you know, 2 his comments to everybody were like, you know, where are you 3 from, and, you know, oh, had you wanted to come to Amherst 4 for a long time. And all of my friends had some kind of 5 connection to Amherst, like a relative had gone there or a 6 teacher had gone there or, you know, they had met him when 7 they visited, things like that, he knew their parents. 8 Except when he got to me, and he said, where are 9 you from, and I said Chicago, he said, you know, you're very 10 lucky to be here. And he knew absolutely nothing about me, 11 except that I was from Chicago, and he said, you're very 12 lucky to be here. 13 And, you know, then he asked me more about, you 14 know, what do your parents do, and I said my mom is a 15 factory worker. And, you know, where did you go to high 16 school, and I said Whitney Young. And he said, I don't 17 think we ever had anybody come here from there. You know, 18 finally, he said, you have a lot to teach people here. 19 So, I mean, that kind of set the tone for my whole 20 experience at the school, you know, if the President of the 21 college could, you know, make all of these judgments about 22 me in the first five minutes that I met him. 23 Q Was that the same treatment that white students 24 received? 25 A No. I mean, there was -- the way that he talked to, 180 1 you know, the girls that I went with, who happened to be 2 white, and he didn't say anything like that to them. And, 3 you know, there was also this white student, his name was 4 Rich Lawson, and he was also poor, he wasn't as poor as I 5 was, but his parents were divorced and he lived with his 6 father and he just told everybody, you know, like that when 7 he was little he lived in a trailer park, and, you know, 8 that his father worked so hard, but that he really couldn't 9 afford to go to Amherst, but, you know, that here he was, 10 because he had worked so hard. 11 And everybody, you know, treated him like he was 12 some kind of hero, you know, like he had overcome so much 13 and that he was so extraordinary. And, you know, he was 14 always chosen to represent the school at functions where 15 they asked alumni to donate money for financial aid and 16 he would stand up there and tell this story that became a 17 canned story, you know. I could mouth the words that he 18 was going to say when he told the story, you know. And he 19 was received very positively. He wasn't given the message, 20 you're very lucky to be here, you have a lot to teach. 21 And, you know, at the end of our four years, at 22 graduation, he got this award that the school gives 23 basically to a poor student to help them make the transition 24 from college to working or whatever they are going to do 25 after college. It was a $5,000 award. And, you know, when 181 1 the President was introducing him, he told the story that 2 this kid had always told. 3 So I just really saw a contrast that way, that, I 4 mean, I had suspected sometimes, maybe I'm getting treated 5 this way because I'm poor, you know, maybe it's not because 6 I'm not white, but when I saw how this kid was treated, it 7 made me think that it was because of race. 8 Q Did you have a roommate there your first year? 9 A Yes. 10 Q And did you ever have any conflicts with her? 11 A Yes, I had constant conflicts with her. She was very 12 wealthy. Her father was a lawyer and her mom was a doctor 13 and they each owned several businesses. And, you know, she 14 hadn't needed financial aid, of course. And, you know, we 15 tried to get along in the beginning, but she was just so 16 wealthy, I just -- I just couldn't believe it. 17 She didn't even know how to do laundry, you know. 18 In the first few days when I was going to do laundry, she 19 said, can you show me how to do laundry? And I said no, 20 figure it out, right? And then she had gone down and tried 21 to figure it out and she had put her clothes in a dryer and 22 put detergent in there and hadn't realized that it was a 23 dryer, you know, so then we went down and she learned how 24 to do laundry. 25 And, you know, I told my friends this story, like, 182 1 can you believe that, you know, she didn't know what a 2 washing machine was, and they would say, no, I have known 3 people who didn't know what a washing machine was. And 4 I don't know, I just thought that was really weird. 5 But one day early in the year, like two or three 6 weeks after school started, I had gone down to do my 7 laundry, and, you know, she was in the room, and so I went 8 down, put my clothes, you know, in the machine, and I came 9 up with my empty laundry basket and when I open the door to 10 our room, the light was off, and it was nighttime, so it 11 was dark outside, so the whole room was totally dark, so I 12 opened the door, and then she jumped out at me and said boo. 13 So I just reacted and I punch her in the nose. And, 14 you know, she just grabbed her nose and, you know, ran past 15 me out the door and started knocking on people's doors to 16 say that I had punched her in the nose and that I was a 17 violent person. And then, you know, our neighbors came 18 out, you know, to hear what had happened, and, you know, 19 their reaction was that I was a violent person, and that 20 that's why I had punched her in the nose. 21 And I said, no, I'm not a violent person, you know, 22 it's just she scared me and I reacted, and I apologized, 23 and, you know, but she -- 24 Q Was it different for you -- what was different about 25 that experience for you that she didn't understand? 183 1 A I think -- well, that was my first time being in a 2 safe environment, you know, where I just didn't have to 3 worry about my safety, and I think that was part of my 4 instincts to react by punching. If somebody jumped out at 5 me, it was because that's how I would have reacted, you 6 know, in my other world, and, I mean, but I just explained 7 to her that it was involuntary, that I was just surprised, 8 that I was scared, and that's why punched her. 9 But she wasn't satisfied. And so she called the 10 Dean of Students to complain about me. She wanted -- she 11 wanted some kind of action taken against me for assault, 12 basically. And then he called me into his office, and, you 13 know, he asked me what happened, and I told him. And he 14 just laughed about it, but, you know, and he said, well, you 15 know, just -- you know, just try not to let it happen again, 16 because people here have never been around violence before. 17 And, you know, for that whole year after that 18 happened I was just known as the violent student, and, you 19 know, my friends would joke about it, like, you know, when 20 they would say something and we would start to argue, you 21 know, just a regular argument about anything, they would 22 say, oh, you know, you're not going to punch me in the nose, 23 are you? Or if we got into a real argument where I was 24 angry, you know, they would not want to finish the argument, 25 and, you know, and then they would just say, you know, no, 184 1 I just feel afraid of you. And I would say, why, you know, 2 like, what is so different about the way, you know, I'm 3 acting that makes you afraid. And they would say, I don't 4 know, you know, you just seem violent. 5 Q And were they talking about you as an individual? 6 A No. I feel like they were talking about me as a brown 7 person, especially because it happened so early in the year. 8 There was nothing else there that would justify anybody 9 calling or, you know, making such a quick judgment to say 10 that I was a violent person. 11 Q And this followed you for the whole year? 12 A Yes. 13 Q What was it -- what was the racial composition of your 14 classes? 15 A Well, they were almost all white. Even though most 16 classes were small, the classes that I took, because they 17 were freshman classes, three of them had over 100 people in 18 them -- no, maybe about 100 people in them, and one of them 19 had about 35 students in them, but each of those classes was 20 overwhelmingly white. 21 Q Were you forthcoming with speaking in those classes? 22 A No. I mean, my very first day of class was in one of 23 those big classes and I thought of myself as well prepared. 24 I hadn't realized by the first day of class that I wasn't as 25 well prepared as the other students in my freshman class. I 185 1 thought, you know, that I was intelligent, that I had taken 2 the most challenging classes in my high school, that I had 3 done very well. 4 You know, the administrators at my high school were 5 always reminding us that we were the best high school in the 6 city, so I felt very competent, but that first day, just in 7 reaction to the number of white students that I saw in that 8 classroom, that lecture hall, it just made me feel less 9 confident and like I didn't want to be noticed, because I 10 just felt like that to them I was going to be representative 11 of brown people, because there were, in my entire freshman 12 class, there was only one other Latino student, no Native 13 American students, a number of Asians, maybe fifteen Asians. 14 Q Out of how many? 15 A Out of 416 students. 16 So that very first day, I just felt like that these 17 students were going to look to me as a representative of the 18 brown race, because I was aware by then that almost everyone 19 had come from homogeneous backgrounds, and I was just 20 convinced that that's what it was going to be. 21 Q What did you enter Amherst wanting to focus on? 22 A I wanted to be pre-med. 23 Q And was there anything that dissuaded you from 24 pursuing that? 25 A Well, when I started there was a -- well, there was 186 1 a session that we went to before classes began for students 2 interested in the pre-med program. It's not a major, 3 per se, but it's a set of classes that you should take in 4 order to be able to take the, you know, the MCAT, which is 5 the entrance examine for medical school, and we had been 6 given this list. 7 And so I was taking a calculus class and a chemistry 8 class and English and one other course called Introduction 9 to Liberal Studies, that was a class, basically a statistics 10 class, and right away, like in the first couple of weeks, 11 I started doing very badly in the three math and science 12 classes and pretty well in my English class. But where 13 the math and science classes started, they started from a 14 foundation that I didn't even have, but that I think every 15 other student in the class had. 16 Like the professors assumed that students had had 17 calculus already and I hadn't. I had only had pre-calculus. 18 Calculus was offered at my high school, but it was only 19 offered for the students who had started in the seventh 20 grade program at my high school, so there was no way that I 21 could have taken it, unless I had known that that's where 22 they were going to start, and if I had done, you know, 23 extra like in summer school or something. 24 I started doing very badly, and, you know, I told my 25 advisor, like before the middle of the semester, and, you 187 1 know, most of my neighbors on my floor who were my friends, 2 about half of them were also doing the pre-medical program 3 and we had weekly quizzes and homework and stuff and they 4 were all doing very well, so I didn't want to tell them 5 because of feeling, you know, like I was the representative 6 of, you know, Latino USA. I didn't want to tell them that I 7 wasn't doing well in these classes, because I felt like they 8 already had, you know, negative stereotypes in their minds 9 about Latinos, I didn't want to confirm them. 10 So I went to my advisor and I asked for a tutor and 11 he gave me one, but it was just impossible for me to catch 12 up, because the starting point of all of the classes, you 13 know, was something that I hadn't had yet. So I gave up 14 on the tutors and I pretty much went into the final exams 15 expecting to fail, but I didn't fail, I just did poorly. 16 And, you know, when I did so poorly, I considered, 17 you know, since I had passed, I considered taking like the 18 next set of classes still in the pre-med program, but during 19 that first semester there had been a lot of discussions 20 where a lot of female and minority students had complaints 21 about a number of professors in the biology and chemistry 22 departments, you know, and a lot of our classes are in those 23 departments. Some of them were for sexual harassment, some 24 of them were for racial harassment, and the majority of 25 complaints were from students who either had already 188 1 graduated or were about to graduate who had gotten bad 2 recommendations from a committee chaired by a biology 3 professor, one of the ones accused of racial harassment, 4 where he gave bad recommendations to these students, and 5 all of the students were either female or of color. 6 And so it was a pattern at Amherst that most female 7 and students of color didn't go to Ivy League medical 8 schools, only white males went to Ivy League medical 9 schools, and they tied this directly to this recommendation 10 that they had gotten from this committee chaired by this 11 professor. 12 So, you know, when I heard all of these discussions 13 going on, coupled with my doing badly anyway, you know, a 14 suggestion that a black student had given me was that I 15 shouldn't do the program at Amherst, specifically because 16 she said, you know, if you're not doing that well anyway, 17 and then you get to this committee your senior year, you're 18 not going to get a good recommendation even if you're doing 19 well, you know, so why waste your time, why sabotage your 20 chances, you should just do a post-baccalaureate, you know, 21 pre-medical program, where you do it all in one year. And 22 then, you know, I thought that that's just what I would 23 just do. 24 Q You said something a few minutes ago about your other 25 world. Did you have to make any decisions at Amherst about 189 1 how you were going to sustain yourself at the school given 2 the atmosphere there? 3 A Yeah, I mean, I felt like that I couldn't talk 4 about -- well, about where I came from to my class, in my 5 classes or to my friends, you know, who were all rich. 6 And, you know, my close group of friends were, you 7 know, two white women and three Asian women and they were 8 all rich, and they just had no idea about the kind of 9 community that I had grown up in, the kind of school that I 10 had gone to. 11 I could see then, you know, after a few weeks at 12 college how bad my high school had been, that there were 13 just so many things that we didn't have that all of my 14 classmates had had as high school students, you know, that 15 really made me feel like administrators at my high school 16 were stupid, because, you know, here they thought that they 17 had the best high school in the city, that they were really 18 doing something right, they were doing something good for 19 minority students in Chicago, you know, but that really we 20 had a crappy school. 21 Like, I didn't want to -- I didn't want to fit the 22 stereotype for my friends, you know, which I felt like I 23 did, or like my family did. So like, you know, early on in 24 the year we had had discussions like with the girls on my 25 floor, you know, where everybody talked about where they 190 1 came from, what their school was like, what their parents 2 did, things like that, and so I just didn't feel comfortable 3 sharing about, you know, the truth about my background, so 4 I lied and I said that my mother was a teacher, and that's 5 about it. I mean, I just said I was from Chicago. I didn't 6 say where in Chicago. You know, I just didn't talk much 7 about who I was. 8 And I mean, my family really had, like, very little 9 idea about what I was doing, where I was going to school, 10 what my classmates were like. They just had no clue. 11 Q Did you talk to your mother at some point about how 12 you were doing? 13 A Yeah, I mean, I was 17 years old then, you know, and 14 very idealistic, you know. I thought things like that, it 15 didn't matter if you were poor, that we all had the same 16 opportunities, you know, this was America, we're all equal, 17 and one of the things, too, was that your parents should 18 be there for you, that you could tell your parents anything, 19 right? 20 But I felt bad telling my mom, you know, that I 21 wasn't having such a good time at school, the thing about 22 me punching my roommate in the nose and the repercussions of 23 that, because I just felt like she would be so disappointed. 24 You know, for me to have gone to college was such a big 25 deal, you know, I was the first in my family to do that and 191 1 I got into, you know, one of the very best schools and, you 2 know, to my mom, that just -- she just thought that that was 3 totally wonderful. 4 So I didn't really want to tell her, but then I got 5 to a point where I felt like I had to tell somebody, so, you 6 know, so thinking, well, you can tell your parents anything, 7 right? I told her one day that I was -- that I wasn't 8 really happy at school, you know, that like some of the 9 things that my friends had said, just about minorities in 10 general, you know, how they resented like having anything 11 that, you know, like was in support of minority studies, 12 just describing racism in general to my mom. 13 And she -- it made her really sad, and I felt really 14 bad about it, but I felt like, well, you know, my mom will 15 help me, but it turned out then that, you know, she worried 16 and that night couldn't sleep. And like, the next day she 17 went to work and her job at the -- I mean, I have never seen 18 what it is exactly her job is, but I know she has to push 19 like metal sheets through this machine where there are just 20 a bunch of blades cutting the metal, and the next day that I 21 had told her that, she was pushing the metal sheet in and 22 cut one of her fingers, and it sliced like half of it off. 23 And I felt like it was my fault, you know, for having 24 distressed her that way. 25 I mean, so that was pretty much the end of me, you 192 1 know, sharing any of this with my mom. I just acted like, 2 you know, like I was in school and like, oh, I was learning 3 so much, and, you know, that I was going to have better 4 opportunities later on. I mean, basically, confirming the 5 ideas that she had, you know, about what my life was like. 6 I just went along with it. 7 Q And did the Amherst world ever meet your Chicago 8 world? 9 A Yeah. Well, at Thanksgiving, before Thanksgiving 10 break, my brother, I have an older brother, he was 21 at the 11 time, he was going to come to the college to drive me home 12 for the break, and I had a neighbor who was from a suburb of 13 Chicago and she was going to drive with us, and, you know, 14 so my brother had told me he was coming, and, you know, he 15 said he could be there around 10:00 in the morning. 16 And so, you know, I had told him to just, like, go 17 to this parking lot and that I would meet him there, because 18 I didn't want my friends to meet him. I was embarrassed. 19 You know, he had graduated from college, but, you know, 20 he -- not from college, sorry, from high school, but he, 21 you know -- well, he went to a semester of college, but he 22 dropped out, and, you know, he smoked marijuana, he had a 23 factory job, things like that, like, that if my friends 24 met him, they would feel scared. And you know, instead of 25 feeling angry about that, I felt embarrassed and ashamed, 193 1 so I didn't want them to meet him. 2 But it turned out, you know, he drove with these two 3 guys, two friends of his, and he got there early, at 6:00 in 4 the morning, and he found my dorm and the three of them just 5 busted into my room at 6:00 in the morning. 6 And, you know, my roommate and I shared just a 7 single, just one room, so, you know, they came in, and it's, 8 you know, these three guys, and like they all had beards, 9 they all had tatoos, they were high, and, you know, they 10 just came in and woke us up. 11 And, you know, my brother was like, hey, we have 12 munchies, what food do you have. And I didn't have any 13 food, but my roommate had a little refrigerator, and so they 14 ate her food. And, you know, she got really mad at that. 15 But they were in and out within like half an hour. 16 And when they were carrying my stuff down, you know, 17 and I was just like putting stuff away, my roommate, you 18 know, came in, you know, sat on my bed and she was like, 19 your mom is not a teacher, is she? And I said no. And, 20 you know, that was the end of that episode. 21 And then, you know, we drove home with my neighbor, 22 and then she and I drove back. I drove my brother's car 23 back with my neighbor. And my neighbor and I were going 24 to drive back to Chicago for spring -- I mean for Christmas 25 break. And so we were driving and on the way back for 194 1 Christmas break there was a storm and the car kept falling 2 into ditches, so finally, you know, like the third time it 3 fell into a ditch, they couldn't pull it out, so I tried 4 calling my brother, you know, because it was in Indiana, 5 which was about maybe five hours away from home. So I 6 called my brother collect to ask him to come, and he 7 answered the phone, but he didn't accept the call. 8 So then my friend, you know, whose father was a 9 neurologist and very busy, she called him, and, you know, 10 his response was, well, just get to a hotel, you know, get 11 some sleep, and I'll be there as soon as I can. And so then 12 he drove out, you know, in the storm to pick us up, and, you 13 know, we just left the car in the ditch. 14 And, you know, he said, well, you know, we will 15 drive you home. And, you know, we were on the expressway, 16 and he was asking me the directions to my house. They lived 17 in a suburb of Chicago that was south of Chicago, so we had 18 to go past their suburb into the city, and we were mostly on 19 the expressway, but when we got to where my house was, you 20 know, we got off the expressway, and we, you know, had to 21 pass like the railroad tracks and then the neighborhood got 22 really bad and they just got really tense. 23 And, you know, my friend's dad had been really 24 affable, really talkative the whole way until we got off 25 of the expressway, and then, you know, my friend was just 195 1 saying like, oh, my God, you know, and I was just so 2 embarrassed. They had never been to a neighborhood like 3 that, and it was like 10:00 at night. 4 And, you know, every corner that we passed there 5 were men drinking, or, you know, teenage gang bangers 6 standing on the corner, and, you know, when we finally got 7 to my house it was 10:00, and, you know, even around my 8 house there were gang bangers hanging out, and there was 9 nobody home, but my friend's dad just dropped me off and 10 took off, you know, without -- without waiting to see if 11 I got into the house okay. 12 And I, you know, I ended up just waiting at the 13 neighbor's house, but, you know, when we got back to school 14 after that, you know, this friend, this friend who had been, 15 I don't know, closer to me than the other friends, like went 16 around telling everybody that she just couldn't believe 17 where I lived, she just couldn't believe. 18 THE COURT: Is this a good time to break? Let's 19 break until tomorrow. 20 MS. MASLEY: I think we're just about done. 21 THE COURT: Oh, okay. Go on. 22 BY MS. MASLEY: 23 Q So you made it through Amherst and you did well; is 24 that right? 25 A Yes. 196 1 Q Did your GPA -- was it able to express this? 2 A After my first semester, I did well. I started with 3 a C plus average my first semester, and I ended with an 4 A minus average. 5 Q What does your GPA represent? 6 A I think what I thought was that my Amherst world, that 7 that was my future, that wherever I worked, wherever I went 8 to school, wherever I was going to function after college 9 was going to be that kind of environment, so that I should 10 get used to it, that I should just learn to cope, learn to 11 function. And I feel like that's what I did. I just turned 12 off to my world at home when I was at school. 13 I mean, after, after the first year, it was less 14 so that way, because there were more Latino students that I 15 could relate to, that had a lot of the same issues. Things 16 that I would have been ashamed to tell, you know, to share 17 with any student who wasn't Latino or black, I would laugh 18 about with other Latino students. So it got better that 19 way. And my professors were always encouraging, you know. 20 I got the sense that they wanted me to do well. But, I 21 mean, as far as the GPA, it represents progress to me, 22 and, well, hard work. 23 Q Is there a white student any more qualified to attend 24 the U of M Law School than you are? 25 A I don't think so. 197 1 MS. MASLEY: I have no more questions. 2 THE COURT: Mr. Payton? 3 MR. PAYTON: Your Honor and Ms. Escobar, if you 4 can indulge me for thirty seconds. 5 THE COURT: For no other reason than that? 6 MR. PAYTON: It's my indulgence, but I want to say 7 that Dean Lehman who was here this morning and who had very 8 much wanted to hear your testimony has class this afternoon 9 and we tried to rearrange your testimony this afternoon so 10 that it could take place tomorrow so he could hear it, but 11 that wasn't possible, and so my indulgence is, he apologizes 12 for not being able to hear your testimony. He very much 13 wanted to hear it. 14 I don't have any questions. 15 THE COURT: That's fine. And he has been here every 16 single time, so it's a shame that he wasn't here. 17 Plaintiff? 18 MR. RICHTER: We don't have any questions, Your 19 Honor. 20 THE COURT: Thank you very much for sharing with us. 21 Appreciate it. 22 Court will be recessed until tomorrow morning, 9:00. 23 Thank you. 24 (Proceedings adjourned at 4:54 p.m.) 25 198 1 CERTIFICATE 2 I, JOAN L. MORGAN, Official Court Reporter 3 for the United States District Court for the Eastern 4 District of Michigan, appointed pursuant to the provisions 5 of Title 28, United States Code, Section 753, do hereby 6 certify that the foregoing proceedings were had in the 7 within entitled and number cause of the date hereinbefore 8 set forth; and I do further certify that the foregoing 9 transcript has been prepared by me or under my direction. 10 11 12 13 ____________________________ 14 JOAN L. MORGAN, CSR 15 Official Court Reporter 16 Detroit, Michigan 48226 17 18 Date: _______________________ 19 20 21 22 23 24 25