1 1 UNITED STATES DISTRICT COURT 2 FOR THE EASTERN DISTRICT OF MICHIGAN 3 SOUTHERN DIVISION 4 5 BARBARA GRUTTER, 6 For herself and all others 7 Similarly situated -- 8 Plaintiff, 9 -v- Case Number: 97-CV-75928 10 LEE BOLLINGER, JEFFREY LEHMAN, 11 DENNIS SHIELDS, and REGENTS OF 12 THE UNIVERSITY OF MICHIGAN, 13 Defendants, 14 And 15 KIMBERLY JAMES, et al., 16 Intervening Defendants. 17 ------------------------------------/ VOLUME 12 18 BENCH TRIAL 19 BEFORE THE HONORABLE BERNARD A. FRIEDMAN 20 United States District Judge 21 238 U.S. Courthouse & Federal Building 22 231 Lafayette Boulevard West 23 Detroit, Michigan 24 SATURDAY, FEBRUARY 10, 2001 25 2 1 APPEARANCES: 2 3 FOR PLAINTIFF: Kirk O. Kolbo, Esq. 4 R. Lawrence Purdy, Esq. 5 6 FOR DEFENDANTS: John Payton, Esq. 7 Craig Goldblatt, Esq. 8 Stuart Delery, Esq. 9 On behalf of Defendants. 10 11 George B. Washington, Esq. 12 Miranda K.S. Massie, Esq. 13 On behalf of Intervening Defendants. 14 15 COURT REPORTER: Joan L. Morgan, CSR 16 Official Court Reporter 17 18 19 20 21 22 23 24 25 3 1 I N D E X 2 3 WITNESS: PAGE: 4 5 ORAL ARGUMENTS RE: WITNESS, GAIL HERIOT 6 By Mr. Goldblatt 5 7 By Mr. Washington 10 8 By Mr. Kolbo 11 9 10 REBUTTAL WITNESS CALLED ON BEHALF OF PLAINTIFF 11 KINLEY LARNTZ, PhD 12 Direct Examination by Mr. Kolbo 19 13 Cross Examination by Mr. Delery 56 14 Cross Examination by Ms. Massie 101 15 16 17 E X H I B I T S 18 MARKED RECEIVED 19 Trial Exhibit 225 54 20 Trial Exhibit 226 54 21 Trial Exhibit 227 54 22 23 24 25 4 1 Detroit, Michigan 2 Saturday, February 10, 2001 3 (At or about 8:33 a.m.) 4 -- --- -- 5 THE COURT: Let's talk first, if we might just take 6 a second, and we have -- 7 MR. WASHINGTON: Could we wait just a minute? I 8 think Ms. Massie is having car trouble, and she is going to 9 be handling today, so if we could wait for a minute, I'd 10 appreciate it. 11 THE COURT: Oh, I'm sorry. I didn't notice she 12 wasn't here. Of course, it may go much quicker. 13 MR. WASHINGTON: I understand. 14 THE COURT: I'm kidding. No, we will definitely 15 wait. 16 Can we argue the motion in terms of the witness, 17 Gail Heriot, or is Ms. Massie handling that, also? 18 MR. WASHINGTON: I can argue that, I suppose. 19 THE COURT: It's up to you. 20 MR. WASHINGTON: We are really supporting the 21 University's position, so, yeah, fine. We would be happy 22 to do that. 23 MR. PAYTON: Your Honor, Mr. Goldblatt will be 24 arguing this motion. 25 MR. GOLDBLATT: Good morning, Your Honor. 5 1 THE COURT: Good morning. 2 MR. GOLDBLATT: The Defendants have brought this 3 motion to exclude the proffered expert testimony of Gail 4 Heriot. She is a proposed expert submitted by the 5 Plaintiffs. She has submitted an expert report in this 6 matter. I believe it's Proposed Exhibit 136, Your Honor, 7 and that's a report about diversity. 8 It's a report about the opinions of Professor Heriot 9 that policies like the ones at issue here that involve the 10 consideration of race as a factor in admissions don't bring 11 about benefits, don't improve, in sum, the educational 12 experience. 13 As the Court is aware, the Defendants have prepared 14 a case directed exactly at that issue. We believe it is the 15 most rigorous, comprehensive, thorough-going empirical case 16 proving that students who are brought together on campuses 17 that are diverse in many ways, including with respect to 18 race, receive a better education. They learn better, the 19 schools prepare better citizens, the law schools prepare 20 better lawyers. 21 And that interest, that diversity is compelling, not 22 just as that term is used as a matter of legal doctrine, but 23 in its plain English sense that that interest is compelling. 24 It's a case, Your Honor, that the Defendants and the 25 University are quite proud of. We would be delighted to put 6 1 on that case. 2 And we had, of course, a number of witnesses 3 prepared to speak to that issue, Professor Garin, Darren 4 Bok, Thomas Shagru, Albert Camareo. The Court hasn't seen 5 any of those witnesses, and the reason for that, of course, 6 Your Honor, is that this Court's ruling on December 22nd 7 said that that question was out of this trial. 8 Our testimony, the reports that we put in, are part 9 of the summary judgment record, and everyone agrees, they 10 are going to travel with this case as it makes its way 11 through the Federal Judiciary, and that's the Court's ruling 12 and we will, of course -- 13 THE COURT: You mean everyone is not accepting my 14 ruling in this case? 15 I'm just kidding, go on. I thank God for the Court 16 of Appeals every day, so. 17 MR. GOLDBLATT: Your Honor, this is what Mr. Purdy 18 said when we were talking about this issue. He said, "And 19 of course, Your Honor, as everyone has made clear, no one is 20 contesting that there are educational benefits of diversity. 21 It is simply not an issue in this case." 22 Now, what Professor Heriot's report says is that 23 there aren't sufficient benefits to justify policies like 24 these, and in that sense, what the Plaintiff is purporting 25 to do is to put on a witness who is going to say that which 7 1 Mr. Purdy told this Court isn't true. And Your Honor, we 2 just think that the Plaintiffs should be stopped from doing 3 that. 4 But there is another reason, also, Your Honor. It's 5 that Professor Heriot isn't an expert within the meaning of 6 the Federal Rules of Evidence. She shouldn't be allowed to 7 offer opinion testimony in this case. She certainly has 8 opinions, but there are cases, as this Court is of course 9 aware, like Daubert and Cumo Tire that make clear that 10 before those opinions become evidence in a case this Court 11 plays a gatekeeping role, and before she is permitted to 12 offer opinion testimony there are certain standards that 13 need to be satisfied. 14 Professor Heriot doesn't have any degree that's 15 relevant to her testimony at all. She is a lawyer. She has 16 the same degree that many of us have. She teaches court 17 law. She says that she has, quote, "studied the policy and 18 legal aspects of the issue of racial preferences." That's 19 from her deposition, Your Honor. And she said those two 20 things, the policy aspects and the legal aspects, are 21 interconnected. But this isn't a subject that she has 22 even taught. 23 The law is clear, Your Honor, that the testimonial 24 latitude, I think that's a quote from Daubert, that expert 25 witnesses get that allow them to say more than just their 8 1 firsthand factual testimony isn't something a witness earns 2 just by having an opinion. There are standards, and with 3 respect to the specialized knowledge to which an expert 4 witness will testify there needs to be evidence that the 5 testimony is reliable, that it's more than just one person's 6 opinion. 7 This Court mentioned yesterday that there was a copy 8 of the Daubert opinion on the Bench, and it's actually an 9 encapsulation of that opinion and subsequent Supreme Court 10 case that I mentioned, it's the Cumo Tire case, and on 11 pages, I think it's 149 and 150 of that case, the Supreme 12 Court tells us exactly what the questions are that we're 13 supposed to ask here. There are essentially four questions. 14 Is there a theory or a technique that's been tested? 15 THE COURT: Does it say -- I don't have the case 16 here, I have the same thing you have, the four questions 17 right there. 18 MR. GOLDBLATT: Exactly, Your Honor. 19 Is there a theory or a technique that's been 20 tested? Have the opinions been subject to peer review 21 and publication? Are there standards we can look to to 22 determine that the opinions are reliable? Is there some 23 technique or theory that's gained general acceptance within 24 a relevant scientific community? 25 Your Honor, the answer to every single one of those 9 1 questions is no. This isn't a close case under Daubert and 2 Cumo Tire. 3 Another subject that Professor Heriot discusses in 4 her report is Proposition 209 and its effects, but it should 5 be noted, Your Honor, that Professor Heriot doesn't even 6 teach at a public university in the University of California 7 system. She teaches at a private university, the University 8 of San Diego Law School, a university that, by the terms 9 of Prop 209, isn't affected. It's not, Your Honor, the 10 University of California at San Diego that Dean Garcia was 11 discussing, it's a private university not affected by 12 Proposition 209. 13 And in this case, the Daubert qualifications of the 14 witness with respect to that question, the effects of 209, 15 aren't any better than they are with respect to the benefits 16 of diversity. All four questions, again, all four Daubert 17 questions are answered no. 18 Your Honor, this is really just one person's 19 opinion. It's exactly the kind of testimony that cases like 20 Daubert and Cumo Tire say aren't evidence, aren't evidence 21 in the Federal Court under the Federal Rules. I don't doubt 22 that these opinions are genuine, that they are sincerely 23 held, but that doesn't make them competent evidence and 24 Professor Heriot should be precluded from offering them 25 here. 10 1 THE COURT: Thank you. 2 MR. WASHINGTON: Your Honor, we certainly agree with 3 Mr. Goldblatt's remarks on this. I would just add a couple 4 more. 5 I think insofar as Professor Heriot purports to 6 testify as to the effects of 209 on the educational system 7 of the State of California, there is no evidence that she 8 has ever taught at the University of California other than 9 in speeches or forums organized by the proponents of 209. 10 There is not even any indication that she has ever had any 11 connection with the University of California. There is no 12 indication that she has published any studies other than 13 popular articles about the University of California. 14 Her report contains at most some alleged statements 15 about grades at the public university in the City of San 16 Diego, which as I understand is a city of something like 17 three million people, and there is no indication where or 18 how she got that or how she has any particular expertise 19 at all. 20 There is no technique or theory here that this 21 witness has. She has not submitted anything to any peer 22 review. There is no kind of background in education, 23 anything of that nature, and in that regard, I think this 24 is really the rankest of lay testimony by a partisan in the 25 debate brought here to say things which aren't true, but 11 1 more relevant than that, are not competent as testimony. 2 So we certainly think this witness should not be 3 allowed to testify. 4 MR. KOLBO: Your Honor, I guess my response in a 5 few words is, surely they jest. They must be making these 6 arguments, in part, tongue in cheek, and I want to address, 7 first of all, the relevance issue. 8 It's true that we made some objections early in this 9 case to any testimony about the value of diversity, and I 10 suspect if there hadn't been any testimony about the value 11 of diversity, and there has been a lot in this case from 12 both the Defendants and the Intervenors, that we wouldn't 13 be at least putting on Professor Heriot to talk about that 14 issue. 15 And we understand that the Trial Court's issue is 16 limited, but this case, as has been mentioned, is going 17 to be heard on appeal and we think it would just not be 18 appropriate for there to be lots of evidence on the 19 Defendants' side of the case, on the Intervenors' side of 20 the case, about the value of diversity, but none, even 21 from one witness, the one witness that the Plaintiffs have 22 offered in rebuttal on that particular issue. We think that 23 would be just highly inappropriate. 24 And just so I can illustrate my point, Your Honor, 25 when Mr. Goldblatt says that they had a case that they 12 1 wanted to bring with respect to diversity and they didn't 2 bring it, it's true, there are some witnesses that they had 3 on that issue that they haven't called, but they elicited 4 very directly and clearly and over our objection a number, a 5 substantial amount of testimony on the issue of the value of 6 diversity, what critical mass is, and why it's important. 7 They obtained that testimony, Your Honor, not 8 necessarily always from qualified or formally qualified 9 experts, but from people who are educators. 10 President Bollinger testified why he believes 11 diversity is important, why critical mass is important. 12 Dean Lehman testified about why diversity is important, why 13 critical mass is important and how you go about achieving 14 it. Professor Lempert testified on that issue, as well, 15 at length, and of course, they called Professor Severud 16 on that issue, really only on that issue in terms of the 17 importance of diversity and how you get it and so forth. 18 So it just seems absurd to me, Your Honor, to 19 suggest that it's not an issue in the case, and therefore, 20 the Plaintiffs shouldn't be able to call a witness even in 21 rebuttal on those issues. It just doesn't make -- it 22 doesn't pass the smile test, Your Honor, is I guess what 23 I would say about that. 24 With respect to the substance of Professor Heriot's 25 testimony, she is not going to testify just about diversity 13 1 and whether it's important. Her report, Your Honor, is 2 before the Court and I think you have a sense about what 3 she is going to testify about. Her view is, there is 4 nothing wrong with diversity, certainly diversity, 5 intellectual diversity is a good thing. You'd rather have 6 it. 7 But one of the things she is going to testify to is 8 how you go about getting it, and what she says, you don't 9 have to work very hard to get diversity in the classroom, 10 it pretty much comes naturally. If you're going to have a 11 classroom of 30 students, people are individuals, they bring 12 by their very nature different experiences, outlooks and 13 backgrounds, so you don't have to do much to bring it and 14 you certainly don't have to use racial preferences to 15 achieve diversity in the schools. 16 We think that's relevant to the issues in the case 17 and it's certainly, it's certainly relevant rebuttal 18 testimony to the voluminous testimony that the Defendants 19 have brought. And I haven't even mentioned the Intervenors, 20 Your Honor. They have marched student after student up here 21 to talk about their classroom experiences and so forth. We 22 had Professor Garcia yesterday talking about diversity on 23 campuses that he's not even involved with at the University 24 of California system. 25 So I think it's just frankly a little absurd to 14 1 say that the issue isn't in the case, and therefore, the 2 Plaintiffs shouldn't be able to bring a witness to respond 3 to their witnesses. What they are really trying to do, 4 Your Honor, is to suggest that there should only be one side 5 heard on this issue. Their side has been heard and our side 6 should be heard, and from the one witness that we intend to 7 call on this issue. 8 If I may then, Your Honor, address the issue of 9 qualifications, again, I have to say that this brings a 10 little bit of a smile to my face when I hear that Professor 11 Heriot, first of all, is not qualified to testify on the 12 subject of legal education, the very same subject that, 13 again, President Bollinger testified about, Professor and 14 Dean Lehman testified about, Professor Lempert testified 15 about, Professor Severud testified about. 16 Our witness, Your Honor, Gail Heriot, is a professor 17 of law and a full professor of law. She has been teaching 18 since 1989. She was Dean for a year at George Mason 19 University. When we're talking about diversity, as I 20 understand, what I have heard from the witnesses on the 21 other side, we're talking about what goes on in the 22 classroom, what's important, what does diversity bring to 23 the classroom, what does it bring out of the students, how 24 does it affect the teacher's performance and so forth. 25 I don't know how anybody could bring more expertise 15 1 to that subject than someone who is, in fact, a legal 2 educator and I think that's all that's required to pass the 3 expert testimony threshold in this case. That's all it took 4 for their experts on that subject. That's all it took for 5 their lay witnesses to testify on the subject of the value 6 of diversity. So there is just -- it's just not a serious 7 argument to suggest that our witness, a full professor of 8 law and a teacher of education and one who has thought about 9 and researched some of these issues, doesn't have the basic 10 threshold qualifications to present testimony. 11 Now, I understand they think their experts are 12 better qualified. They think more of Professor Severud than 13 they think of Professor Heriot in terms of credentials and 14 they are certainly entitled to think that and they can 15 certainly argue in terms of the weight of the testimony on 16 that. We think Professor Heriot is very highly qualified 17 and her opinions will be or should be well received by the 18 Court, so it's just not a matter of throwing her testimony 19 out. 20 And finally, Your Honor, with respect to the issue 21 of California, again, I have to smile. They are suggesting 22 that a professor in California is not appropriate to testify 23 about the effects of Proposition 209 in California. I 24 didn't hear exactly what Professor or President Bollinger's 25 or Dean Lehman's or Professor Lempert's or Dean Severud's 16 1 qualifications were to testify about California, and they 2 did testify about what they thought would be the dire 3 consequences in Michigan, based on California, if race 4 couldn't be used as a factor, so it's just not a serious 5 argument to suggest that their Michigan professors are 6 better able to tell this Court about California than a 7 California professor. 8 And finally, Your Honor, with respect to that issue, 9 they seem to misunderstand the point that we thought has 10 already been brought out in this case, which is the 11 consequences of Proposition 209 aren't just limited to 12 the Berkeley campus and they aren't just limited to the 13 University of California system, and they are not even just 14 limited to the public schools of California. 15 One of the things that Professor Heriot is going to 16 testify concerning is that Proposition 209 has had positive 17 consequences, and those positive consequences extend to 18 schools that aren't even in the California system, schools 19 like hers, where she has seen improvements in the quality 20 of classroom discussion and so forth as a result of 21 Proposition 209 and what she believes should be attributed 22 to it. So there is a great deal of relevance there and she 23 certainly has foundation to testify in these matters, Your 24 Honor. 25 THE COURT: In this matter, I know, Mr. Goldblatt, 17 1 you would like to say some more, but let me just -- no use 2 taking a lot of time on it. 3 As to qualifications, I will have to listen to what 4 the qualifications are. I hear both sides arguing what the 5 qualifications may or may not be, but I'll tell you in no 6 uncertain terms that I'm going to use the same standard 7 that I have used throughout this case, and there have been 8 witnesses that have testified that have done no studies, 9 there have been witnesses that have testified that they have 10 had no formal teaching and so forth, but that the Court has 11 ruled that they could testify, because the reason being is, 12 that they do have special knowledge because of their 13 position and so forth. 14 For instance, I think Mr. White was very helpful 15 after we heard his testimony and I think he was very 16 credible in terms of what he had to say and the statistics 17 and so forth, though he had no formal training, hadn't done 18 any personal studies or anything of that nature, and I am 19 going to use the same kind of test. 20 Even Dean Severud, obviously, had no formal 21 training, but he had been around a long time and had taught 22 courses, he had done things that perhaps, judicial Daubert, 23 you know, you would say, well, that wouldn't apply. 24 I only give you these examples because I know that 25 this person is coming from a long way and so forth, but I'm 18 1 saying that I can't rule on her qualifications until I hear 2 them, until I have had an opportunity to have the Plaintiff 3 present them and the Defense and the Intervenors voir dire 4 the witness and then I can make that determination. It's 5 hard to do in the abstract, though. 6 As I indicated, I have listened to both sides and 7 I want both sides to know that I'm going to use the same 8 standards that I have used for everybody at this point. 9 My other issue is that certain matters have arisen 10 that were important and I wanted to hear them, and I have 11 heard them, that perhaps the Plaintiff was not anticipating 12 or so forth, and this is rebuttal, and I'm going to treat 13 it as rebuttal and it's going to be limited to -- her 14 testimony, if permitted, is going to be limited to rebut 15 those things that have been put into the record by the 16 Defense and the Intervenors. 17 So with that said, I suspect the Plaintiffs will 18 call and we will have some voir dire as to qualifications 19 and I will make that ruling, and my ruling, should I allow 20 her to testify, will be that it's limited to rebut those 21 issues that have been raised by the Defense or by the 22 Intervenors. 23 Okay. Mr. Goldblatt? 24 MR. GOLDBLATT: No, thank you, Your Honor. 25 MR. KOLBO: Your Honor, we appreciate the Court and 19 1 the Parties' agreement to take our first rebuttal witness 2 out of order. We will recall Professor Kinley Larntz to 3 the stand, Your Honor. 4 THE COURT: Welcome. 5 K I N L E Y L A R N T Z, P h D, 6 having been previously called as a witness herein, and after 7 having been first duly sworn to tell the truth, the whole 8 truth, and nothing but the truth, was examined and testified 9 as follows: 10 DIRECT EXAMINATION 11 BY MR. KOLBO: 12 Q Dr. Larntz, thank you for coming again. 13 In the interim, I think it's been a couple of weeks 14 or so since your testimony, you understand that Professor 15 Steven Raudenbush offered testimony with resect to the same 16 issues, in some respects, in response to your testimony? 17 A Yes, I understand that. 18 Q Now, you weren't here in the courtroom for that 19 testimony, were you? 20 A I was not. 21 Q But have you been able to review a transcript of 22 Dr. Raudenbush's testimony? 23 A I have. 24 Q And do you understand that Dr. Raudenbush took issue 25 with some of -- some aspects of your testimony and report? 20 1 A Yes. 2 Q Have you also reviewed your own transcript, trial 3 transcript testimony in this case? 4 A Yes. 5 Q Were you able to ascertain from reading, perhaps 6 rereading or reading your testimony, as well as 7 Dr. Raudenbush's testimony, were you able to ascertain 8 whether you had already addressed or covered some of the 9 issues that Dr. Raudenbush took issue with in his testimony? 10 A Well, certainly. I mean, we exchanged our reports 11 before and I, in my direct testimony, offered information 12 on a number of criticisms that he talked about in his 13 testimony, yes. 14 Q And did you, in reviewing your testimony -- you 15 understood that you had already responded to some of those 16 or anticipated some of those in your direct testimony? 17 A A good number, I think, yes. 18 Q Did you also come to the conclusion that there were 19 some aspects of Dr. Raudenbush's testimony that you hadn't 20 really addressed or hadn't been explained in your direct or 21 cross examination that occurred? 22 A There are a few points, yes. 23 Q Okay. I want to just limit, obviously, our questions 24 and answers this morning to that, so please keep your 25 answers limited to that respect. 21 1 And I'm going to ask you about some criticisms that 2 Dr. Raudenbush raised and to the extent that you don't 3 believe they were addressed either in your prior direct or 4 in your cross examination, I would like you to respond to 5 some of these criticisms, if you have some opinions. 6 First of all, do you recall that Dr. Raudenbush had 7 a criticism with respect to the fact that there are -- that 8 in determining or ascertaining odds ratios, there are high 9 odds ratios that don't reflect large differences in 10 probability; that is, at higher ranges of probabilities, so 11 they don't reflect large odds ratios? 12 A There were cases presented by Dr. Raudenbush, and also 13 I think in my cross examination relating to odds ratios that 14 may be relatively high where the difference in probabilities 15 is not -- well, depending on how you think, not great. 16 Q Do you understand what the criticism is of your report 17 and testimony in that area? 18 A Well, I think my understanding is that the -- while 19 there may be relatively high odds ratios, there isn't much 20 of a difference in probability, and I think it's important, 21 I think it's very important that the Court understand and 22 that everyone understand the effect of odds ratios on 23 probabilities. 24 And I tried originally to provide some illustrations 25 and they provided some illustrations and I just want to make 22 1 sure that we're very clear on the relationship from a 2 baseline probability how a particular odds ratio would 3 translate into another probability. 4 So for instance, a baseline probability, I think in 5 my direct examination we talked about a baseline probability 6 of 10 percent going to 90 percent from an odds ratio of 81 7 and so I wanted to -- I wanted to make sure we were clear 8 about how that works over the whole range of probabilities. 9 Q And did you read in the reports of Dr. Raudenbush's 10 testimony where he compared relative probabilities of, say, 11 90 versus 99 percent and the effect of going from 90 to 99 12 to 999 and so forth? 13 A Yes, yes, I did read that. 14 Q And what is demonstrated by that? 15 A It's again illustrating for a certain baseline 16 probability how relative odds of a particular value would 17 translate into another probability for another group. 18 Q And I think you have testified, you indicated this 19 morning that you have prepared something to illustrate the 20 range of probabilities? 21 A What I did, what I did is it's in the booklet that I 22 think you have, and the last page of the booklet, what I did 23 was prepare just a table, and we have it as a slide. 24 I prepared a table for a range of probabilities over 25 the whole range, rather than just choose a certain point, 23 1 where I used 10 percent as a base, and they, Dr. Raudenbush, 2 used 90 percent for the whole range of probabilities, going 3 from a base of one percent, a baseline probability of one 4 percent up here in the corner, all the way down to, I think 5 it's 99, down in the bottom. 6 And so what I did is I did it for various relative 7 odds and I chose values that we have seen results of, 5, 10, 8 20, 50, 100, 200. I have chosen -- what I have done is I 9 have said, how does this baseline probabilities translate 10 into the probability of another group that has a relative 11 odds of a given amount. 12 So for instance, for instance, Dr. Raudenbush had 13 90 percent, and I think was looking at 90 versus 99, and 14 that's a relative odds of 11. I don't have 11 on this 15 table, but I have 90. I have 10, and 90 goes to what, 98.9. 16 And then talking about 111, I think, was one of the ones he 17 used, and 90 goes to -- he had 111, I have 100, and 90 goes 18 to 99.89. 19 So these are -- I'm referring to them as fractions, 20 but we often talk in terms of percentages. And other 21 values, well, for instance, 10 percent, I had reported 22 10 percent with an odds ratio of 81 going to 90. Well, we 23 can see 10 percent here going, well, at 100, goes to 91.7. 24 And all I have done on this table is just to make sure we're 25 clear what the effect of a relative odds is on a baseline 24 1 probability. And that's the only purpose of the table, is 2 for clarification. 3 Q Let me ask, Doctor, Dr. Larntz, do you disagree with 4 Dr. Raudenbush's premise that when you start with a high 5 baseline probability like 90 and then have a comparative 6 probability of 99, that odds ratios go up rapidly as 7 those probabilities diverge? 8 A Well, sure, the odds do, the odds ratios do go up 9 mathematically. They are what they are, and that's what 10 this table is trying to just illustrate, what the 11 mathematics is, and I think that's clear. 12 In point of fact, in that particular case, if you 13 look at the percentage denied, obviously they go up or 14 they go from 10 percent to one percent, so that's a big 15 difference in that probability scale. 16 Q So as I understand it, you agree with Dr. Raudenbush 17 on that fundamental principal? 18 A On the calculation, absolutely, there is no 19 disagreement on the calculation. That's why I provided 20 the table, so you can see the whole range of possible 21 calculations. 22 Q And does that mathematical fact, does that change 23 anything with respect to your opinions on the extent to 24 which race is taken into account in the admissions process? 25 A No, no. 25 1 Q Dr. Larntz, were you also -- did you also understand 2 that Professor Raudenbush in his testimony took issue with 3 your -- or criticized you for what he called discarding some 4 of the data in your analysis or selectively attending to the 5 data? 6 A Yes, I did read that, yes. 7 Q Do you have an opinion with respect to the validity of 8 that criticism? 9 A You mean the fact that different parts of the data 10 provide different amounts of information? I certainly have 11 an opinion that that's true, and I'm not quite sure -- in 12 the sense that certain cells in the grid provide more 13 information than other cells, that's true. 14 Q What would be an example of that? 15 A Well, I mean, obviously cells that have no one 16 admitted would provide no information about admission rates, 17 in comparison of admission rates. Cells that have everyone 18 admitted provide no comparative information. 19 And that's what we're talking about here, because 20 we're trying to compare the admissions of minority students 21 to majority students. So the different cell grids in fact 22 do provide different amounts of information. 23 Q Okay. And as I understand it, Dr. Raudenbush has 24 suggested that you did not consider certain cells in the 25 data. 26 1 A Well, in the calculation, every cell entered the 2 calculation and the computer made the judgment of how much 3 information would be -- you know, how much information was 4 there. There are some cells that the computer found that 5 there was no comparative information and other cells where 6 it found comparative information and used all the cells that 7 had comparative information in deriving the summary relative 8 odds ratios that we talked about. 9 Q What would be the kinds of cells that would provide no 10 comparative information, what categories would those fall 11 into, into terms of what you saw? 12 A Well, cells in a grid that would have no one admitted, 13 okay, there is no comparative information, everyone 14 admitted. And there are very few cells like that, but there 15 are a few, okay? Cells, if there are no people in the 16 cells, there is no comparative information. 17 And in fact, the one additional category is if 18 everyone in a cell was a member of one of the ethnic groups, 19 just one, so if everyone in the cell, for instance, if 20 everyone in the cell were caucasian and there were no 21 minority students in that cell, or no member of any other 22 ethnic group, then that wouldn't provide any comparative 23 information, because everyone would be the same. 24 Q What would be an example of a cell that provided some, 25 but not very much information? 27 1 A Oh, there are cells, for instance, a cell where 2 virtually everyone is admitted, but not everyone, then that 3 cell would provide -- may provide comparative information, 4 but the amount of information in how much it contributes to 5 the estimation of the overall relative odds would be small. 6 Q Have you prepared some charts to illustrate what cells 7 provided comparative information in your analysis? 8 A What I have done is for each of the years, and this is 9 also in the book, for each of the years what I have done 10 is -- and we have the first one for 1995. What I have done 11 for each of the years is looked at the original grid, this 12 is for all applicants, and I have highlighted, I have 13 highlighted in yellow the cells that provide comparative 14 information. 15 So all the cells that provide comparative 16 information are highlighted in yellow and the cells that 17 are not, the blank cells, except for the permanent cells, 18 the blank cells in the balance of the table for which there 19 is no comparative information, that is, in the sense that 20 the computer would check and say, well, there isn't anything 21 I can add based on these cells, those are left blank. 22 Q Would there be a way of including the cells that have 23 no applicants, first of all? 24 A Cells that have no applicants? 25 Q Yes. 28 1 A Whatsoever? 2 Q I'm sorry, that have no -- for across racial lines 3 there are no admits, or let's take, for example, no 4 admissions. 5 A Well, there -- if there is no admits, then there is no 6 comparative information, so they -- the computer makes a 7 judgment that there is no comparative information there and 8 so it attributes -- it doesn't actually contribute to the 9 estimation. 10 Q Did Dr. Raudenbush in his analysis use those cells in 11 some fashion, cells in which there are applicants, but no 12 admissions from any racial group? 13 A Oh, I believe that in some of his models he used those 14 cells, yes. 15 Q And does that have any effect, as far as you're 16 concerned, on the validity of the results? 17 A Well, with respect to providing comparative 18 information you would have to have a model that says how 19 they enter, and so based on how his model works, that model, 20 whether that model is true or not depends on how that 21 information would enter. I'll just say it that way, 22 that's true. 23 Q Would it provide a more accurate her less accurate 24 picture of the -- as far as you're concerned -- the extent 25 to which race is taken into account, like including these 29 1 cells in which there are no admissions from any racial 2 group, or would there be a distortion there or what 3 would be -- 4 A Well, I actually don't know what the effect is, I 5 mean, as far as that goes. With respect to the statistical 6 principle of looking at cells with comparative information, 7 they don't provide comparative information, so that's what 8 I know. 9 Q And so you have got in front of you what we have up 10 here, 1995? 11 THE COURT: They don't provide any comparative 12 information, in your opinion. If you included them, would 13 there be a downside to including them, even though they have 14 no information or is there just no reason for it or -- 15 THE WITNESS: Well, I mean, what you have to do 16 is you have to model the whole grid, okay, and that to me, 17 that's a hard process, okay. I looked at it and took the 18 comparative information and compared cell by cell. To me 19 it's harder to model the whole grid. It's just a more 20 difficult problem and your results would depend upon how 21 you did that modeling. 22 THE COURT: So there would be another level of 23 discretion, because that new level of discretion is now an 24 extended modeling of the whole thing? 25 THE WITNESS: That's the way I looked at it. 30 1 THE COURT: I mean, is that pretty much -- I 2 remember you talked about the less that you have to do in 3 terms of discretion, the better the results sometimes are. 4 THE WITNESS: That's right. 5 THE COURT: That's what you told us. 6 THE WITNESS: Right. The kind of modeling would be, 7 there is some kind of -- you know, I'll try to be as clear 8 as I can. You basically have a surface that you're modeling 9 here. You're trying to model a response surface over this 10 whole grid and you have to make assumptions about how to do 11 that, and what I did is, I didn't do that, and then what 12 you -- I have to use two hands, sorry. 13 THE COURT: That's okay. 14 THE WITNESS: You model, say, the majority students 15 one way and the minority students in another and sort of the 16 difference here becomes the odds ratio, right? So depending 17 on how you do that modeling, depending on how you do that 18 modeling, you could get different answers. 19 THE COURT: And the more modeling you do, the more 20 assumptions you make? 21 THE WITNESS: Yeah, well, sure, yeah. 22 THE COURT: Your position is that you did -- you 23 tried to do as little modeling and as little assumptions as 24 you can to actually get the figures? 25 THE WITNESS: Right. And exactly in this way, what 31 1 I did is I said, let's not try to model the whole surface, 2 let's just look at it one point at a time, and look at the 3 difference one point at a time. And that's basically what 4 I did, was look at the difference one point at a time. 5 THE COURT: Creating a situation where you have to 6 make fewer assumptions? 7 THE WITNESS: I think there are fewer assumptions 8 in that, surely there are fewer assumptions in that, yes. 9 BY MR. KOLBO: 10 Q And you have done this for each one of the years in 11 question? 12 A That's right. 13 Q Can we just take a look at 1996? 14 A I think so. 15 Q Same pattern? 16 A Oh, same pattern. The cells that were no admitted 17 were the predominant cells that were left out and they are 18 the ones that have very low test scores and very low GPA's 19 and those students just, you know, they didn't admit them. 20 So that's true universally. There are a few other cells 21 that don't provide very good information for the other 22 reasons I said, but predominantly, they are the lower grade 23 point averages and the lower test scores. 24 Q And 1997, can we see that same pattern here? 25 A Same pattern. 32 1 THE COURT: And '98, perhaps -- 2 THE WITNESS: Same pattern. 3 BY MR. KOLBO: 4 Q And then '99. I wanted to ask you about a cell or a 5 couple of cells for 1999. I don't think these are -- these 6 kinds of cells show up on the earlier versions, but there is 7 a cell, for example, under grade point 3.0 to 3.24 and 170 8 and above in which there are nine applicants and three 9 admissions; correct? 10 A That's correct. That's the cell over on the -- let me 11 see, it's the cell right over here in this corner, 170 and 12 above, and this nine and three here that's not shaded. 13 Q And it's not shaded, meaning that it didn't provide 14 comparative information? 15 A That's right. 16 Q And why in that case was there no comparative 17 information? 18 A Well, I don't recall exactly the group, but all nine 19 of those applicants were from one ethnic group, so they were 20 from one ethnic group. 21 Q That shows no comparative information across racial 22 lines? 23 A That's right. 24 Q Did you -- there has been some testimony, I think, 25 from Dr. Raudenbush about the number of cells or percentage 33 1 of cells or so forth that were used in your analysis or that 2 were -- that, as you have testified, provide comparative 3 information. Can you -- do you have some idea in terms 4 of the number of applicants that contributed comparative 5 information to your analysis across these different years? 6 A Yes, I did calculate the percentage of applicants that 7 are in the shaded areas and I don't remember exactly from 8 each year, but it's between 84 and 88 percent across the 9 six years. 10 Q That all provided comparative information? 11 A That's right. 12 Q Dr. Larntz, then if I can ask you, do you recall 13 that in reading Dr. Raudenbush's testimony that he had a 14 criticism with respect to your reporting of uniform odds for 15 composite odds ratio assumptions? 16 A Yes. 17 Q Do you understand what his criticism is in that 18 regard? 19 A Well, I understand the technical, statistical aspect 20 of the criticism, sure. 21 Q Do you agree or disagree with it? 22 A I disagree on its importance, and I am not sure if 23 that assumption is technically satisfied or not, but in 24 fact, the importance of the assumption is that my composite 25 odds ratios, if in fact that assumption is not perfectly 34 1 satisfied, and no assumptions are ever perfectly satisfied, 2 if that assumption is not perfectly satisfied and there are 3 some cells that have higher odds ratios than others, then 4 the number I gave you would be a composite that would be 5 lower than they would be for the ones that are high and 6 a little bit higher than the ones that are lower. 7 Q Do you have an opinion as to whether Dr. Raudenbush's 8 criticism -- does it change any of your opinions with 9 respect to the extent to which you believe race is 10 considered in the admissions process as reflected by 11 your analysis? 12 A No, it doesn't change my opinion with respect to 13 that at all, no. 14 Q Can you explain why not? 15 A Well, I mean, well, first of all, I think the odds 16 ratios we see are a direct reflection of the cell. We can 17 compare the grid cells and see high odds ratios in the grid 18 cells and the summary numbers that I provided, I think, are 19 a reflection of those, of those values. 20 Q Do you recall Dr. Raudenbush testifying that there are 21 a lot of large odds ratios in the middle ranges of the 22 grids? 23 A Oh, I believe that he said that there are very high -- 24 or there are high odds ratios in the middle, yes. 25 Q And do you agree or disagree with that? 35 1 A Yes, if there weren't higher odds ratios in the middle 2 we wouldn't have composite odds ratios. If there weren't 3 high odds ratios someplace, and they're mostly in the middle 4 where there is a difference in admission rates, then we 5 wouldn't -- the composite odds ratios wouldn't be as large 6 as they were. 7 Q Do you understand Dr. Raudenbush's testimony to be 8 that the odds ratios are smaller at the upper end of the 9 grid? 10 A I think I -- I think the implication of what he said 11 was they were smaller in the upper end, that's right. 12 Q Do you agree or disagree with that? 13 A Well, I think in the upper end it's very hard to tell, 14 because there is really -- when almost everyone is admitted, 15 there is -- you can estimate odds ratios, but there is -- 16 the amount of comparative information up there is relatively 17 so small, so they don't contribute a great deal to the 18 composite odds ratio, because if almost everyone is admitted 19 the comparative information there isn't great, so I think 20 it's very hard to determine the relative odds up in the 21 corners unless you create some kind of model. 22 Q Where does most of the comparative information come 23 from him? 24 A Well, most of the comparative information is down in 25 this area here where there are clear differences and where 36 1 there is some discretion being made with respect to the 2 admissions process where not everyone is admitted. 3 Q The middle of the, primarily, shaded area? 4 A Well, actually, it's the lower part of the shaded 5 area, if you want to call it, the middle of the grid. 6 Q And then do you recall reading a criticism of 7 Dr. Raudenbush with respect to what he termed, I think, 8 the stability of the estimates over the years? 9 A Yes, I do remember reading that. 10 Q And do you have an opinion as to whether you disagree 11 or agree with his criticisms there? 12 A Well, I certainly think that they are relatively 13 stable over the years and I think that we have to -- I think 14 we have to do -- probably do a little bit of work, a little 15 bit of statistical work in the courtroom, to show that. 16 Q So you disagree with Dr. Raudenbush's conclusions? 17 A I do disagree that there is a -- that there is any 18 kind of major problem with stability over the years, yes, 19 I disagree completely. 20 Q And could you explain the bases, and using the board 21 if the Court will permit? 22 THE COURT: Sure. 23 A Well, a great deal was made of a comparison -- a great 24 deal was made of a comparison between African American 25 relative odds in 1997 and African American relative odds in 37 1 the year 2000, and I think that a great deal was made of 2 that comparison, and I think we need to talk about African 3 American relative odds. I have to say, if you select two 4 years out then you might get a different comparison, so I 5 want to look at all of them across all six years, okay, and 6 so I want -- can I go to the board? 7 Q Sure. Have you prepared some notes? 8 A I have a little note that summarizes things that I had 9 from before. 10 THE COURT: Can I make a suggestion? 11 THE WITNESS: Yes, sure. 12 THE COURT: I didn't know where you were, I thought 13 you were sitting. 14 If we put the board over here just for purposes 15 of this, then everybody can still have a seat and sit and 16 see it. 17 MR. KOLBO: In fact, I think we're done with the 18 projector. 19 MS. MASSIE: Thanks, Judge. 20 (Discussion off the record at 9:15 a.m.) 21 THE WITNESS: What I want to do is, I want to look 22 at all the years, so 1995, 1996, 1997, 1998, 1999 and 2000. 23 And I want to talk about relative odds and I want to talk 24 about comparing relative odds, and I said I have to -- I 25 have to do a little bit of statistics and a little bit of 38 1 math here, so that's just because it's been raised and I 2 want to make sure that we all understand. 3 So the relative odds, and I'll just call it the 4 estimated relative odds that we calculated, the estimated 5 relative odds from the report for 1995 was 257.93. That's 6 1995. This is, we're looking at the relative odds for 7 preference for African Americans, and that's always to a 8 baseline of caucasian. 9 For 1996, it was 313.59. For 1997 it was 53.49. 10 And for 1998 it was 132.16. For 1999, it was 206.45. And 11 for 19 -- whatever it is -- 2000, it was 443.26. 12 So what was talked about, what was talked about was 13 the comparison of this value, the 53, to the 443. Now, I 14 presume, I presume those were chosen because, well, if you 15 look down the list, what's the smallest one? It's 53. And 16 what's the largest one? It's 443. Would there be -- would 17 there always be a largest and a smallest? Well, that's 18 mathematics. There will be a smallest and a largest. And 19 if you're doing comparisons, well, depending, you would 20 typically look at the -- might look at the extreme, that's 21 fine, okay. 22 Now, I also had a number of standard deviations, a 23 Z score, and I need to look at that, and I'll talk about 24 that in a second, but I just want to record the one. These 25 are straight out of the reports and we saw these before, and 39 1 the standard deviation, and I'll need these, so 14.40, 2 13.18, 13.96, 13.46, 12.64, and 12.53 -- or 51, excuse me, 3 51. So this is how this is reported. These were reported 4 in the first report that I did, these first four, and these 5 were in two supplemental reports. And these are the 6 standard deviations which measure the level of statistical 7 significance of how far these are away from one, okay. So 8 this is all summary information. 9 Now, how do we actually do the calculations? Where 10 do they come from? They come from logistic regression. If 11 you recall, that's the technique we used. And logistic 12 regression works in terms, like every regression, works 13 in terms of regression coefficients. 14 And what are the regression coefficients that are 15 behind these numbers, what are the regression coefficients? 16 Well, in fact, these are not the regression coefficients 17 themselves, there is -- in the logistic regression there are 18 coefficients that we can use to calculate these. In fact, 19 the coefficients, the coefficients themselves turn out to 20 be, to be careful, it's the natural logarithm of these 21 values, the natural logarithm, not the log to the base ten, 22 but the natural logarithm. On your calculators, that's the 23 LN function, right? 24 So in fact, the regression coefficients, the 25 coefficients, the coefficients themselves, the coefficients 40 1 themselves are what you use for doing all the inferences. 2 And so what I'm going to do is write down what the 3 coefficients are, and that's what you use for comparison, 4 are the coefficients, because that's the basic underlying 5 calculation mechanism, are these coefficients. 6 And so the coefficients are, and I'll put them in 7 order and say them, the coefficients are, and I'm going 8 to round them to two places, the coefficients are -- for 9 corresponding to 57.93, the natural logarithm is 5.55. For 10 313.59, the natural logarithm is 5.75. For 53.49, the 11 natural logarithm is 3.98. And for 132.16, the natural 12 logarithm is 4.88. And for 206.45 the natural logarithm 13 is 5.33. And for 443.26, the natural logarithm is 6.09. 14 So these are the actual coefficients in the computer 15 output that you would find in the logistic regression 16 computer output. 17 Now, a relative odds of one means no preference. If 18 you take the logarithm of one, you get zero, that's just the 19 way it works, and so these need to be compared to zero. And 20 in fact, the way these standard deviations are calculated 21 is they take the coefficient, this is out of this, and 22 divide it by something called the standard error of the 23 coefficient, and you get the number of standard deviations. 24 Now, I think it's informative to plot these values, 25 because this is the scale for comparison, the comparisons in 41 1 the regression coefficients, so I need to plot these values, 2 and I'll do the best I can, okay. Zero is here, and then we 3 have got one, two, three, four, five, six, and you can see 4 if I did a reasonable job of making those equally spaced. I 5 tried, okay. 6 And then what I would do is plot values, plot the 7 values here, so I better give myself a code, one, two, 8 three, four, five. So 5.55 is here. And 5.75 is here. 9 And 3.98 is here. And 4.88 is here. And 5.33 is here. 10 And 6.09 is here, okay. 11 So this, these are the six years. This is '97 and 12 this is 2000, these are the extremes, but this is where the 13 coefficients fall in terms of the regression coefficients 14 and this is the scale for comparison. 15 Now, statistically what you're doing is comparing 16 each of these values to zero. You're comparing each of 17 these to zero, because that's in the regression coefficient 18 scale and clearly these are clustered relatively far away 19 from zero. 20 In fact, in fact, the standard deviation tells us 21 that for -- well, this standard deviation number tells us 22 how many standard deviations we are away from zero and so, 23 for instance, the year 2000 one is 12.5 standards deviations 24 away relatively. Each one of these has a standard error 25 comparative measurement attached to it, and so we're fairly 42 1 always far away. So when I say that these results are 2 statistically relatively stable, I'm making the statement 3 of how they are in the regression coefficient scale. So 4 that's how to illustrate and say that the variation from 5 year to year is within reasonable statistical standards. 6 Now, there are a couple of more things we could do 7 with this, and I'm not sure how far we should go, but one 8 more thing we could do is we could, for instance, provide 9 confidence limits for these relative odds if we wanted to. 10 I was asked about, could you provide confidence limits, and 11 I think I was criticized for not providing confidence 12 limits, I'm not quite sure. It's not hard to do. 13 So I could provide confidence limits, and if I do 14 one of them, if I do one of them and write down the results 15 of the -- for the others, we can see how that goes and I'm 16 going to give -- so what I need to do to provide confidence 17 limits for the relative odds is I need to determine the 18 standard error of the coefficient, because the way I would 19 provide confidence intervals is, I have to provide a 20 confidence interval for the coefficient and then I have to 21 retranslate it into relative odds. 22 Remember, the coefficient is the logarithm, so 23 what I have to do is provide confidence intervals for the 24 coefficient and then retranslate it, okay. 25 So to get standard errors, I actually -- the 43 1 standard deviation is the coefficient divided by the 2 standard error. It turns out if we take the coefficient 3 and divide it by the standard deviation, we have to get the 4 standard error. So the way the standard can be calculated 5 is we will take this coefficient value and divide by the 6 standard deviation, so 5.55 divided by 14.40, and that will 7 actually -- this is, we're doing this backwards in some 8 sense, because this is actually in the computer output, but 9 just to show where it comes from, this is the standard error 10 of the coefficient, 0.39. That's this value divided by that 11 (indicating). And we can do the same for all of these. So 12 we get 0.44 for 1996, 0.29 for 1997, 0.36 for 1998, 13 0.42 for 1999, and 0.49 for the year 2000. 14 And from this, from those standard errors, we 15 can calculate confidence intervals. The usual way of 16 calculating confidence intervals, I'm going to do it 17 approximately, because it won't make any difference, is to 18 get 95 percent confidence bounds, we take the confidence 19 coefficient, the regression coefficient, and take a plus or 20 minus two standard errors and that gives us 95 percent 21 confidence amounts. 22 So I could do that for this, and so, for instance, 23 in 95 percent confidence, 95 percent confidence interval, 24 then, for the regression coefficient, take 5.55 and subtract 25 two times this, and the lower bound, then, is 4.77 and the 44 1 upper bound is 6.33. So that's a confidence interval for 2 the true value of the coefficient. 3 And I can do that for each of these, and I'm not 4 going to write them all down right now unless you would like 5 me to, but what I can do now is that I can translate these 6 into confidence intervals for the relative odds. And 7 what I do is I use the, let me see, inverse logarithm, 8 exponentiation, so I undo, to get into terms of relative 9 odds, and in terms of relative odds, the confidence interval 10 runs from 118, that's the lower bound, to 561. 11 I think I had indicated earlier that these numbers, 12 I don't believe these numbers exactly, and maybe, you know, 13 in some sense I should never put two decimal points down on 14 something I don't believe too accurately, but we can see 15 that this is a confidence interval for the relative odds. 16 These are all high numbers. You know, I think they are all 17 high values, but these are confidence intervals for the 18 relative odds. 19 So if I might, I think I'll just -- could I write 20 down -- I'll write down the confidence interval for relative 21 odds for each of these, without going through all the other 22 calculations. 23 So the confidence interval for 1996 then becomes 24 130 to 757. Notice they are wide, and they are actually, 25 because of the way it works, they have to be wider on the 45 1 high end, because the relative odds, as they go up, there 2 isn't that much -- as much difference as they go higher and 3 higher. 4 For the 53.49, lower bound is 30, again, and the 5 upper bound is 96, so there is a spread, but these lower 6 bounds are all what I consider big relative odds. 7 THE COURT: Even 30? 8 THE WITNESS: Oh, sure. Oh, sure. Ten is big, 9 okay. I think I said that before. 10 THE COURT: You did. 11 THE WITNESS: Okay. 12 And 64 to 270, 89 to 478, and the last one is 13 166 to 1,176, okay. 14 So the lower bounds are all, you know, in what I 15 consider large relative odds. The upper bounds, of course, 16 are -- well, you know, they just become harder to estimate 17 out at higher relative odds. 18 So they are all -- so this gives you an idea of some 19 variation. So in my estimation, these are all indicating 20 the same thing, that in fact, there is a substantial, a very 21 large allowance for African American applicants for each of 22 the years, and they are consistent. 23 They are consistent, not technically statistically 24 consistent, but substantively consistent, but substantively 25 consistent from year to year. There may be a technical 46 1 difference between them. 2 So that's something I wanted to make sure we 3 demonstrate, the stability in the appropriate scale, which 4 is the regression coefficient scale, and the corresponding 5 confidence intervals that we could generate for relative 6 odds. 7 Q So in short, in a short sentence here, what you have 8 just explained is the basis for your opinion that you 9 disagree with Dr. Raudenbush's conclusion that the relative 10 odds ratios are unstable across years? 11 A Well, that's part of it, yes. That's certainly the 12 basis. This is my thinking of how I would look at this 13 and certainly I believe they are certainly substantively 14 stable, if not technically statistically stable across 15 years. 16 Q Now, I think Dr. Raudenbush criticized you for not 17 reporting confidence intervals as you have just illustrated 18 here for these years for African Americans. Could 19 Dr. Raudenbush have calculated confidence intervals for your 20 relative odds based on the data that he had available to 21 him? 22 A Oh, sure. This is not -- this is straightforward 23 calculations and the computer output was all provided. 24 Everything was there to do this, yes. 25 Q Now, you -- 47 1 A But you can do it from the reports, actually, as we 2 just did from that information. 3 Q And you have reported that all of these, as far as 4 you're concerned for these years for African Americans, all 5 of these show very large preferences? 6 A Oh, sure, yes. 7 Q Are you able to tell whether that's true or not for 8 the other races that you did odds ratios analysis on for all 9 these years? 10 A Well, if you computed confidence intervals for 11 relative odds, they would all be, you know, relatively -- or 12 distributed around the estimated values, but they would all 13 show large preferences, yes. 14 Q You know, Dr. Raudenbush testified, I believe, that he 15 believed that the standard deviation -- I probably won't say 16 this correctly so you will have to correct my question, 17 perhaps, and then answer it. I believe he testified that 18 there were 11 standard deviations in different separated 19 odds ratios between 1997 and 2000. Did you understand that 20 he testified along those lines? 21 A Yes, I remember being asked that in cross and I read 22 his testimony to that effect that the difference was about 23 11, I think he said, almost 11 standard deviations. 24 Q Do you disagree with that opinion? 25 A Oh, yes. That's wrong. Excuse me. 48 1 Q Do you have any understanding based on what you have 2 seen from Dr. Raudenbush's testimony and the data that you 3 have as to how he could have come to that conclusion? 4 A Well, if I might, can I do the calculation for the 5 Court, do the calculation in the regression coefficient in 6 the appropriate scale and then show you what the difference 7 is? And then I obviously don't know exactly how he did 8 the calculations, but I have an idea of how he did the 9 calculations, okay? 10 THE COURT: Certainly. Why don't we -- just for the 11 record, why don't we just put some identification on that. 12 MR. KOLBO: I was going to ask if we could mark it, 13 Your Honor, and offer it, as well. 14 THE COURT: Why don't you at least mark it and then 15 we can talk about offering it. 16 MR. KOLBO: 226. 17 THE COURT: What's the next number? 18 MR. KOLBO: Do you want to write 226? 19 THE COURT: Ex. 226, just so we can talk about it, 20 then you can move it later. Just so we have something for 21 the record that we know where it is, okay. 22 THE WITNESS: I need to do the comparison. I have 23 to do the -- compare standard errors. 24 MR. DELERY: We have another easel with another pen. 25 THE WITNESS: That's okay, let me record them, then 49 1 we will do it. 2 1997, the coefficient was what, 3.98, and the 3 standard error was .29. That's okay. We're fine. And it 4 was 6.09, is that right, and the standard error was 0.4, 5 okay. That's all we need. 6 THE COURT: And then if you'd just put on the 7 right-hand bottom corner again, 227, Ex. 227, and the record 8 will reflect that he is now working on 227, proposed. Okay. 9 THE WITNESS: Well, from these values we can see how 10 different they are. And I am going to bring this back just 11 to show you, then I'll move it back over. 12 You can see this value here is what, 12 standard 13 deviations from zero, right, 12 standard deviations from 14 zero, and this one, this one here is what, 14 standards 15 deviations from zero, and what Dr. Raudenbush has testified 16 to is that if we compare these two, we get something close 17 to 11. 18 Well, I mean, statistically, and I think I hope 19 I reacted appropriately when someone asked me, would it 20 surprise you that the difference between these two is 11, 21 given this one all the way down here is twelve and this one 22 all the way down here is 14, would it surprise you that 23 this difference is 11, and the answer is, it would totally 24 surprise me, and the reason is that we're carrying this to a 25 fixed value of zero, and these are compared to each other, 50 1 and each of them has a standard error attached to it, so it 2 doesn't -- it's not reasonable that this difference should 3 be 11. 4 So I went back and did the calculation. And I'm 5 going to show you how to do the calculation now and show 6 you what the number is, okay. What I need is I'm going to 7 compare these two coefficients, so I'm going to compare 8 6.09, the difference between 6.09 and 3.98. That's what I 9 need to compare. 10 In order to compare this difference, I have to get 11 what's called the standard error for the difference, and 12 this is something that we teach in our first course in 13 statistics. And if we have got two standard errors and we 14 combine them to the standard error for the difference, 15 assuming they are computed from different data sets -- which 16 they are, I didn't put anything together, '97 was done 17 completely separate from 2000 -- so the standard error of 18 the difference, actually the way we do it in statistics, we 19 always have to do things in the variance scale, so we have 20 to square these things. So we put them together, 0.29, we 21 square it, and 0.49, we square it, and then to get the 22 standard error we have to go back and we have to take the 23 square root of that. So that's the math. We square these 24 two numbers, add them up, and take the difference. 25 Now, I should have a calculator. 51 1 MR. DELERY: It's .31. 2 THE WITNESS: You got .31? Let's see. 3 THE COURT: He did it in his head, see. 4 THE WITNESS: But he is not testifying. 5 THE COURT: That's okay. For your profession, the 6 calculator is the most important tool. For lawyers, it's 7 their business card. 8 THE WITNESS: Okay. Let's just make sure I do it 9 right. So I have to make sure I do it right, and the value 10 I get, this one is .49, it can't be any smaller than that, 11 it's got to be bigger than that, bigger than the total, each 12 one separately, and it turns out to be 0.57, 0.57, okay. 13 And so we divide this by 0.57. That's the standard 14 error of the difference. And this tells us how many 15 standard deviations these two numbers are apart, and that 16 number turns out to be 3.7, 3.7. 17 Now, that's in the range of big. Two or three is 18 big. This is a big number, but it's not 11. Am I concerned 19 that this number is this big? Well, in fact, I think there 20 probably are some year to year differences, and I think I 21 testified to that before. There probably are. We don't 22 expect these coefficients to be the same year to year. 23 There is variation. There might be differences in the way 24 things are done. 25 Is this a really large number? In the context of 52 1 looking, what, at the most extreme case, right? 1997 versus 2 2000, those are the most extreme cases. Is this a large 3 number? The answer is, it's not large when you consider 4 that it's the most extreme and the fact that we don't expect 5 it to be exactly the same year to year. So this is a number 6 that I think is consistent with the kind of variation we 7 saw on the previous plot, that these are not 11 apart or 8 almost 11. 9 THE COURT: And 11 would be a large number, you 10 would agree with that? 11 THE WITNESS: I think I agreed in my cross 12 examination that I thought 11 was -- I was surprised. 13 I couldn't believe it was 11, okay, but I didn't dare do the 14 calculations right there on the spot. So 3.7, that's given 15 where we are, that's the value. Well, it is the value, so 16 it's reflected in the plot. 17 Q Dr. Larntz, do you have any -- based on what you 18 understand from the data, is there any way you could 19 understand how someone could conclude that the standard 20 deviation was 11? 21 A Well, I don't know exactly what Dr. Raudenbush did, 22 but if, if you didn't go into the regression coefficient 23 scale, if you tried to work in the relative odds scale, if 24 you didn't go to the basic logistic regression coefficient 25 scale and tried to do these directly and used this standard 53 1 deviation, but didn't take this, that we had to work in 2 coefficient scale, if you did the math, the same math that 3 we did over here, but you just took these numbers and 4 assumed that you could take the standard error of this one 5 by dividing by that, if you did that, if you did that, 6 which is -- 7 THE COURT: That's not an acceptable statistical 8 procedure? 9 THE WITNESS: Well, it's not right. It's not right. 10 It's not right, because these have more variations. We saw 11 that from the confidences. These have more variations than 12 that. I mean, the standard deviation associated with this 13 isn't about ten, it's much more variation than that, or 11 14 or 15 or whatever. 15 So, but if you did that, if you did the math that 16 way, and did the same, to my understanding, did the same 17 thing, you would get 10.94 if you did it that way. So I 18 don't know what was done, but I know if you did that, you 19 would get 10.94. 20 THE COURT: And you know if that was the way it was 21 done, it's not done directly, in your opinion? 22 THE WITNESS: Oh, absolutely, that's right. 23 BY MR. KOLBO: 24 Q Is that something statisticians could reasonably 25 disagree with, whether it's appropriate? 54 1 A There is no disagreement about that at all. I mean, 2 the logistic regression coefficient scale is the scale for 3 comparison, not the -- and the relative odds are derived 4 from those directly for purposes of understanding the model. 5 Q I just have a few more questions, Dr. Larntz. 6 I think Dr. Raudenbush testified -- I understood him 7 to testify that your analysis really demonstrates only that 8 a race factor was used as a factor in the admissions 9 process. It doesn't demonstrate anything about the extent 10 to which race was used in the admissions process. Do you 11 agree or disagree with that? 12 A Well, I would disagree, and shortly -- can I just 13 briefly describe? 14 I disagree in the sense that these are big 15 representative odds. There is a tremendously large 16 allowance given to race, and it's hard to believe that any 17 other factor could explain that away, unless it were just a 18 surrogate for race. 19 MR. KOLBO: I have nothing further, Your Honor. 20 I would like to offer the exhibits, Your Honor. 21 THE COURT: Any objection? 22 MR. DELERY: No objection, Your Honor. 23 THE COURT: Fair enough. 24 MR. KOLBO: It's 225, 226 and 227. 225 is the first 25 handout. 55 1 MR. DELERY: Oh, the charts, okay. All right. 2 THE COURT: You know, if you remind me Monday, we 3 may be able to see if Court Services -- actually, when we 4 have those and admit them, they have a Polaroid, I think 5 they still have it, and we can take some Polaroids of that 6 so that everybody will -- I don't know how else to get it to 7 a form so everybody will be able to have it, so if somebody 8 reminds me Monday, I'll try to see if they have a Polaroid 9 still up there and take some pictures. If not, I'm not sure 10 how we will do it. 11 MR. PURDY: Just a suggestion, if the Court would 12 prefer, we're happy to have these typed up. We would be 13 happy to do that and then just -- 14 THE COURT: That's even better. Nobody objects? We 15 will keep these. You can all sign off on them. 16 MR. PURDY: We won't even take them out of the 17 courtroom. We will just copy them and present them. 18 MR. DELERY: Or mark Dr. Larntz' notes if they are 19 the same thing. 20 THE COURT: I think we should use what he used and 21 preserve those for the official record and type them up and 22 if for some reason you can't do that, or if you can't do it, 23 we will take the numbers, we will get them down here to take 24 pictures. 25 In Judge Taylor's courtroom, they have the white 56 1 board now that you write on and then you push a button and 2 the thing comes out. 3 CROSS EXAMINATION 4 BY MR. DELERY: 5 Q Good morning, Dr. Larntz. 6 A Good morning. 7 Q Welcome back. 8 Like Mr. Kolbo, I think I'm going to -- certainly 9 hope that I stick to what you have covered here this 10 morning. We're not going to go back over the ground that we 11 covered last month when you were here the first time, and I 12 think I'll start where we left off and then work back from 13 there. 14 MR. DELERY: If I may approach, Your Honor? 15 THE COURT: Absolutely. 16 BY MR. DELERY: 17 Q The first two columns here in Exhibit 226 were taken 18 from your expert reports; is that correct? 19 A That's correct. 20 Q The first relative odds numbers came from model one 21 for each year; is that right? 22 A Yes, I think that's right. That's the first set of 23 analyses, that's true. 24 Q You reported three composite odds ratios for each 25 year; correct? 57 1 A That's true. That's true. I was responding to the 2 criticism of this particular comparison. 3 Q Right. I just want to get it clear what these numbers 4 are. These were the model one numbers from each year? 5 A Sure. 6 Q Okay. And then the second column here that you have 7 headed SD are what your report called the standard 8 deviations on the same tables from model one? 9 A That's right. 10 Q And the standard deviations were reported in your 11 report on the log scale; correct? I mean, these are 12 standard deviations in the log scale? 13 A They are standard deviations for the regression 14 coefficient, that's right. 15 Q And the regression coefficients are in the log scale? 16 A Well, the logistic, l-o-g, regression, does analysis 17 on the log odds, so that's correct. 18 Q And the odds ratios, the estimated relative odds in 19 the first column, are not on the log scale? 20 A Oh, that's true, yes. 21 Q And is there a name for not on the log scale, is there 22 a term for that? 23 A You mean on just the normal scale? 24 Q Is that what it's called? 25 A The real scale? I mean, relative odds scale, the 58 1 relative odds scale. It's the relative odds scale. 2 Q It's the normal scale; is that -- 3 A I would call it the relative odds scale. 4 Q Okay. 5 A As opposed to the log odds scale. 6 Q And -- okay. The scales are not -- if you line them 7 up, they work differently; correct? 8 A I mean, there is a direct translation one to another, 9 but obviously the relative odds are it's a trans -- they are 10 a direct translation of each other. 11 Q So just so I'm clear, the two numbers reported in your 12 reports for each of these model results were on different 13 scales? 14 A Well, I guess I wouldn't consider it that way. I 15 would consider that, in fact, I gave the relative odds, 16 that's an appropriate way to summarize the value, and this 17 is the significance to understand the relative odds. So 18 I think the standard deviations are directly related to 19 the significance of relative odds. 20 Q The standard deviations are not the standard 21 deviations of the relative odds; is that right? 22 A Oh, no, no. They are an indication of the statistical 23 significance of the relative odds. 24 Q But they are actually the standard deviations for the 25 regression coefficient? 59 1 A For the regression coefficient, that's exactly how you 2 have to do that, yes. 3 Q Okay. And that's the way you reported it in your 4 report? 5 A The report, I reported with those two columns, that's 6 exactly right. 7 Q And then it was the calculations that you have done 8 here that led to the third column for the regression 9 coefficient; is that right? I guess that's actually the 10 fourth column on this page. 11 A The numbers there, I re -- I back calculated, but they 12 are the numbers from the -- if you looked at the computer 13 output for logistic regression, that those are the numbers 14 that you would see in the computer output, yes. 15 Q And so the relative odds is just the -- well, I'll go 16 the other way. To get the regression coefficient, you just 17 take the log of the relative odds? 18 A Of course, the way you get the relative odds is you 19 exponentiate the regression coefficient, that's how, but the 20 regression coefficient is the basic number that you work 21 from and then you exponentiate that to get it in relative 22 odds. That's a standard output in logistic regression, is 23 to report both of those. 24 Q So you can move back -- what you are saying is you can 25 move back and forth between the log scale and the relative 60 1 odds scale? 2 A Well, they represent the same thing. 3 Q Turning now to Exhibit 227, the standard error of 4 the difference here between the odds ratio -- or the 5 coefficient, I'm sorry -- for 1997 and 2000, you calculate 6 as 3.7? 7 A That's right. 8 Q That's a fair statement of what's on this page? 9 A I mean, someone can check the math, but -- 10 Q Okay. Am I right that anything over two is considered 11 statistically significant? 12 A Without a selection bias in the sense of looking at 13 extremes, anything over two would have a five percent 14 statistical significance, but we're not doing just any two 15 here, we're doing the most extreme two, so we have selected 16 the two we're looking at out of a group, and I would say 17 that in that case you need a value, I don't know what the 18 value is, but it certainly would range, when you're 19 selecting out of, you know, the most extreme cases, that two 20 doesn't go anymore. You have got to take account of the 21 fact that you're selecting from the most extreme cases. 22 And so typically values of -- you know, you wouldn't be 23 surprised if you got values of three or four when they are 24 the most extreme. 25 Q Okay. Putting aside the most extreme context, two 61 1 ordinarily would be statistically significant? 2 A I mean, you want me to ignore the fact that -- where 3 the data came from? 4 Q For this purpose, I'm just asking you, if -- 5 A If I ignore where the data came from, then you have 6 about a five percent chance of getting a value outside the 7 ranges of two if the data comes from a normal distribution. 8 Q So greater than two ordinarily would be considered 9 statistically significant? 10 A In the context of something that wasn't generated as a 11 selection -- 12 Q All right. 13 A -- that would be true, in something that wasn't 14 generated as a selected. 15 Q And greater than three would ordinarily be considered 16 highly statistically significant, is that fair to say? 17 A In the same context, where you're not -- where you're 18 just looking straight away, not where you're looking at 19 extremes. 20 Q Okay. Let's go back, if we could, to Exhibit 226. 21 Your view is that all of the odds ratios that you have 22 reported here, both in the first column from your report 23 under the RO column, and then in the confidence interval 24 parentheses, are very large, is that your testimony? 25 A By the standard of statistical practice, they are very 62 1 large, absolutely. 2 Q So in the year 2000, for example, what your results 3 show is that the relative odds for African Americans as 4 opposed to whites could be anywhere from 166 to 1,176? 5 A The data are consistent with those values, yes, at 6 the 95 percent, at the 95 percent confidence level. 7 Q And so we can't really be sure where in that range it 8 falls, but you're fairly confident that it's somewhere in 9 that range? 10 A I mean, the interpretation of a confidence interval 11 now, you want to do that? Okay. 12 Q I mean, have I fairly -- 13 A Those are the values that are consistent with the 14 data at the 95 percent confidence interval, confidence 15 coefficient level, that's right. 16 Q Okay. And so we can do the same thing for the other 17 years, 1999, it's somewhere between 89 and 478? 18 A Well, knowing that those statements are made with 19 error rates that are five percent, that's right. 20 Q But the conclusion you draw is that all of these are 21 large, so these differences don't trouble you? 22 A Oh, that's right. They are all substantively 23 consistent, that's what I said, and that's exactly right. 24 Q And then when you plotted it here, you said that the 25 reason that these differences, the differences in the first 63 1 odds ratios that you reported don't trouble you is because 2 they are all quite far away from zero? 3 A They are all quite far away from zero and they cluster 4 together. 5 Q So again, here zero means a relative odds of one, am I 6 right? 7 A That's right. 8 Q Which means that the members of both groups have an 9 equal likelihood of being admitted? 10 A That's true. 11 Q So from these odds ratios you can be quite sure that 12 you have disproved a contention that both groups have the 13 same likelihood of being admitted as a percentage basis? 14 A You certainly can disprove that and you can disprove a 15 lot of other odds ratios, because the lower bounds are far 16 away from one, that's right. 17 Q So you think that you can be confident of more than 18 just that the so-called null hypothesis is disproved from 19 these? 20 A Oh, certainly, certainly, certainly, and from the 21 size of the standard deviations, the number of standard 22 deviations and the corresponding confidence, yes, certainly. 23 Q You believe, in fact, that you have quantified, 24 I believe that you said, the role that race plays in 25 admissions based on these numbers; is that right? 64 1 A Well, I am just a statistician, so I'll be careful of 2 what I did, okay? What I have done is I have described what 3 the admissions office did, okay? And so this quantifies the 4 admissions decisions, okay? And so what I have done is 5 described, quantified the admissions decisions, and this is 6 from their data and this is what their data tells me about 7 that. 8 Q And I want to be clear about that, because we 9 obviously had some fairly lengthy discussions of this issue 10 when you were here last month. 11 Today, Mr. Kolbo asked you some questions about 12 whether anything that Dr. Raudenbush said in his testimony 13 changed your opinions concerning the effect that race plays 14 in the admissions process. Do you remember those questions 15 from Mr. Kolbo? 16 A I'm not sure of the exact wording, but I took that to 17 mean if was there any change in my opinion from previously, 18 and I answered that there was no change in my opinion 19 previously. 20 Q Okay. So putting aside that, now I want to ask the 21 follow-up question. Do you believe that you have expressed 22 an opinion concerning the extent -- and I'm sorry, I guess I 23 may have misspoken a moment ago. I think Mr. Kolbo used the 24 word extent, not effect, if I misspoke, but do you believe 25 that you have expressed an opinion concerning the extent to 65 1 which race is taken into account in the admissions process? 2 A What I have done is expressed an opinion of the size 3 of the allowance that is shown in the data that's from the 4 admissions office for individuals that have the similar 5 credentials, the advantage that's given based on ethnic 6 groups in those for individuals with similar credentials, 7 and that's what I testified to, I hope, and that's really 8 what the conclusion is, is that I have quantified 9 statistically the size of the allowance that's given 10 for individuals with similar credentials. 11 Q And just so we're clear, by size of the allowance, do 12 you mean by an admissions officer sitting down to read a 13 file in making the decision? 14 A I mean describing the admissions decisions, what show 15 up in their -- from their data. 16 Q What shows up as a result of the decisions? 17 A As a result of the decisions, absolutely. It's the 18 results of the decision. I don't think I have ever said 19 or didn't mean to say that I did anything other than to 20 describe. It's their decision. I'm describing what the 21 results are. 22 Q Right. But again, just so we're clear, that's 23 different from describing how the decisions were made; 24 correct? 25 A These are the results of the decisions. 66 1 Q So is the answer to my question yes? 2 A Well, I don't know exactly how decisions are made. 3 Q Okay. And these data don't say anything about how 4 decisions are made? 5 A They say a good deal about the results of those 6 discussions and that's what I would say. 7 Q But not how the decisions were made? 8 A The mechanism for making decisions, that's not 9 statistics. 10 Q The process? 11 A That's not statistics. 12 Q I believe that you said earlier in asking -- answering 13 the questions for Mr. Kolbo that you're confident in the 14 substantive stability of your results, but that the odds 15 ratios were not technically statistically stable across the 16 years, did I get that right? 17 A I think that what I would say is that the substantive 18 results are clearly consistent, okay, clearly consistent. 19 There may be, and I think there probably is, I would expect 20 there to be some technical, in the sense of statistical, 21 variation from year to year. I wouldn't expect the 22 variation from year to year to year, so what I was 23 responding to is I think that the variation from year to 24 year is probably not zero. There is probably some -- in 25 the underlying true scale, that there probably is some year 67 1 to year variation, but it's not large enough to change the 2 substantive conclusions, that's right. 3 Q So the differences among the odds ratios across years 4 don't lead you to alter the basic conclusions that you have 5 reached, that's the bottom line of what you're saying? 6 A Substantively they are the same from year to year, 7 that's what I said. 8 Q I think maybe now I would like to go, if we could, 9 back to the last page of the packet that you have. It's 10 Exhibit 225, I believe, if you could look at that. 11 A I have it. 12 Q The last page was the chart of the relative odds 13 effect on baseline probabilities. 14 A That's correct. 15 Q Am I right to understand that this is basically a way 16 to translate odds to probabilities, is that what this is 17 doing? 18 A It's showing the effect of a relative odds value on 19 certain -- on a range. I tried to do the whole range of 20 probabilities. I think both sides offered their own 21 baseline probabilities. I mean, we both did. And so I 22 thought we should probably, just for clarification, give the 23 whole range of probabilities. 24 Q Okay. So if we look across the top, you have various 25 relative odds, 5, 10, 20, 50, 100, 200? 68 1 A Sure. 2 Q Do you see that? 3 If all we know is an odds ratio number like 100, 4 if that's all we know, we don't know anything about the 5 relative chances of admission of two groups, is that fair 6 to say? 7 A We don't know the probabilities, that's right. 8 Q Okay, because on your chart here, an odds ratio of 9 100 could translate to any of the probabilities that you 10 have listed under that heading as compared to the baselines; 11 right? 12 A Well, it depends on the baseline, yeah, sure. 13 Q But the point is that unless you know, unless you 14 know the underlying probabilities, you can't evaluate the 15 significance of an odds ratio like 100? 16 A Well, it would mean different things for different 17 baseline probabilities, that's what I'm trying to 18 illustrate, exactly. 19 Q Now, I believe you testified when you were here before 20 that an odds ratio of two or three, you would consider 21 large; is that right? 22 A In my work, two or three is a big number in odds ratio 23 terms, yes, absolutely. And I saw one of eight the other 24 day, so everyone was astonished in the room. 25 Q So a select group, I guess. 69 1 A Well, they were clinical medical researchers and they 2 were astonished that it was as big as that. Eight was big. 3 Q And that's -- if you think about it in terms of a 4 medical study for a second, you don't have two or three here 5 on the last page of Exhibit 225, but you do have five. 6 A Oh, yeah, sure, sure. 7 Q Which, if two or three is large, then five is clearly 8 large? 9 A Five is a considerable effect. 10 Q You have, if we look here, a baseline probabilities 11 of .1 or 10 percent for a relative odds of five, the other 12 group's probability would be .35, .36, if you round, is that 13 what this table shows? 14 A I'm not with you. I'm sorry. 15 Q Okay. If we look under relative odds of five, so the 16 first column. 17 A I have that. 18 Q Starting with the baseline probability of .1. 19 A Oh, .1, I'm sorry. I misunderstood. 20 Q Yes, .1 for 10 percent? 21 A So 10 percent, 10 percent translates into -- if the 22 relative odds is five to 35.7 percent, that's right. 23 Q And so that's the kind of effect that you're 24 considering to be really large based on your experience, 25 for example, in medical studies? 70 1 A Well, it is a large effect, yes. 2 Q Okay. And is it fair to say that it's your opinion 3 that in the context of this case, the differences in all of 4 the odds ratios that you have reported are not significant, 5 because they are all so far beyond this two, three, five 6 level that you find really large? 7 A I think that the odds ratios here are large. I think 8 some are larger than others. I don't say that there are no 9 differences between them statistically or substantively. 10 I think that they are larger for some ethnic groups than 11 others, but they are -- but for the selected minority 12 groups, they all looked large, yes. 13 Q Is it fair to say that in order to interpret the 14 significance of a reported relative odds number, you need 15 to look at the underlying probabilities? 16 A Oh, absolutely. And we did that. We looked at the 17 grids and these are just -- the relative odds summaries are 18 just summaries of what goes on in the grids, and the grids 19 give us those baseline probabilities, absolutely, 20 absolutely. I absolutely agree, you need to look at the 21 basic data in the grids, yes. 22 Q Would you also say that it's important to look at 23 the sample size or the pool size of the groups that are 24 contributing to the relative odds calculation in order to 25 understand the significance of a reported relative odds 71 1 ratio? 2 A That's actually taken into account in the statistical 3 significance calculation, so different sample sizes enter 4 into that. The amount of information about a particular 5 ethnic group is accounted for in that, but absolutely, if 6 there are very small numbers, very small numbers in the 7 aggregate, then those would be -- well, correspondingly 8 the confidence bounds for those would be wider. 9 Q So it's fair to say that to really interpret and 10 report estimated relative odds, you need to look at the 11 odds ratio number, the underlying probabilities and the 12 underlying pool sizes, is that fair to say? 13 A Well, I think a good bit of that information is 14 accounted for in the statistical significance, but that's 15 right, that is certainly -- you can certainly say that you 16 could look at those and that would be informative, yes, 17 absolutely. 18 Q Okay. Is it fair to say that if you didn't look at 19 any one of those, in your view, you would be missing an 20 important piece of evaluative information? 21 A I mean, those are things you would look at. Whether 22 you needed to look at each one of them for each particular 23 context, it would depend on the context. 24 Q Okay. I think I understand from what you have just 25 said, in other words, that you might draw different 72 1 conclusions about a reported relative odds number if you 2 were talking about three members of a group as opposed to 3 30 or 300, is that fair to say? 4 A Well, if the data came from three, I'm sure we 5 wouldn't have gotten any statistical significance, and so 6 the size of the standard deviations accounts for the sample 7 size to a great extent. 8 Q But for the same odds ratio like 100 that we were 9 talking about before, you would draw different conclusions 10 from it if it were based on three members of a group as 11 opposed to 30 or 300, is that fair to say? 12 A Well, we would surely have different measures of 13 statistical significance from those cases. 14 Q And therefore, from that, you would -- you would reach 15 different conclusions based on that number? 16 A I would reach different conclusions in the sense that 17 for the smaller group, the confidence intervals would be 18 much wider and so I would be less sure of that, yes. 19 Q Okay. If you could, actually, I would like for you to 20 look at one of your earlier reports. It's Exhibit 138. 21 And if you could look -- do you have it now, 22 Dr. Larntz? 23 A It's the February 21 supplemental report. 24 Q Yes. 25 A Yes. 73 1 Q And if you look, there is a series of 24 pages of what 2 we have called cell by cell odds ratio calculations there? 3 A Yes. 4 Q Do you see that? 5 If you would look at page six of 24 there -- 6 MR. DELERY: This is again Exhibit 138, Your Honor. 7 THE COURT: Page 24? 8 MR. DELERY: The last set of the report, the back 9 of the report, is a series of pages, 24 pages numbered 10 page one of 24 and so forth, and I would like to look at 11 page six of 24. 12 THE COURT: Okay. 13 BY MR. DELERY. 14 Q And particularly the second block there of lines for 15 LSAT ranges from 164 to 166, I think it is. Do you see 16 that? 17 A Yes, I see that. 18 Q It's the second paragraph, so to speak, on this page. 19 The first cell there for 3.75 GPA and up has a 20 minority -- three minority applicants and three were 21 admitted, and 131 majority applicants and 56 admitted; is 22 that correct? 23 A That's correct. 24 Q And you have calculated infinite odds ratio for that? 25 A The computer did the calculation, yes. 74 1 Q Right. And then for the next cell down you have again 2 three minority applicants, only two of whom were admitted, 3 and then 216 majority applicants, 73 of whom were admitted? 4 A Yes, I see that. 5 Q And because two out of three minority students were 6 admitted as opposed to all three out of three, the odds 7 ratio goes from infinity down to 3.92; is that right? 8 A That's correct. 9 Q And that's due to the fact that there is such a 10 small sample size of minority students; isn't that right? 11 A Yes. If we're looking at each individual cell, and 12 these cell by cell comparisons, you recall, were not the 13 primary, now, these are supplemental analyses, and I didn't 14 naturally expect to see statistical significance in these, 15 because there would be small sample sizes, and in fact, for 16 each of these two, the P values are bigger than .05, and so 17 in and of themselves, which is -- again, they were never 18 meant to be stand-alone, but in and of themselves, neither 19 one shows statistical significance, that's right. 20 Q But it is fair to say that many of the cells that you 21 have highlighted on the charts that you showed us earlier 22 had very small numbers of minority applicants in them; isn't 23 that right? 24 A But the analysis doesn't do it cell by cell, it 25 generates the information across all the comparative cells 75 1 or comparative information, so that's a very different story 2 than looking at an individual cell and saying there are few 3 numbers there. Your composite estimate combines across all 4 the cells, all the cells with comparative information. 5 So I think we have to make sure you're talking about 6 the cell by cell analysis. This was done specifically to 7 look and see if there was, you know, an effect of odds ratio 8 across these cells, and this is -- this is for these 9 particular cells. They are small, and I don't disagree 10 with that. 11 Q All right. And I actually want to move, use this 12 as sort of a transition into the charts that you put up 13 earlier and how you calculated the composite odds ratios. 14 Actually, maybe we should -- if we could, can we put 15 one of those back up? 16 While we're waiting, I think I can ask you some 17 preliminary questions. 18 You highlighted the cells for whom applicants were 19 included in the composite odds ratio calculation; is that 20 right? 21 A What I did, yeah, is I highlighted the cells that 22 provided comparative information, that's true. 23 Q And the comparative information that they provided was 24 on a cell by cell basis; right? 25 A The comparative information is across the cell by cell 76 1 for -- it's across all the ethnic groups, that's right. 2 Q Across all the ethnic groups within each individual 3 cell; correct? 4 A Well, I mean, I talked earlier about the fact that 5 we're looking at differences in a cell by cell fashion as 6 opposed to modeling it overall, yes. 7 Q Right. So you -- 8 A So the composite -- I'm sorry, if I might -- the 9 composite puts together the estimates cell by cell, which 10 may be quite variable, and puts them into a composite that 11 accounts for all the information, all the comparative 12 information. 13 Q But it uses as its basic building block, if I 14 understand it correctly, the cell that's defined on the 15 grid; correct? 16 A Well, the basic building block is to look at people 17 with similar credentials, and what we did is looked at 18 individuals with similar credentials, similar grade point 19 averages, similar LSAT as defined by the law school, and 20 their grid, the grid -- remember, the grid boundaries came 21 from the law school, so they provided these grid boundaries, 22 and we looked at the comparative information in looking, 23 combining it across these individual cells, that's right. 24 Q And in fact, you created a variable for each cell that 25 contributed comparative information; isn't that right? 77 1 A The computer actually created cells -- created one for 2 every cell and then evaluated whether it was comparative. I 3 mean, every cell was included in the analysis and I didn't 4 go in and have it exclude cells, no. 5 Q But I'll come back to the computer here in a minute, 6 but to follow on this point, the cell itself contributed a 7 variable in your analysis; in other words -- 8 A What we did is we looked at each cell to see if 9 that -- the computer looked at each cell and the computer 10 determined how much it would contribute and added in the 11 amount of comparative information, depending on what was 12 in the cell, that's always true. That's true of any 13 statistical analysis. 14 Q Okay. You mentioned the computer doing a lot of 15 things here this morning, but the computer only did what 16 you told it to do; isn't that right? 17 A I think that would be -- that would be in the ideal 18 world, yes. 19 Q Do you have any examples of where the computer 20 overrode your instructions and did something on its own? 21 THE COURT: In this case? 22 BY MR. DELERY: 23 Q Putting aside your microwave? 24 A Computers are only as good as the programming that 25 goes into the computer, that's always true. 78 1 Are there errors in the programs that I used? I 2 used programs that I think are reliable. I checked them 3 out. I feel quite confident that it was doing what I wanted 4 it to do. It did a lot more than I wanted it to do, because 5 that's what computer programs do, they provide you with -- 6 we used to have a term, a side inch, in the old green paper, 7 we used to call them how many side inches of output did you 8 get, you know, and how many side inches. We don't do that 9 anymore, we save it all on disk, but the number of side 10 inches was considerable in this case. 11 Q You're not trying to distance yourself, though, in 12 some sense, from what your computer did? 13 A Oh, no, not in any sense. No, I don't mean to do 14 that. If I give that impression, it wasn't -- I'm not -- I 15 am not, in any sense, saying that I didn't control what the 16 computer did. 17 Q You weren't helpless in the face of the computer that 18 was excluding this data; right? 19 A I enjoy it when the clerk says the computer lost your 20 reservation, yes, I enjoy that, in the sense that I know 21 it's a person that made a mistake that caused that to 22 happen. I'm very, very cognizant of the fact that computers 23 do what people instruct them to do, yes. 24 MR. DELERY: Okay. So if we could put up one of 25 the -- I guess the 1995 grid, which was the first page of 79 1 Exhibit 225, just so we're all looking, and if you could 2 slide it over so it's all there. 3 BY MR. DELERY: 4 Q Just so we have a sense of what is here, I believe you 5 said -- let me make sure I get this right. You highlighted 6 the cells that contributed comparative information; is that 7 right? 8 A That's what I attempted to do and the highlighting was 9 done by me. 10 Q Not the computer. The computer told you which ones it 11 had from -- 12 A No, no. I actually -- I actually went back and looked 13 at the cell grids and determined which ones would have 14 contributed, so I did it. 15 Q Okay. And then I think you said that about -- across 16 these years -- 84 to 88 percent of the total applicants are 17 in the shaded cells; is that right? 18 A I did a calculation and I think that's right, you 19 know, best that I could punch my numbers in. I actually -- 20 I actually counted across, and just to be clear, and totaled 21 the number, and I counted down and totaled the number, and 22 then I counted down and across and made sure that they came 23 out to be the same value. So I actually -- I actually did 24 check to see the best I could that -- those numbers that 25 are not highlighted, actually, is what I counted, and then 80 1 calculated the percentage of highlighted. 2 Q And just so I understand that correctly, the reason 3 that you excluded, for example, this cell, which is at 4 3.5 to 3.71 -- 74, hard to tell -- and then I think that's 5 148 to 150, so this cell here, 35 applicants, zero in its -- 6 A That's right. 7 Q The reason that that cell is not highlighted is 8 because, although there were members of different racial 9 groups, everybody was denied, and so in your definition of 10 comparative information, that doesn't contribute any? 11 A There would be no comparative information from 12 that cell. 13 Q From that cell, all right. And that's true even 14 though those -- that cell reflects actual decisions by 15 the admissions office? 16 A Right. Decisions that were the same for everyone in 17 that cell, that's right. 18 Q And in your view, I think you said, cells can -- even 19 cells that you shaded contributed different amounts of 20 comparative information? 21 A I think that's -- well, that's certainly true. I 22 mean, every cell contributes a different amount -- well, I 23 mean, I won't say every -- yeah, I think every cell probably 24 contributes a different amount. 25 Q So the way you set up your model, the closer the two 81 1 groups were to being treated the same, the less information 2 was contributed to the composite odds ratio; is that a fair 3 statement? 4 A That's not true, no. 5 Q So what do you -- 6 A I mean, that is not necessarily true. I mean, if in 7 fact they were treated the same across the cells, if the 8 admission rates were similar, then we would have gotten odds 9 ratios that would have used those, that information, and 10 calculated odds ratios appropriately. 11 Q Well, you said earlier that for cells where members of 12 majority groups on the one hand and minority groups on the 13 other hand were not treated exactly the same, but were 14 treated close to the same, the cells contributed less 15 information? 16 A Well, I think I better be careful, then, and make sure 17 we understand. In cases where virtually everyone is 18 admitted or virtually everyone is denied, particularly 19 ones where everyone is admitted, then that will contribute 20 relatively small amounts of comparative information. We 21 see in the grids back before, there was not statistical 22 significance in some of those cells. 23 Q So it's not just whether the groups are treated the 24 same, it's whether the relative probabilities are at one or 25 the other extreme; is that right? 82 1 A Oh, absolutely. Absolutely. Absolutely. I mean, 2 what we're doing is doing comparative analysis. We're 3 comparing the relative probabilities. That's what we're 4 doing. 5 Q So where two groups are treated the same and the 6 probabilities are extremely high or extremely low, that 7 contributes little information? 8 A Where they are treated the same -- 9 Q And both probabilities, the probabilities for both 10 groups are high or low. 11 A Now, I think it's true. I mean, it's a continuum in 12 the sense that if everyone were admitted, then there is no 13 comparative information. If virtually everyone is admitted, 14 amount of comparative information is small. That's true, 15 okay. I can say that, and that is what I can say. 16 Q Okay. Now, if 84 to 88 percent of the applicants are 17 in the shaded cells, that means somewhere, 12 to 15 percent, 18 16 percent of the applicants are in cells that you excluded; 19 correct? 20 A Cells that provide no comparative information, that's 21 true. 22 Q Okay. Do you consider 12 to 16 percent of the data 23 not providing comparative information to be a high 24 proportion? 25 A Oh, it's not high at all. It's not high at all. I 83 1 mean, it's cells that, particularly when you look at the 2 cells that we see what kind of cells they are, there were 3 cells for the most part, not 100 percent, but for the most 4 part, they are cells where people have presented themselves 5 with very low credentials. I mean, grade point averages 6 less than 2.5, LSATs less than 148. I mean, those are 7 the -- I mean, look where the area is, yeah, here, all the 8 cells, they are all down in this area, right, and up in 9 here, and with an occasional cell here and there, just 10 depending, as we would expect. 11 Q So 12 to 16 percent of the applicants not in cells 12 contributing comparative information doesn't trouble you. 13 What about 20 percent, would that concern you? 14 A It depends on where the -- it depends on where the 15 credentials are. If in fact everyone moved, if in fact, 16 if in fact the admissions office decided we needed higher 17 LSATs and higher grade points and we moved it up, I mean, 18 the cells where they made different decisions for 19 individuals, where they were making decisions other than 20 just all rejects or all admits, but all rejects, in 21 particular, the ones that we're losing here, in that sense, 22 if they moved it up I would be satisfied, statistically, 23 because I want to look at where there is a potential for 24 difference in potential, not whether there is, but potential 25 for a difference in admission rates, that's what I want to 84 1 look at. 2 Q So in fact, the proportion of applicants excluded 3 doesn't really trouble, doesn't concern you for purposes of 4 this analysis, that's not a number that you need to look at 5 in order to evaluate your results? 6 A I mean, it's a number I offered for information. 7 It's not a number in the sense that I really want to do 8 an analysis of the comparative information, so that's what 9 I did. 10 Q Okay. When you were looking at the proportion of 11 the applicants excluded or not providing comparative 12 information, to use you phrase, did you look to see what 13 percentage of the minority applicants fell in the cells that 14 are not highlighted on these charts? 15 A I didn't calculate it separately for each group, no. 16 Q Did you calculate it as an aggregate for all of the 17 minority groups put together? 18 A No. 19 Q For 1995, would it surprise you to learn that there 20 are 245 selected minority -- or let's see, I guess it's a 21 total of 267, 267 minority students, in other words, African 22 Americans, Native Americans and Hispanics who fall into the 23 cells that you haven't shaded? 24 A Well, I'm not going to -- I'm not going to act 25 surprised or not with respect to this statement. It is 85 1 true that minority applicants have on average presented 2 themselves with lower grade point averages and lower test 3 scores, on average, so in that sense, it may be that there 4 are more of them that are in the unshaded areas, that 5 certainly could be true. 6 Q Well, not -- I don't want to say more, because in 7 fact, by our count there are 595 applicants in the 8 non-shaded areas, 267 were minority students, and 328 9 were majority students. 10 A Okay. 11 Q If I represent that to you, does recognizing that 12 you haven't counted it, does that -- is there a reason to 13 correct me right off the bat as you did earlier? 14 A Certainly, I didn't do the calculations, so I'm not 15 going to correct you right off the bat. 16 Q And 267, if we have done the calculations right, comes 17 out, 267 minority applicants comes out to 39 percent of the 18 total number of minority applicants, my question is whether 19 excluding or constructing a model so that 39 percent, almost 20 40 percent of the minority applicants don't contribute 21 comparative information, whether that causes you any concern 22 for the design of your model. 23 A Well, the reason they are -- the reason they don't 24 provide comparative information has to do with the decisions 25 made, the results of the review of their credentials, and so 86 1 this is where we're at. I would -- I think they -- if they 2 are not admitting minority students or majority students in 3 these particular cells, then those cells are people with 4 credentials that they don't offer admission to. 5 Q So if my numbers are right, then your composite odds 6 ratio calculations are based on calculations that don't even 7 look at the information provided by 40 percent of the 8 minority applicants; is that right? 9 A Well, that would be if the percentages -- we can go 10 back and calculate, anyone can go back and calculate, 11 because I have given you all the grids, you can count and 12 see how many are in these cells. And we were looking at the 13 cells with comparative information, and that's what we did. 14 Q But assuming that that percentage is right, that would 15 mean that your composite odds ratios were calculated without 16 looking at the information provided by 40 percent of the 17 minority applicants; is that right? 18 A All the information was looked at. All the 19 information was looked at. I didn't go in and exclude 20 information. All the information was looked at. They 21 didn't provide comparative information on that composite. 22 Q And so the computer, going back to the computer, 23 didn't take that information into account when it calculated 24 the composite odds ratios that we looked at earlier? 25 A It got all the information out of there that was there 87 1 in computing the odds ratio and there wasn't any information 2 for comparison. 3 Q As you define comparative information? 4 A That's true. 5 Q Now, in your medical work, in your drug studies, have 6 you had occasion to reach conclusions about the relative 7 odds of a treatment when, say, 40 percent of the people who 8 took the placebos were excluded from the analysis? 9 A It would depend on what's -- where they were at in a 10 particular study. I don't know that that percentage has 11 ever arisen. I don't think that has ever arisen, but it 12 would depend on where they are at. Certainly, with respect 13 to analyses, we often exclude large numbers of people. For 14 instance, for instance, if we're looking at the effect of a 15 treatment on death, it will turn out -- 16 Q Treatment for death? I'm sorry. 17 A A treatment, the effect of a treatment. 18 Q I'm sorry. 19 A The effect on death of a treatment, so for instance, 20 you're giving some kind of medical treatment, a heart 21 treatment or something like that, it turns out that the 22 risk of death is highly related to age, which is, you know, 23 that seems -- there seems to be evidence of that and the 24 data would do that, would relate that, and we would often be 25 in a study where we have hundreds and hundreds of people, 88 1 but for instance, there would be no deaths from people 2 under age 40, say, in the study, and the comparisons there 3 would -- for looking at death would exclude those people for 4 whom there were no deaths, groups from whom there were no 5 deaths, if we took account of age in the analysis. 6 So what I'm saying is, depends on the special 7 circumstances of the analysis and the variables, but for 8 instance, in a medical study such as that, we may wind up 9 doing the same -- we wouldn't -- this is a standard method, 10 so we wouldn't wind up doing this kind of comparison and it 11 may be that large numbers of -- large numbers of individuals 12 would not be included in the composite odds ratio that we 13 would report for a particular effect. 14 Q Is that kind of study that you just described the 15 template that you had in your mind when you designed your 16 work here? 17 A The template I had in mind is doing a good statistical 18 analysis for comparison, using methods appropriate to a 19 binary response. 20 Q Do you equate the cells where no one was admitted, 21 the non-highlighted cells in this particular chart, to the 22 situation that you just described where there were no 23 deaths? 24 A Or where everyone died, either way. I mean, there are 25 cells that don't provide us comparative information for the 89 1 issue at interest, that's right. 2 Q So from your perspective, those two situations are 3 analogous in terms of designing the study? 4 A Well, I mean, it's important to understand that 5 you want to estimate the effects where there will be 6 probabilities, probabilities other than zero and one, 7 I mean, that's what we would do. 8 Q Now, you mentioned a moment ago that you chose the 9 cells that were defined by the law school and we went over 10 those? 11 A Yeah, followed the law school grids, that's right. 12 Q And this was a document from 1995, it's Exhibit 16, 13 the one that you had been given at the outset of your work, 14 right? 15 A That's right. 16 Q And you just took the grid definitions that you found 17 in Exhibit 16 and applied it for the other years; is that 18 right? 19 A These grid definitions came from that exhibit, that's 20 right. 21 Q Did you look at -- did you ever consider whether you 22 should use different LSAT ranges, for example, to define 23 your cells? 24 A Did I ever look at other ones? 25 Q Yes. 90 1 A No. 2 Q So across the top here, and it's a little hard to 3 read, but the cells go -- you know, figure out how to say 4 this, the third column in, for example, has an LSAT of 146 5 and 147, do you see that? 6 A Yes. 7 Q That's how it's defined? 8 A Yes. 9 Q So that's two LSAT points in that cell. 10 The next one to the right has 148 to 150, so that 11 has three LSAT points? 12 A That's right. 13 Q The next one has three again, 151 to 153, am I right? 14 A Oh, I'm sorry, yes. 15 Q And then the one continuing to the right has only two, 16 154 to 155? 17 A That's right. 18 Q And we go back to three, 156, 157, 158. 19 Did you consider, based on this pattern, whether 20 there was a -- whether it made sense to keep this definition 21 of the cells when you were designing your analysis? 22 A Whether it made sense to keep -- did I think from that 23 pattern -- no, I didn't, I didn't worry about that, no. 24 Q Just didn't consider one way or the other whether 25 these particular lines that were drawn in Exhibit 16 should 91 1 be changed for purposes of your analysis? 2 A Well, what I did is I specifically wanted to use 3 someone else's lines, if I might say. I mean, they used 4 these lines, so I used the ones they used. 5 Q Okay. Even though some cells happened to include 6 two LSAT points and some happened to include three? 7 A That's the way they looked at the data. 8 Q Okay. And if you shifted it, obviously, if you put 9 the 153, for example, people with LSAT scores of 153 into 10 the cell to the right of that and went with the 154 and 155, 11 you would change the number of applicants and the number of 12 admits in some of these cells, you would expect? 13 A Sure. 14 Q And you didn't consider whether you should adjust your 15 analysis in any way to take that into account? 16 A No, I didn't adjust my analysis to take into account 17 of that, no. 18 Q When you designed your structure based on Exhibit 16, 19 the grids that you received, did you make any inquiries to 20 try to find out whether the admissions office based its 21 decisions in any way on these particular lines, for example, 22 for LSAT scores? 23 A No, I don't think -- I don't think that that's -- 24 I don't think I made any -- I know I didn't make any 25 inquiries, if that's what you're asking. 92 1 MR. DELERY: Okay. If I could, Your Honor, I would 2 like to put up a board which is just a copy of this same 3 page without the highlighting on it. 4 THE COURT: Sure. 5 MR. DELERY: Just so we can see that. I think we 6 can take this down, I don't know. We need to figure out how 7 to turn off the light. 8 BY MR. DELERY: 9 Q Now, I think when you were here before, in your 10 powerpoint presentation -- can you see it? I should perhaps 11 turn it more so that you can have a better view of it. Can 12 you see that? 13 A Right. I see there is a number missing out of that 14 thing. I mean, someone didn't put a number up. 15 Q Okay. Actually, I think it turned out that all of our 16 copies didn't have -- copies that we had originally didn't 17 have the numbers in it, but it's 151, I think. 18 A Yeah, yeah, okay. Right. 19 Q So I'll just write that in. I think in the copying 20 process, when it came over to us, that was missing and so we 21 have figured out that it's 151. 22 But earlier, when you were here the last time, you 23 highlighted a cell here that had -- and I'm going to put 24 a black box around it on this chart, 198 applicants and 25 17 admits. Do you recall that from your powerpoint 93 1 presentation? 2 A Well, I certainly highlighted a cell and -- 3 Q I think it's Exhibit 143, if you would like to look. 4 I just want to use this same one as a starting point from 5 where you were here. 6 A Well, okay. We can start there. That's fine. 7 Q It starts on slide 16, you had a series where you 8 took us through a cell, and I think this is the cell. 9 A Yes, I see that. I see that. I see that. That's 10 fine. 11 Q Okay. At any point did you consider what would happen 12 if you had used a larger set? 13 A Whether I used these cells -- 14 MR. DELERY: Actually, let me -- if I can put up one 15 other board, Your Honor. 16 THE COURT: Sure, but I'm not sure where you're 17 going. Don't forget, this is rebuttal. 18 MR. DELERY: If you'll bear with me, I'm building 19 back up to the highlighted section, if I could do that for 20 a second. 21 THE COURT: I'm with you. 22 MR. KOLBO: Your Honor, I think it is beyond the 23 scope of cross examination at this point, and I do want to 24 lodge that objection. 25 THE COURT: I'm going to bear with him for just a 94 1 short while, but this is -- as I have indicated, rebuttal is 2 rebuttal. 3 MR. DELERY: Sure. I'll just -- I'll do it very 4 quickly, Your Honor, if I could. 5 BY MR. DELERY: 6 Q You could have drawn a cell, for example, that 7 included the eight cells around the one that you started 8 with? 9 A One could, one could do that. 10 Q And one could look at what the odds ratios would be 11 for that cell, for example, and how much comparative 12 information that would contribute as that definition was 13 made? 14 A Well, I mean, there is going to be comparative 15 information, because they -- all these cells, they are all 16 shaded, I think. 17 Q Okay. Oh, you're right, they are all within the 18 shaded range on the exhibit that you brought today. 19 A Right. If you made cells bigger and included some 20 of the unshaded ones, you would have more cells and more 21 individuals, that's math, yes. 22 Q Okay. Similarly, we could -- you could slide this 23 nine cell grouping up to the right so that you get the 24 uppermost right nine cells, for example, isn't that right? 25 You could calculate an odds ratio for that? 95 1 A One could do that for any combination of cells you 2 want, sure. 3 Q Now, you said earlier that your shaded areas included 4 80-some percent of the total number of applicants? 5 A Right. 6 Q Did you do any analysis to look at where the admits 7 were coming from, which cells contributed proportions of the 8 admitted students to the law school class? 9 A Did I calculate how many -- what percentage of the 10 admits were included? 11 Q Yes. 12 A Well, I mean, we know virtually all the admits were 13 included. Not all, but virtually all the admits were 14 included. All the cases of admissions except for the cells 15 where everyone was admitted or in the few cells where there 16 was only one ethic group, so virtually all the admits are -- 17 virtually all are included. 18 Q Okay. And in fact, isn't it the case that about 19 75 percent of the admitted students come from these nine 20 cells in the red box that I have drawn here, the uppermost 21 right-hand nine cells? 22 A Those would be the cells with the students who present 23 with very high credentials and for whom the decisions are 24 made based on those very high credentials and they -- and 25 there are -- it's not -- if I can read that from here, but 96 1 it's not uniform across that, there is quite a difference in 2 admission rates across that, but a large number of admits 3 come with that cell -- that area, excuse me. 4 Q Okay. But you didn't look to see what proportion 5 and whether, for example, it was about 75 percent of the 6 admitted students came from these very uppermost right-hand 7 corner cells? 8 A I never calculated that percentage, no. 9 Q Okay. And I guess similarly, we could draw a cell 10 above that included the students above a 150 LSAT, which is 11 about the 50th percentile, right, and above a 3.0, and could 12 calculate an odds ratio for the admitted students in that 13 larger cell; right? 14 A We can do this for any group of cells. I said that. 15 Q For any combination. 16 Would it surprise you if I represented to you that 17 95 percent of the admitted students in 1995 came from the 18 cells in this grid -- I'm sorry, I have misdrawn the line 19 slightly. I want to drawn it down one. 20 So in this range, that 95 percent of the admitted 21 applicants came from that upper right-hand quadrant, above a 22 150 and above a 3.0? 23 A I mean, you can do some calculations based on the 24 totals over here. Clearly a large percentage of the 25 admissions, admitted students, come in those cells, 97 1 that's right. 2 Q And that's consistent with your -- with your 3 highlighting, because those are essentially the cells, with 4 a few exceptions, that you highlighted this morning; 5 correct? 6 A I think almost virtually all the cells that had 7 admitted students were highlighted. I think we said this 8 already. 9 MR. KOLBO: Are you done with this? 10 MR. DELERY: For the most part. I want to ask one 11 more question. 12 BY MR. DELERY: 13 Q So would it surprise you to learn that the odds ratio 14 for that larger cell, accounting for 95 percent of the 15 applicants, is just over two? 16 A Oh, the odds ratio for a larger grid where you ignore 17 the difference in LSAT and GPA within there? 18 Q Yes. 19 A Would it surprise me that it gets down lower? 20 Q 2.45? 21 A It's going to be lower. It's clear from looking at 22 this grid, it's clear that there are incredible differences 23 in admission rates across those cells, and so combining 24 those would be -- you can do it, and you can calculate an 25 odds ratio, but you're losing a lot of information about the 98 1 individual, the credentials. 2 Q Are you familiar with the testimony of -- that has 3 occurred in the trial from admission officers concerning the 4 range of qualified applicants, what the admissions office 5 considers to be qualified applicants? 6 MR. KOLBO: Your Honor, I just want to object. 7 THE COURT: Your objection is sustained. We're way 8 beyond. 9 MR. DELERY: All right. Thank you, Your Honor. 10 BY MR. DELERY: 11 Q So a cell -- let me just ask you this one question. 12 I think it's back connected to the highlighted tables. 13 The cell that we have drawn there, the large red 14 box, has substantial overlap with the highlighted cells 15 that you identified earlier as contributing comparative 16 information to the odds ratios; correct? 17 A I think I have answered that, yes. 18 Q So the same -- the same cell that you used to 19 calculate an odds ratio above 200, the cells that you used 20 to calculate an odds ratio above 200, taken together, yield 21 an odds ratio of about 2.4? 22 THE COURT: He has already answered. He says he 23 doesn't know exactly, but it would be a different amount. 24 MR. DELERY: Okay. Fair enough, Your Honor. 25 BY MR. DELERY: 99 1 Q Last thing I want to ask you about, Dr. Larntz, is 2 your discussion of assumptions earlier this morning. You 3 indicated in response to some questions from Judge Friedman 4 that a model that took into account all of the data in the 5 cell, in all of the cells, would have more assumptions than 6 your model, which looked at only the ones that contributed 7 what you called comparative information; correct? 8 A In my estimation, my statistical opinion, that would 9 require some kind of assumption connecting the cells and I 10 made no assumption connecting the cells. 11 Q Do you believe that a model that takes into account 12 all of the cells would actually have more assumptions or 13 are you just saying that it would have this additional 14 assumption, but it might have fewer in other respects? 15 Do you understand my question? 16 A When we're counting assumptions, I mean, we're -- 17 there is a whole slew of different ways of doing the 18 counting, okay, so I'm -- I don't want to -- 19 Q That's exactly what I thought, which is why I asked 20 the question. 21 A More or less, it depends on the particular aspects 22 you're looking at, and so making fewer assumptions is 23 generally better, but you also want to validate the 24 importance of those assumptions. 25 Q Okay. Is it necessarily the case that a model that 100 1 included more of the cells in the table would include more 2 variables, is that necessarily the case? 3 A Oh, no. In fact, typically you would make -- if you 4 connect -- the more variables in the technical regression, 5 logistic regression sense, I mean, the number of variables 6 included in the model would probably be fewer in those -- in 7 that sense, right. I mean, but including variables in a 8 model is not the same as making a set of assumptions. 9 Q But both the assumptions that you make and the number 10 of variables that you include contribute to the reliability 11 of the model; isn't that right? 12 A That sounds like it's got to be true, yes. 13 MR. DELERY: Thank you, Your Honor. I have no 14 further questions. 15 THE COURT: Intervenors? 16 MS. MASSIE: Can we take a very quick break? 17 THE COURT: Yes, I am going to take a break. I 18 wanted to know if you had any questions. 19 MS. MASSIE: Oh, yes. 20 (Recess taken at 10:47 a.m.) 21 -- --- -- 22 23 24 25 SATURDAY, FEBRUARY 10TH, 2001 BENCH TRIAL - VOLUME 12 101 1 2 MS. MASSIE: We just have three questions, Judge 3 Friedman. 4 THE COURT: Okay. 5 CROSS-EXAMINATION 6 BY MS. MASSIE: 7 Q Profess Larntz, you haven't done any analysis since we've 8 last spoke in the relative odds by race of getting into any 9 particular cell in the grids -- 10 A That's correct. 11 Q And that's because doing that analysis would require an 12 investigation into inequalities in schooling and the effect of 13 stigma and stereotype on performance and racial bias in 14 standardized testing and a whole host of other questions about 15 race that are well outside the scope of your expertize; isn't 16 that right? 17 A Well, I'm not sure what it would involve, but certainly 18 I've not done any analysis like that. 19 Q So you can't tell us whether the affirmative action plan 20 at the University of Michigan Law School has the effect of 21 taking account of those reflections of the problem of race and 22 racism in our society; right? 23 A That's correct. 24 MS. MASSIE: That's it. 25 THE COURT: Oh, a real three questions. SATURDAY, FEBRUARY 10TH, 2001 BENCH TRIAL - VOLUME 12 102 1 MR. KOLBO: Nothing, your Honor. 2 THE COURT: None? 3 MR. KOLBO: Nothing, your Honor. 4 MR. DELERY: Your Honor, we would intend to call 5 Professor Raudenbush to respond to the particular points that 6 Dr. Larntz has made here this morning. And we've talked with 7 counsel and they have agreed to Monday morning if that is 8 consistent with your -- 9 THE COURT: Well, the problem I have and I don't 10 have a specific problem is that it's surrebuttal, and we've 11 talked about that the other day. Everything that I've heard 12 today is true rebuttal. I generally don't -- because then 13 we're going to get into well, two things, we're going to have 14 to bring in -- keep bringing witnesses back. However, I don't 15 know what plaintiff's position is, but I don't think it's 16 going to take long to get his testimony because his testimony 17 didn't take long, and I'm really not too worried about the 18 time or anything, I'm more worried about getting into the 19 situation where it's rebuttal, surrebuttal. I generally do 20 not allow surrebuttal, and I'm very limited on this. What's 21 the plaintiff's -- 22 MR. KOLBO: Well, I had a conversation, your Honor, 23 and we talked about scheduling, he's here, that's fine. I 24 really wasn't really thinking about the substantive reason of 25 whether or not to bring back Dr. Raudenbush, I'm not sure if SATURDAY, FEBRUARY 10TH, 2001 BENCH TRIAL - VOLUME 12 103 1 it's appropriate to do that or not. 2 THE COURT: See, my problem is if we bring him back 3 then what's going to happen -- especially when we deal with 4 experts, you know, it's even more acute when we deal with 5 experts, then I suspect he's going to say something that Dr. 6 Larntz is going to say, wait a minute, I disagree with that. 7 MR. DELERY: Your Honor, I think the reason for this 8 particular situation goes back to the issue of who really has 9 the burden here, and we're in a fairly unusual circumstance 10 where on the key issues the plaintiffs have agreed that 11 essentially the University has the burden on these points and 12 so -- that's why I think we feel it's appropriate to have him 13 respond -- 14 THE COURT: How about now? You've got an hour. 15 MR. DELERY: I think -- 16 THE COURT: I mean it can't take too long. His 17 testimony -- it's got to be very, very narrow. 18 MR. DELERY: I think, your Honor, he would like to 19 have a chance to -- and I think the reason is the particular 20 calculations that, you know, Dr. Larntz agreed that he hadn't 21 wanted to do them on the fly when he was here earlier. I 22 think Dr. Raudenbush would like to have a chance to look at 23 those. 24 THE COURT: The only difference is -- and I'm not, 25 you know -- the only difference is that you knew exactly what SATURDAY, FEBRUARY 10TH, 2001 BENCH TRIAL - VOLUME 12 104 1 the rebuttal was going to be about. It could only be about 2 one thing, and that was to rebut what your witness had to say 3 in terms of criticizing his report. That's the only thing it 4 could be. I wouldn't allow anything else. That's the only 5 thing to rebut. Everything else could have been anticipated 6 and could have been asked because they had the report, they 7 had everything about it. The only thing that couldn't have -- 8 you know, specific of that -- as I said, he's going to come 9 back and he's -- Dr. Larntz is going to say -- these are all 10 professionals, these people that believe in what they're doing 11 and so forth. So, as I say, I don't think it's going to take 12 long so it's not the time, it's just the element of how do we 13 get around it. 14 I'll give you some surrebuttal, but it's only in 15 this case, and it's only -- I mean, just limited to -- as I 16 said -- well, actually, there's only two propositions upon 17 which he probably professionally want to explain his position. 18 MR. DELERY: That's fine, your Honor. 19 THE COURT: Monday morning is fine with me. I have 20 no problems. 21 MR. DELERY: Thank you, your Honor. 22 THE COURT: Okay. We'll see you Monday morning at 23 9:00 o'clock. 24 (Proceedings concluded, 11:15 a.m.) 25 -- --- -- SATURDAY, FEBRUARY 10TH, 2001 BENCH TRIAL - VOLUME 12 105 1 2 CERTIFICATE 3 I, JOAN L.MORGAN, Official Court Reporter for the United 4 States District Court for the Eastern District of Michigan, 5 appointed pursuant to the provisions of Title 28, United States 6 Code, Section 753, do hereby certify that the foregoing 7 proceedings were had in the within entitled and numbered 8 cause of the date hereinbefore set forth; and I do further 9 certify that the foregoing transcript has been prepared by me 10 or under my direction. 11 12 ____________________ JOAN L. MORGAN, CSR 13 Official Court Reporter 14 15 Date: __________________ 16 17 18 19 20 21 22 23 24 25