1 1 UNITED STATES OF AMERICA 2 IN THE UNITED STATES DISTRICT COURT 3 FOR THE EASTERN DISTRICT OF MICHIGAN 4 SOUTHERN DIVISION 5 BARBARA GRUTTER, 6 for herself and all others 7 similarly situated, 8 Plaintiff, 9 -vs- Case Number: 10 97-CV-75928 11 LEE BOLLINGER, JEFFREY LEHMAN, 12 DENNIS SHIELDS, and REGENTS OF 13 THE UNIVERSITY OF MICHIGAN, 14 Defendants. 15 -and- 16 KIMBERLY JAMES, et. al., 17 Intervening Defendants. 18 ______________________________________/ VOLUME II 19 BENCH TRIAL BEFORE THE HONORABLE BERNARD A. FRIEDMAN 20 United States District Judge 238 U.S. Courthouse & Federal Building 21 231 Lafayette Boulevard West Detroit, Michigan 48226 22 Wednesday, January 17, 2001 23 APPEARANCES: 24 FOR PLAINTIFF: Kirk O. Kolbo, Esq. 25 R. Lawrence Purdy, Esq. 2 1 APPEARANCES (CONTINUING) 2 FOR DEFENDANTS: John Payton, Esq. 3 Craig Goldblatt, Esq. 4 Stuart Delery, Esq. 5 On behalf of the Defendants 6 Bollinger, et. al. 7 8 George B. Washington, Esq.. 9 Miranda K.S. Massie, Esq. 10 On behalf of Intervening Defendants. 11 12 COURT REPORTER: MARY F. WISNESKI, CSR-0231 13 Official Court Reporter 14 15 16 Proceedings recorded by mechanical stenography. 17 Transcript produced by computer-assisted 18 transcription 19 20 21 22 23 24 25 3 1 I N D E X 2 WITNESS PAGE 3 KINLEY LARNTZ, Ph. D. 4 Direct Examination by Mr. Kolbo 7 5 Cross Examination by Mr. Delery 116 6 7 8 E X H I B I T S 9 10 NUMBER IDENTIFICATION ADMITTED 11 137 Dr. Larntz Report 28 12 142 Fifth Supplemental Expert Report 30 13 68 Dr. Larntz Report 30 14 16 1995 Final Grid 93 15 143 Policy 115 16 17 18 19 20 21 22 23 24 25 4 1/17/01 - BENCH TRIAL - VOLUME II 1 Detroit, Michigan 2 January 17, 2001 3 * * * 4 THE COURT: Ms. Massie called. Is she here yet? 5 I was going to say we'll wait for you. 6 MS. MASSIE: Actually we got to the source of the 7 problem just after. 8 THE COURT: Murphy's Law. That always happens to 9 me. I'm in the middle of a jam of the expressway and as 10 soon as I hang up, it's gone and I'm here. 11 MS. MASSIE: Wait a minute, what you are doing. 12 I'll have to try it next time. 13 THE COURT: Everybody here? Let the record 14 reflect, looks like she's here. 15 MR. PAYTON: Your Honor, I just wanted to say 16 something quickly about the -- 17 THE COURT: Sure. 18 MR. PAYTON: Chart that I used yesterday that 19 showed the plot of all of the admitted students in 1997, 20 minority and majority. 21 THE COURT: 182, 183? 22 MR. PAYTON: That's correct. 23 THE COURT: Yes. 24 MR. PAYTON: There were questions about whether or 25 not there are LSAT scores at the range of, say, twenty. 5 1/17/01 - BENCH TRIAL - VOLUME II 1 Let me explain just briefly what we did. Although there 2 aren't LSAT scores of twenty or forty, and most of them are 3 120 or 180, in fact, Law Services does report LSAT scores 4 of zero for everyone who takes a non-standard form of the 5 tests; that is persons who are given it by hand because 6 they are disabled. And that shows up as a data point of 7 zero. And that's in the database. So in the database, 8 there are LSAT scores of zero. And they show on our chart, 9 and if you look at the line, it will be scores on the very 10 bottom line of zero, because there's reported like that. 11 So we try to present a chart that could account for all of 12 the data, including the zeros. 13 THE COURT: Okay. 14 MR. PAYTON: Okay. We also wanted to talk about, 15 and I've discussed that with all the parties, what I just 16 said. We also wanted to question something about how we 17 see the rest of our trial days for the next several days. 18 THE COURT: Great. 19 MR. KOLBO: Just on the last point, Your Honor, 20 I'm going to reserve any comments we have on that graphic. 21 And my understanding is Mr. Payton may come here with 22 another graphic as well. And I just wanted to say that for 23 the Court. 24 THE COURT: Okay. 25 MR. PAYTON: Here's, I think, a rough estimate of 6 1/17/01 - BENCH TRIAL - VOLUME II 1 where we're going to go. We believe that we will spend 2 most of today with Mr. Larntz. We believe that tomorrow we 3 will put on President Bollinger, Professor Lempert. We 4 believe that on Friday we will have our Professor 5 Raudenbush and Mr. Shields. And we believe all the rest of 6 our witnesses will probably be done on Monday. We may push 7 over a little bit, but I'm just saying that. 8 THE COURT: Great. No, I appreciate that. And 9 that also gives the Interveners some advance notice to be 10 able to start lining up their witnesses. 11 MR. PAYTON: That's correct. 12 THE COURT: I'll tell you, and I'll say it 13 probably a hundred times. There's nothing like having good 14 lawyers in the case, and that are civil to each other. 15 It's just such a nice way to preside over a case, I'm sure 16 such a nice way for each of you to practice. 17 Generally we have to fight for those kind of 18 things and here you are all agreeing and giving the 19 courtesy to each other to line up witnesses. And as I say, 20 I'll say it a hundred times because that's not enough. 21 It's really a nice way to do it. Okay. 22 MR. KOLBO: Your Honor, Plaintiffs call as our 23 next witness, Dr. Kinley Larntz. 24 THE COURT: Very well. 25 MR. PAYTON: Your Honor, I would also like to 7 1/17/01 - BENCH TRIAL - VOLUME II 1 introduce my partner, Mr. Delery, who will be conducting 2 the examination. 3 THE COURT: Great. Let me have your name one more 4 time. 5 MR. DELERY: Stewart Delery, D-e-l-e-r-y. 6 THE COURT: Thank you. 7 MR. DELERY: You're welcome. 8 THE COURT: Okay. 9 MR. KOLBO: Thank you, Your Honor. 10 K I L N E Y L A R N T Z, Ph. D. 11 was called as a witness and after having been 12 sworn was examined and testified as follows: 13 DIRECT EXAMINATION 14 BY MR. KOLBO: 15 Q. Good morning, Dr. Larntz? 16 A. Good morning. 17 Q. Could you state your full name, please? 18 A. Kinley Larntz. I'll spell it. It's K-i-n-l-e-y 19 Larntz. Last name is spelled, L-a-r-n-t-z. 20 Q. And what is your -- where are you located? Where are 21 you from? 22 A. I'm a statistician. I reside in Minnesota. 23 Q. And are you currently employed? 24 A. I'm currently self-employed. 25 Q. Okay. And how are you -- in the area of statistics? 8 1/17/01 - BENCH TRIAL - VOLUME II 1 A. As a statistician, yes. 2 Q. Okay. Prior to being self-employed, were you employed 3 by others? 4 A. Yes. I was twenty-seven years a faculty member in 5 statistics at the University of Minnesota. 6 Q. Can you tell me, just briefly run through the history 7 of your career at the University of Minnesota, what 8 positions you held there, maybe starting from the beginning 9 and bringing yourself to date. 10 A. Yes. I started at the university in 1971 as a, as we 11 would say, a lowly Assistant Professor, beginning Assistant 12 Professor. I continued, was promoted to Associate 13 Professor in 1977 and promoted to full professor in 1982. 14 And that's the position I held until I retired from the 15 university in 1998. Actually I still maintain a title. 16 It's nice of them to do that. I'm now referred to as 17 Professor Emeritus. 18 Q. Could you describe just briefly your formal 19 educational background? 20 A. Yes. Starting with college, I was an undergraduate 21 math major at Dartmouth, graduated in 1967, and continued 22 my studies as a graduate student at the University of 23 Chicago and did my Ph.D. in 1971 in statistics. 24 Q. Do you have any professional association or 25 memberships that you belong to? 9 1/17/01 - BENCH TRIAL - VOLUME II 1 A. Well, the two I maintain now are, the American 2 Statistical Association is the primary society for 3 statisticians, professional association for statisticians 4 in the country and the American Society of Quality, which 5 is, serves the same function for people interested in 6 issues of quality control. 7 Q. And have you been involved in any publications in your 8 field of expertise? 9 A. I have published. I have to say, I wouldn't have been 10 an Associate Professor or Full Professor if I had not 11 published, that's for sure. 12 Q. What's the principal journal in your area of expertise 13 in statistics? 14 A. Well, there are several journals. I guess Journal of 15 the American Statistical Association is a major journal. 16 I've certainly published there. Journals, I'm just trying 17 to think. There's a whole series been published by 18 different societies. 19 Q. Have you been involved in any editorial positions with 20 any of these statistical journals? 21 A. Yes. Sure. I served as associate editor for the 22 Journal of the American Statistical Association on several 23 occasions and I also served as editor of another journal of 24 the association called the American Statistician. 25 Q. In addition to your academic post at the University of 10 1/17/01 - BENCH TRIAL - VOLUME II 1 Minnesota, have you, in the course of your career, engaged 2 in outside consulting work in the area of statistics? 3 A. Yes. 4 Q. Can you just give us a description of what kind of 5 consulting work you've done over the years? 6 A. Well, consulting work I've done, I have to say, as 7 part of my duties at the university. I was also a 8 consultant within the university, so I worked quite 9 extensively as part of my job collaborating with other 10 researchers on research projects. And I did do some, and I 11 do a bit more now, consult for companies and government, 12 agencies. 13 Q. Can you give us some examples of the government 14 consulting that you've done over the years? 15 A. Okay, sure. I've been, worked with National Science 16 Foundation, National Institutes of Health and extensively 17 with an agency called, the National Institute of Justice, 18 overseeing a series of experiments that were done in police 19 departments on responses to domestic violence. I've worked 20 as a consultant, actually a member of a Scientific Advisory 21 Board to the Environmental Protection Agency concerning 22 small particulates. And, as a say, on the Scientific 23 Advisory Commitment that made recommendations concerning 24 that. 25 My current activity, with respect to government 11 1/17/01 - BENCH TRIAL - VOLUME II 1 consulting is I'm a consultant, statistical advisor, to the 2 Food & Drug Administration, primarily in the area of 3 orthopedic devices. And so I sit on the panel, typically 4 sit on the panel that advises the FDA on whether the new 5 medical device are to be approved or not. 6 Q. Can you give us, your -- I believe you indicated that 7 you, your post at the University of Minnesota has been in 8 the area of applied statistics? 9 A. Well, universities are organized in different ways, as 10 I'm sure everyone knows. And in Minnesota, we had two 11 departments in statistics, or actually several statistic's 12 departments. But one was called Applied Statistics and 13 that was where my appointment was. 14 And I explained that I did internal consulting 15 work within the university, and that was a responsibility 16 of each person in that department to carry on that 17 collaborative research. 18 And applied statistics as it was defined there, 19 and as I generally define it, is the area of using 20 statistics in subject matter fields. So making sure that 21 when you look at data or gather data, design studies to 22 gather data, that they are done statistically 23 appropriately. 24 Q. Can you just give us an idea of what kind of fields 25 one would use applied statistics? 12 1/17/01 - BENCH TRIAL - VOLUME II 1 A. I mean, statistics is useful everywhere. 2 Q. Yes. 3 A. You know that. And I believe that, sir. 4 THE COURT: That's why I went to law school, so I 5 didn't have to take statistics. 6 A. You had no statistics in law school? 7 THE COURT: None. Otherwise I would have probably 8 been in podiatry school or something. 9 A. I've had no law, of courses, either. 10 Q. Let me ask you, Dr. Larntz, for example, in the area 11 of medicine, medical devices, medical cures, is applied 12 statistics used in that area at all? 13 A. Certainly. I use that in that area. I was going to 14 say before that I, I basically am probably too broad in the 15 sense that I've used statistics in lots of areas across the 16 board, starting from the -- well, let's see, I've worked on 17 the, determining the wealth of the United States in 1775. 18 That's a long time ago, and I worked on that. 19 I worked on the composition of fishery by catch by 20 Japanese fishing boats in the Pacific to see what they take 21 into their nets, to see how many tuna they get and how many 22 other things arrive. 23 I've worked, as you say, in medical devices. I do 24 quite a bit of work in that area now. I've worked in 25 engineering. And I've actually written software that's 13 1/17/01 - BENCH TRIAL - VOLUME II 1 used for designing experiments by engineers. 2 Q. Have you done any consulting work in the past, in 3 connection with discrimination cases involving, say, 4 employment or other areas? 5 A. Yes, I have. 6 Q. Were you asked in this case to take a look at certain 7 issues regarding the use of race as a factor in the 8 admissions process? 9 THE COURT: Can I just ask one question? You said 10 that at the university when you applied the statistics 11 department that you were required to do some internal 12 statistics for the university. 13 A. Yes. 14 THE COURT: What kind of statistics did you do for 15 the university? 16 A. Oh, internal. What I did was I worked with 17 researchers in various areas. 18 THE COURT: I see. 19 A. So someone, say, for instance, who was working -- in 20 fact, a large part of my work was with people in medicine. 21 Someone who was getting a grant or wanted to get a grant, 22 study the effect of a drug on HIV, which I worked on for 23 about seven years. 24 THE COURT: But not academic statistics at the 25 university as part of your -- 14 1/17/01 - BENCH TRIAL - VOLUME II 1 A. -- Academic statistics in the sense of? 2 THE COURT: I don't know, whatever they may be? 3 A. You mean in the sense of administrative statistics? 4 THE COURT: Yes. Yes. 5 A. That was not what I worked on, per se at the 6 university, no. 7 THE COURT: Okay. 8 A. I was not part of the administration. I was part of 9 a, I'll call it the Research Academic Corps. 10 THE COURT: Okay. Thank you. I'm sorry. 11 Q. Were you asked in this case to take a look at, from a 12 statistical point of view whether, and to the extent to 13 which the University of Michigan Law School takes race into 14 account in the admissions process? 15 A. I was asked to look at the question of examining data 16 for, from the University of Michigan Admissions Office, I 17 presume that's where the data arrived, concerning the role 18 that ethnicity played with respect to admission's 19 decisions, yes. 20 Q. When did you become involved in looking at those 21 issues? 22 A. I believe it was the fall of 1998. 23 Q. And what were you asked to do, specifically? 24 A. Well, I was asked to, if I would be willing to look at 25 the data and see what the data would say concerning the 15 1/17/01 - BENCH TRIAL - VOLUME II 1 role of ethnicity and admissions. 2 Q. Okay. 3 A. And I wanted to, and I took on that job. 4 Q. What information did you consider in, in doing your 5 analysis? 6 A. Well, when I first started, I certainly looked at some 7 materials that were prepared by the law school concerning 8 their admissions policy. The first thing I asked for, you 9 know, being a statistician, the first thing you ask for is, 10 you know, you want to find out what kind of data are 11 available. That's what we, that's our bread and butter. 12 And so I, I would understand that there would be a 13 large number of applicants. So I wanted to try and make 14 sure I got data in a computerized form. And I eventually 15 got computerized data bases of material concerning law 16 school admissions. 17 Q. Okay. Did you look at any documents with respect to 18 the law school admissions? 19 A. I certainly looked at some documents. I think I 20 looked at some, there's certainly a version of an 21 admission's policy. There was, I think a visitor's report 22 that discussed the law school and some written materials, 23 some tabulations presented by the law school. 24 Q. I think you may have up there with you a free-standing 25 copy of Exhibit 4. Do you see that? Is there such a 16 1/17/01 - BENCH TRIAL - VOLUME II 1 document up there? 2 A. Yes. 3 Q. Can you just take a look at that and tell me if that's 4 one of the documents you took a look at as part of 5 undertaking your analysis? 6 A. Yes. I've looked at this document before. 7 Q. Okay. And was that represented to you to be the law 8 school admission's policy that was in effect for the years 9 that you were going to take a look at the data? 10 A. Well, it says it's admissions policies. That's what 11 it says on here. It's dated 4/22/92, and that predates the 12 database dates. I don't, I don't know that there's another 13 admission's policy after that. 14 Q. That's one of the documents you looked at? 15 A. Yes. 16 Q. In addressing the issues you looked at? 17 A. Yes. 18 Q. How many years of -- you mentioned, I think that you 19 obtained in electronic form the data from the law school, 20 certain data from the law school? 21 A. Yes. Over a period of time, I received large 22 databases, yes. 23 Q. And how many, how many years of law school data did 24 you look at? 25 A. I was given datas, initially from 19, they covered the 17 1/17/01 - BENCH TRIAL - VOLUME II 1 admission years of 1995 through '98. And then subsequently 2 received 1999 admission's data and the year 2000 3 admission's data. 4 Q. Okay. And in what form, you mentioned that it was 5 electronic. Can you just describe the format in which the 6 data was supplied to you? 7 A. It was a Microsoft access database. 8 Q. Okay. And what kinds of data were included in the 9 database? What kinds of information, generally speaking, 10 was contained in the database? 11 A. Well, there were a large number of tables in that 12 database. And they contained information, for instance, on 13 ethnicity, which given the subject of this case, that would 14 be important to have. Information on credentials, grade 15 point average, admissions test score. And there was an 16 index, maybe, I call it a selection index, information in 17 coded form on what school people went to as undergraduates, 18 a variety of information of that sort. 19 Q. Okay. And was the data provided to you in 20 substantially the same form for each of the years that you 21 were provided data? 22 A. Essentially. I mean, there's probably some small 23 changes in coding, but, essentially, the same form, for 24 which I admit I was grateful, so I didn't have to change 25 everything each year. 18 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Okay. Can you describe just generally, in a very 2 general fashion to begin with, what types of statistical 3 analysis you performed with respect to the data that you 4 received, just very generally. 5 A. In general the terms I -- I used some descriptive 6 statistics to look at the characteristics of the 7 individuals in the database, the applicants, and, and then 8 I also then used, I guess, in general terms, I used 9 statistical methods to examine the odds or chance of 10 admission as a function of credentials of the applicants. 11 And I guess in statistics when we get fancy, we'll call 12 that inferential statistics, but first we call it 13 descriptive statistics. 14 Q. Okay. We'll get into the details then of your 15 conclusions. We'll go through a number of items. But did 16 you form some overall conclusions? You stated some general 17 fashion of some overall conclusions that you came to with 18 respect to your analysis of these databases? 19 A. Yes. 20 Q. And were those conclusions formed to a reasonable 21 degree of certainty in the field in which you practice? 22 A. Oh, I feel quite comfortable with my conclusions, yes. 23 Q. Okay. Can you just generally describe, in general 24 fashion, what conclusions you, you drew from the data that 25 you reviewed? 19 1/17/01 - BENCH TRIAL - VOLUME II 1 A. From the data I reviewed, which is the description, in 2 trying to describe as best I could, as it was done, the 3 admissions that were done by the University of Michigan Law 4 School. And with respect to ethnicity, I found, this is 5 not a statistically technical term, but I found an 6 incredibly large allowance given to members of selected 7 minority groups with respect to their chance of admission. 8 And that all of my analyses that I did confirmed and 9 continued to confirm that for individuals that have similar 10 credentials. 11 Q. And was that true for all the years you looked at, or 12 just some of the years or? 13 A. The basic substantive conclusion. The basic 14 substantive conclusion. Of course the numbers are going to 15 vary from year to year. And if they didn't, I wouldn't be 16 in business, I guess, as a statistician. But the numbers 17 varied a fair bit. But the actual substantive conclusions 18 were exactly the same across over the years, yes. 19 Q. Now, you mentioned before that you did different types 20 of analysis. And I just want to spend a little time on 21 some detail there. You mentioned descriptive statistics. 22 What does that refer to, descriptive statistics? 23 A. Well, it describes the data, how's that? What it does 24 is try to give some summary numbers, and in a couple cases 25 I looked at summary pictures of the data to try to 20 1/17/01 - BENCH TRIAL - VOLUME II 1 understand the characteristics of the applicants, and in 2 particular, the characteristics of applicants who were 3 admitted to the law school. 4 Q. And then you've mentioned that you also employed 5 inferential statistics in your analysis? 6 A. Well, I employed statistics that would allow us to, at 7 least, when I say draw conclusions, as to whether or not 8 the, the things we were seeing are just due to chance or 9 not. And that's what I mean by inferential statistics. 10 And I certainly, the main technique I used in that 11 was a technique called logistic regression, which is a 12 standard technique that we use for looking at a binary 13 response, binary in the sense of admit or not admit, to the 14 law school, and analyzing what, what would effect that, 15 that particular response and particular, the relation of 16 that to grades, undergraduate grades, admissions test 17 scores and other factors. 18 Q. Is an examination of relative odds or odds ratios, is 19 that a form of inferential statistical analysis? 20 A. Well, it is, and logistic regression specifically 21 looks at trying to understand odds as the response, the 22 odds of, say, odds in this case, odds of admission, yes. 23 Q. So relative odds analysis is related to an analysis 24 using a logistic regression? 25 A. Logistic regression, actually the technique itself 21 1/17/01 - BENCH TRIAL - VOLUME II 1 analyzes its response, technically, the log rhythm of the 2 odds. And then when you're doing comparisons between one 3 odds and another odds, and you'll get to that, I presume. 4 We would calculate what I call relative odds or odds 5 ratios. 6 My students always accuse us in statistics of 7 having one concept in fourteen names. It makes taking 8 exams difficult. And odds, and relativity odds and odds 9 ratios are the same, the same thing. 10 Q. Can you just then define for us what in the area of 11 statistics what odds and relative odds mean? 12 A. I think, yeah, it's probably a good idea, because odds 13 is used a lot in, in, well, in our everyday life, so I can 14 do that. Is it possible I can write on the board a little 15 bit? 16 MR. KOLBO: With the Court's permission. 17 THE COURT: I have no problem. The only problem 18 it's going to kind of screw up your Elmo. 19 MR. KOLBO: I think we're actually going to use 20 this over. 21 (Whereupon an off-the-record 22 discussion was had.) 23 Q. Do you have a pen there, Dr. Larntz? 24 A. Okay. Well, what I want to do is just define for you 25 odds. And I know -- I don't presume that you know a lot 22 1/17/01 - BENCH TRIAL - VOLUME II 1 about odds. And I'll just, I'll just. 2 THE COURT: Just in Vegas. 3 A. Well then, I particularly, then I particularly need to 4 define what I mean. And in statistics we talk of the odds 5 in favor of an event. 6 And just to be clear, most odds, as reported in 7 Vegas, and I have to say the odds aren't always, you know, 8 always, they're always put in the form of odds against, 9 typically. And so, and I'll leave my comments out about 10 the Vikings. 11 But we think of the odds as a number, so in 12 statistics, we think of odds as a number. So, for 13 instance, if we have what we think of as fifty, fifty odds, 14 50/50, odds of 50/50, that means that's like a coin flip. 15 THE COURT: If you guys want to sit in the jury 16 box, you're more than welcomed to, or you may stand. 17 Whatever makes you happier. 18 A. I don't think we'll be here forever. And what we do 19 in statistics is we take the ratio of these. So fifty 20 divided by fifty. That is the chance, the chance against, 21 and we divide these. And we call, we make that a number, 22 okay. And so that would be, in our terms, an odds of one. 23 Okay. So what do we do for other chances, or probabilities 24 of admissions? Suppose there's a 25 percent chance; how 25 would we convert that to odds, just so we're clear. 23 1/17/01 - BENCH TRIAL - VOLUME II 1 So that would be what, twenty-five for to 2 seventy-five against. We do the same thing, make it a 3 ratio and then we get twenty-five over seventy-five, and 4 that's a number. The number is one third. 5 Similarly, if we did 75/25, again, whoops, 75/25, 6 that would be a number. And we get the number, 7 seventy-five divided by twenty-five, we get three. So 8 those are, those are how we calculate odds. We can do the 9 same thing for others. And I'll probably take another 10 example in a second. 11 Now, when we're comparing odds, the odds of two 12 events, and that's what we're going to do here a fair bit 13 is compare odds. We take what's call the relative odds or 14 the odds ratio. And the odds ratio then for, say, the 15 event that has probability 75 percent, to the event that 16 has probability of 25 percent, the odds ratio is just the 17 ratio of the odds. And so the odds ratio then would be 18 three divided by one third, and -- and we've divided by 19 fractions at one time in our lifes, yes. 20 THE COURT: I did. 21 A. Okay. And we come up with nine. That's the odds 22 ratio. Take one more example. 23 Suppose the, so an odds ratio of nine corresponds 24 to comparing seventy-five to twenty-five. Another one, 25 say, ten to ninety would give us, what, I think I can skip 24 1/17/01 - BENCH TRIAL - VOLUME II 1 this and just write the odds as one ninth, ten to ninety, 2 say, ninety to ten. That gives is an odds of nine. And if 3 you're comparing an event that has probably 90 percent to 4 an event with the probability of ten percent, we 5 calculation an odds radio, odds ratio, then of nine divided 6 by one ninth, which would be equal to eighty-one. So the 7 concept of odds, essential to comparison of two events, 8 probabilities are odds ratio. 9 Q. Doctor, how would one communicate that in sort of a 10 sentence if one wanted to express the relative odds of a 11 ten percent probability of something occurring, versus a 90 12 percent probability of something occurring? 13 A. In statistical terms we'd say that the odds ratio; 14 that is, the odds of, the odds ratio of the event, whatever 15 we call it, ninety, that has 90 percent probability to the 16 percent that has ten percent probability has an odds ratio 17 of eighty-one. 18 Q. Very good. Do you want to resume, resume the witness 19 stand. And let me ask you, in the area of statistics, does 20 it happen sometimes that matters simply occur by chance? 21 A. Does it happen? 22 Q. Yes. 23 A. In statistics, I think in the real world it happens 24 that things occur by chance. 25 Q. Okay. And in your profession, are there ways in which 25 1/17/01 - BENCH TRIAL - VOLUME II 1 one can summarize in which statisticians summarize the 2 degree to which an observed difference, say, a ten percent 3 probability versus a 90 percent probability might simply be 4 due to chance, as opposed to something else? 5 A. Yes. We often summarize, if we're doing statistical 6 comparison, we want to understand whether or not the events 7 mean, anything can happen. So do the events that we see, 8 are they due to chance alone or not. And we summarize that 9 several ways, quite commonly use what's called a P value to 10 summarize the degree of evidence concerning whether or not 11 something occurred by chance. 12 Q. Is the term, standard deviation, used as well in that 13 context? 14 A. Lots of ways of -- remember in statistics there's 15 often several ways to do the same thing. And so what we do 16 is we have the P value, may be a summary, or we may 17 summarize also in terms of standard deviation. So, for 18 instance, and are we going to use the board a lot? I'm not 19 sure whether these people need to stay up there or not? 20 It's up to you, Your Honor. 21 THE COURT: I think they like it there. They can 22 move whenever they want. 23 A. Okay. I'm going to take a drink of water because I 24 was counting on that when they moved. 25 THE COURT: Help yourself. 26 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Could you just summarize what standard deviation 2 analysis tells a statistician, just kind of summarize just 3 what kind of values you look at for statistical 4 significance? 5 A. What I think I better do is talk about P value and 6 standard deviation both, and we summarize the, we 7 calculate, if we're describing an outcome, whatever that 8 outcome is. We calculate the probability that we see that 9 outcome or a more significant outcome than that, if chance 10 alone were operating. 11 And in statistics we often talk of P values. We 12 have magic numbers -- I shouldn't say that too loud, there 13 might be other statisticians listening. But magic numbers 14 of .05, five percent. 15 So if an event has a five percent or less chance 16 of occurring under chance, we usually call that 17 statistically significant. That's a term that arises. We 18 get, people like us to summarize succinctly, so if it has 19 something, an event has a one percent chance or less, we'd 20 often call that highly statistically significant. And I 21 think we've stopped at that particular point. 22 Now, statistics that we look at we can often 23 summarize the departure from chance in terms of numbers of 24 standard deviations. And this is also done quite often 25 with lots of summary statistics. And so many of the tests 27 1/17/01 - BENCH TRIAL - VOLUME II 1 we do are summarized in terms of a statistic Z score, which 2 gives the number of standard deviations that the outcome is 3 away from, the chance alone outcome. 4 A two standard -- well, I should say, I'll be 5 precise, 1.96 standard deviation corresponds to a five 6 percent P value, so 1.96, we get a little sloppy. We say 7 two, okay, in statistics. And so we'll say about two 8 standard deviations corresponds to an event that's 9 statistically significant. 10 And if we summarize in terms of standard 11 deviations, a one percent event is, see if I remember the 12 number, I once forgot this number in court so I better 13 remember it today. The one percent value is 2.576 and so 14 2.57, 2.58 would correspond to a significance of one 15 percent. So, in general, in statistical terms, events that 16 have, now I'll say it in more summary forms, events that 17 correspond to departures that correspond to two or three 18 standard deviations are generally considered statistically 19 important. Events that have, that further departure are 20 obviously much more statistically significant than, than 21 that. 22 Q. For example, what would a standard deviation of five 23 signify? 24 A. Five? 25 Q. Yes. 28 1/17/01 - BENCH TRIAL - VOLUME II 1 A. Well, five is, and this is actually literally true for 2 most normal tables, it's off the chart in the sense that 3 its, we believe events that would correspond to a five 4 standard deviations have very, very minuscule chance of 5 occurring by chance alone. Calculation -- I don't have 6 that calculation with me, probably less than one in a 7 million is the probability associated with five or more. 8 Q. Now, I think you've indicated that you wrote some 9 reports in connection with the work that you did in this 10 case? 11 A. One of the things that this witness did was write 12 reports, that's true. 13 Q. Can you take a look, or Dwayne, can you show the 14 witness the books that have Exhibit 137 through, I think 15 it's 142. When you get to 137, can you let me know, Dr. 16 Larntz? 17 A. Okay. I am at 137. 18 Q. Yes. Is that a copy of your report? 19 A. It appears to be. It's dated December 14, 1998, 20 expert report of Kinley Larntz. 21 MR. KOLBO: I'd offer Exhibit 137 at this time, 22 Your Honor. 23 THE COURT: Any objection? 24 MR. DELERY: No objection, Your Honor. 25 THE COURT: Received. 29 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Will you take a look at 138 to 142 and just tell me, 2 we can speed things up a little bit if you can confirm to 3 me that those are copies of all your remaining reports 4 generated in this case? 5 A. One thirty eight, 139, 140, 141, yes. Those are the 6 additional reports that I entered in. 7 Q. Okay. And then if you could take a look at exhibit, 8 and this will be in a separate book, I think, at this 9 point, Exhibit 68. 10 A. Yes, I see Exhibit 68. 11 Q. Is that data that you assembled from the database that 12 you reviewed in connection with this case? 13 A. Yes. These are spreadsheets that I, that summarize 14 the data that was in the databases for 1995 through '98 15 concerning law school admissions, yes. 16 MR. KOLBO: Maybe to get things in order here? 17 Your Honor, I will at this time offer Dr. Larntz as an 18 expert witness in this indication. I have to do that? 19 THE COURT: Any voir dire? Anybody have any 20 objection to him testifying? 21 MR. DELERY: No, Your Honor. 22 MS. MASSIE: No. 23 THE COURT: Very well. 24 MR. KOLBO: I don't think I've then offered then 25 exhibits -- it's just been pointed out to me, I'm not sure 30 1/17/01 - BENCH TRIAL - VOLUME II 1 that it makes any difference for our purposes, but 137 also 2 contains Exhibit 1 -- or 137 also contains Exhibit 68, so 3 there's some duplication there. 4 A. Yes. Yes. I actually saw that as I went through. 5 MR. KOLBO: Okay. At this time, then, I would 6 offer the remaining of Dr. Larntz' reports, Exhibits 138 7 through 142, as well as, I guess, Exhibit 68, as long as 8 it's already marked. 9 THE COURT: Any objection? 10 MR. DELERY: No, Your Honor. 11 MS. MASSIE: No Your Honor. 12 THE COURT: Received. 13 Q. Dr. Larntz, in addition to the written reports we have 14 here, have you assembled something in the nature of a video 15 presentation that will help explain your conclusions and 16 analysis that you performed? 17 A. We prepared, picked out some slides which are, for the 18 most part, with a couple of exceptions, copies of the 19 tables and figures from the reports, yes. 20 Q. Let's take a look at the first slide, if we can. Can 21 you tell us, first of all what table one represents? Let 22 me back up a little bit. We're going to go through a 23 number of these slides. Can you tell us, for the most 24 part, where these slides came from? Are they derived from 25 some other records? 31 1/17/01 - BENCH TRIAL - VOLUME II 1 A. This is table one from the first report that I did, so 2 this is from the first, December, 1998 report, table one. 3 Q. For the most part, these, these tables, although maybe 4 in slightly different form, are, in terms of the graphics, 5 the way they're graphically displayed, these are contained 6 in your various reports? 7 A. Exactly, yes. 8 Q. As we go through these? 9 A. Exactly. 10 MS. MASSIE: Mr. Kolbo, can I stop you there for a 11 second? This was not dropped off at our office. And I 12 haven't objected until now because I understood that it was 13 exactly the same as the materials contained in Dr. Larntz' 14 report. If it's not, I'd like a copy so that I can look at 15 what's different. 16 MR. KOLBO: Your Honor, this is an exhibit that 17 was actually delivered with the witness books. It's 18 exhibit what, Dwayne? 19 MS. MASSIE: 143. 20 MR. KOLBO: 143. 21 MS. MASSIE: It was not in the materials that we 22 received. If I can just a have a few minutes to look at 23 it? I assume it's substantially the same? 24 MR. KOLBO: It should have been. I apologize if 25 it wasn't. 32 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: Okay. It's 143 in the book. 2 MR. KOLBO: One forty three in the book. 3 THE COURT: Take your time. Take a look at it. 4 MR. KOLBO: You have it there. 5 MS. MASSIE: We do have it. I apologize. 6 MR. KOLBO: We have an extra copy, Miss Massie, if 7 you'd like. 8 MS. MASSIE: No. You can go ahead. I apologize. 9 Q. Can you tell us, Dr. Larntz, what, what table one 10 summarizes or represents? 11 A. Yes. This first line, which is table one from the 12 first report that I did summarizes by ethnic group 13 classified in exactly the same manner as the variable 14 ethnicity was classified in the law school database. 15 It summarizes by ethnic group the number of 16 applicants for law school for the years 1995 through 1998. 17 So we can see there are about, well, you can see exactly if 18 we read, 4,147 applicants in 1995, and similarly 3,500, 19 about 3,537 applicants in 1998. And by each ethnic group, 20 we can count the number of applicants, so. 21 Q. Dr. Larntz, do you have a pointer there if you want to 22 use one? I'm not suggesting that you have to, but if it's 23 going to be helpful to you to make specific points? 24 A. I do have one, yes. 25 Q. Okay. Can you tell me what, what, if any, general 33 1/17/01 - BENCH TRIAL - VOLUME II 1 conclusions you drew from just table one here? 2 A. Well, these are the numbers of applicants, the raw 3 data. The conclusions are, and by the way, the ordering is 4 exactly in the same order as the variable was in the law 5 school database. I just used that, so if someone asks 6 where did the order come from, that's the order that was 7 used in the code. It just gives the number of applicants. 8 So for instance in 1995 there were 4,147 9 applicants, 45 of the applicants were listed as Native 10 American by ethnicity. 404 African-American, 2,316 11 Caucasian American, 98 listed as Mexican American, 115 as 12 other Hispanic American, Asian Pacific Island American, 13 470, not too dissimilar from the number of 14 African-Americans applicants, Puerto Rican, twenty, small 15 number, foreign, classified as 412, and unknown, 567. So 16 the largest groups with caucasian American, Asian Pacific 17 Island, African-Americans, well, except for the unknowns. 18 Q. Okay. Let's go to the next slide. 19 A. That's similar for all the years, and I think we don't 20 have a prepared a slide for 1999 or 2000, but there were 21 about 3,400 applicants in each of those years and there's a 22 similar distribution. 23 Q. Let's take a look at the next slide. 24 A. Yeah. 25 Q. Can you summarize what this, what information this 34 1/17/01 - BENCH TRIAL - VOLUME II 1 slide provides. 2 A. This is part of the descriptive statistics that I was 3 preparing. And so what I've done here is for the applicant 4 group as a whole; that is. For all applicants, I've 5 summarized the median undergraduate grade point average. 6 So, for instance, in 1995 the overall median was 7 3.49 and the median for Native Americans was 3.14, somewhat 8 lower. These are among applicants, 3.03 for 9 African-Americans, 3.55 for Caucasian Americans, compared 10 to 3.49. 3.31 for Mexican American applicants. 3.56 for 11 other Hispanic American, Asian Pacific Island applicants 12 3.48, 3.14 for Puerto Rican applicants 3.46 for foreign 13 applicants, 3.53 for those unknown ethnicity. And so, and 14 you can see that we've given that, those numbers for 1995, 15 '96, '97, '98, and I think a couple more slides will give 16 '99 and 2000. 17 Q. Let's go to the next slide then. 18 A. And similarly, let's just make sure we see what we're 19 seeing. The median, the median, I guess given the way 20 we're doing things I better make sure I define what a 21 median is. 22 A median is the value in which half the applicants 23 are at or below that value and half are at that value or 24 above. So there are applicants that are higher, higher and 25 lower, but it's the one that splits. 35 1/17/01 - BENCH TRIAL - VOLUME II 1 So, in fact, in 1999 we have actually similar 2 patterns. What we see is with respect to the overall 3 median of 3.52, Caucasian American applicants are, are at 4 3.57, Asian Pacific Island, 3.46, and then Native 5 Americans, 3.37. Probably -- I'm not trying to do this 6 particularly in order, but I'll just point out 7 African-Americans are at 3.15, somewhat lower than, with 8 the median somewhat lower than the overall. Mexican 9 American applicants, 3.36 and Puerto Rican 3.20. 10 Q. Let's go to the next slide down. Is this just for the 11 same information for the next year? 12 A. Exactly. With the same pattern of median. 13 Q. All right. Next slide please, Dwayne. What does that 14 table three summarize there? 15 A. I did the same analysis, looking at LSAT score in 16 calculating the median. These are for applicants as they 17 presented to the, to the law school. And so the median, 18 the median overall for 1995 was 162. And we can see that 19 Native Americans applicants that year had a median that was 20 eight points lower at 154, African-Americans applicants had 21 a median that was 150, twelve points lower, Caucasian 22 Americans applicant had a median that was one point higher, 23 163. Mexican American applicants had a median LSAT of 155, 24 seven points lower than the overall median. Hispanic, 25 other Hispanic American applicants had a median of 156, six 36 1/17/01 - BENCH TRIAL - VOLUME II 1 points lower than the overall. Asian Pacific Island 2 Americans had 161, which was within one point of the 3 overall, Puerto Rican applicants, 155, seven points lower, 4 and you can see foreign and other unknowns also their 5 values down there. 6 Q. And without going into detail you've reported the same 7 information for '98, '96 and '98, correct? 8 A. In '96, 1997, '98 in this, showing, showing similar 9 patterns, yes. 10 Q. Let's go to the next slide down. 11 A. And this is the same, the same display for 1999. 12 Q. Okay. 13 A. And if we look at the next slide, we get the same 14 display for the year, the year 2000. 15 Q. What was the next step in your analysis then? 16 A. Well, well, I just summarized this. I mean, in some 17 sense, the summary of this, we saw, we see that the, a 18 number of ethnic groups had median scores that are lower 19 than the average, consistently across there, and that's 20 among, among the applicants. I just let that be a summary. 21 The next step was, and we're first looking at all 22 applicants, and this is, in some sense, who presented 23 themselves to the law school for admission. Those are the 24 applicants. The next step we looked at is what are the 25 characteristics of the individuals they selected; that is 37 1/17/01 - BENCH TRIAL - VOLUME II 1 the accepted applicants. So we looked at, I looked at, 2 prepared exactly identical tables for accepted applicants. 3 That's the next step. 4 Q. Should we go to the next slide then? 5 A. Sure. 6 Q. And what information is, is summarized just briefly, 7 what information is contained here? 8 A. Sure. This is, I'm sorry. This is the same display, 9 using the median, comparing the actual accepted applicant 10 GPAs. And we can see, for instance, in 1995 the, among the 11 accepted applicants, the average, or not the average, 12 excuse me, the median. The median is a form of average, so 13 I better -- but the median was 3.64, and the Native 14 Americans were at 3.36, somewhat lower, African-Americans 15 were 3.33, among accepted applicants, again, somewhat 16 lower. Caucasian Americans were 3.68, slightly higher. 17 Mexican American applicants were 3.50, and Asian Pacific 18 Island applicants were 3.6, again, within, about a 19 hundredth of a point of the overall and Puerto Rican 20 applicants were 3.3 point, somewhat lower. 21 And so what we see higher, and we see throughout 22 this is that among accepted applicants, Caucasian and 23 American and Asians and non-American applicants had median 24 undergraduate GPAs that were similar or slightly higher 25 than the overall median. 38 1/17/01 - BENCH TRIAL - VOLUME II 1 And applicants from ethnic groups of Native 2 American, African-American, Mexican American, and Puerto 3 Rican, ethnic groups had averages, had undergraduate, 4 median undergraduate GPAs that were somewhat lower. 5 Q. Did you do similar analysis for the later years and 6 have similar conclusions? 7 A. For 1999 and 2000 we did exactly the same thing, yes. 8 Q. Can we have that next slide? 9 A. And that's a report in 1999. And the next slide 10 reports 2000. 11 Q. What was -- let's go to the next slide, Dwayne. 12 A. And then we continued in the same summary manner that 13 we did before for LSAT scores. And LSAT scores for 1995, 14 the overall median was 168. In fact, the overall median 15 was 168 for all four years, for all accepted applicants for 16 '95, '96, '97 and '98. 17 Native American applicants, or accepted 18 applicants, Native American accepted applicants were six 19 points lower the first year, and seven points the second, 20 and seven the third, and eight the fourth year, lower than 21 the overall median. 22 African-Americans accepted applicants were nine 23 points lower the first year, nine points lower the second 24 year, eight points lower the third year and nine points 25 lower the, in 1998. 39 1/17/01 - BENCH TRIAL - VOLUME II 1 Caucasian American applicants were all, either at 2 or one point above their median scores were at or one point 3 above. Mexican American applicants, accepted applicants 4 were eight points lower in '95, five in '96, seven in '97, 5 and eight in '98. 6 Asian Pacific Island American applicants were at 7 or one point above the median for all the years given here, 8 and Puerto Rican applicants, accepted applicants were nine 9 points lower in '95, eight points lower in '96, four points 10 lower in '97 and seven points lower in '98. 11 Q. Let's go, did you do the same analysis then for the 12 later two years again? 13 A. Yes. We have 1999 and we have 2000. 14 Q. Slides twelve and thirteen are for 1999 and 2000? 15 A. That's correct. 16 Q. The same LSAT analysis for accepted applicants? 17 A. The same, reporting the median of accepted applicants, 18 yes. 19 Q. All right. Let's go to the next slide, Dwayne. Can 20 you tell us? 21 THE COURT: What's the difference between "mean" 22 and "median"? 23 A. The median is exactly the value at which half or at 24 that value or above and half or that value or lower. 25 That's the median. Splits the day in half. 40 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: Right. 2 A. We also in percentile terms, that's the fiftieth 3 percentile. 4 THE COURT: Okay. 5 A. The arithmetic mean, the average, if you want to call 6 it that, basically what it does is it sums up all the 7 values and divides by how many you have. Okay. So the 8 average, the arithmetic, we typically call the mean, is, is 9 precisely that calculation. It's just another way of 10 summarizing. We could have done the same thing with 11 averages. 12 THE COURT: How, if you use "mean", how would the 13 charts differ? Other than numbers, would the spread be any 14 different, or how would it look if you used "mean" instead 15 of "median"? 16 (Whereupon an off-the-record 17 discussion was had.) 18 A. Well, I don't recall the exact numbers, Your Honor. I 19 would -- my feeling is it would be similar in, in 20 difference, but I don't. I don't have the actual numbers. 21 THE COURT: I see. That is would probability, if 22 you chart it, it would be about the same? 23 A. In the case what would be different would be if there 24 are extreme values and extreme values would tend to pull 25 the mean up or down, depending on how extreme the value is. 41 1/17/01 - BENCH TRIAL - VOLUME II 1 May I give an example? 2 THE COURT: Sure. 3 A. Not in this area. 4 THE COURT: Yeah. 5 A. The example I used to use in class, I used to use an 6 example of, and I'm try to make it so I can do the math in 7 my head although I've got a conductor here if I have to. 8 THE COURT: Sure. 9 A. So my wife worked in a dental office, okay, and we 10 looked, I looked at the salaries. I'm making up this 11 example, by the way. She did work in the dental office but 12 I'm making up the numbers. 13 THE COURT: Yeah. 14 A. And the individuals working at that dental office, the 15 salaries of the individuals working in the dental office 16 were $20,000 a year, $20,000 a year and $260,00 a year. 17 There were three individuals working in this dental office. 18 Do you see what I'm saying? 19 THE COURT: Uhm-uhm. 20 A. Two at twenty, and two sixty, okay. The arithmetic 21 mean of that is a hundred thousand dollars a year. So 22 that's the average salary in a dental office. And someone 23 might say, that doesn't really summarize what's going on in 24 that office with respect to salary. The median actually in 25 this case would be $20,000, so. 42 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: I see. And why, you decided to use 2 the median here instead of the mean? 3 A. Well, I probably could have used either in this case, 4 but when there are extreme values, the median is not 5 effected by those extreme values. 6 THE COURT: I see. 7 A. And that's, generally, why I would report the median 8 typically, although I don't think there's any evidence in 9 this case that the mean would give any different answer, 10 although I don't have those numbers calculated in front of 11 me. 12 THE COURT: But because there's no extremes like 13 in your example, the median you thought was the way to do 14 it? 15 A. That's what I thought would be the way to do it. You 16 know, there are some reported zeros in the GPAs and the 17 LSAT. I think you actually heard about those earlier, and 18 the median would be effected very little by, if we 19 concluded or excluded those values, and I can't even 20 actually remember whether I did include or exclude those. 21 The mean obviously would be effected quite a lot. It would 22 lower those values quite a bit. 23 THE COURT: Okay. You've answered my question. I 24 appreciate it. 25 Q. Can you tell us in what kinds of analysis you 43 1/17/01 - BENCH TRIAL - VOLUME II 1 performed that you displayed with respect to slide fourteen 2 on your presentation? 3 A. Well, the median isn't the whole story. I wanted to 4 look at the actual distribution of scores and this is, this 5 is a standard summary technique that is used in statistics, 6 part of many statistical passages. A summary that consists 7 of, there are a lot of applicants, let me just say. There 8 are a lot of applicant. 9 And so displaying every single value isn't very 10 useful. And so we need to display a summary, and so what 11 we've done here is display the distribution of the 12 undergraduate grade point averages for accepted applicants 13 and we're displaying this as a function of ethnicity. So 14 let me just try to tell you what this box plot means. 15 What we have here is the, for Native Americans 16 accepted applicants, we have a box and some lines and 17 brackets and another line below, and the values that are 18 highlighted here, if I might, the white space in the 19 middle, that corresponds to the median, okay. So exactly 20 the same values that we have before, the median here would 21 be that white space in the middle of the box. The box 22 extends up to, remember the median was the fiftieth 23 percentile. And what we did is take the value that 75 24 percent of the applicants at or below the seventy-fifth 25 percentile so the box extends out the seventy-fifth 44 1/17/01 - BENCH TRIAL - VOLUME II 1 percentile on the lower and the upper side and the 2 twenty-fifth percentile on the lower side. 3 And then this technique, this box, there's several 4 ways to make them, but what it does, it doesn't report any 5 other values out beyond that unless the values are far away 6 from the box in a relative sense. So what it does is it 7 takes up the other values, the other, let's see there's 75 8 percent here, so about 25 percent of the data up here, so 9 about 25 percent of the accepted applicants in this range 10 here, about 25 percent are in the range below, down to the 11 bracket, unless a value is considered more extreme, 12 relative to its own box. And so there is one value down 13 here, see a bar down here, a line. That corresponds to the 14 undergraduate GPAs of one accepted applicant that's found 15 at that value. 16 Q. Is that one bar down there just representing one 17 applicant? 18 A. I believe it does. It actually could, given the way 19 the computer works. If there were two that had exactly the 20 same score, it would override them. That's true of all the 21 plots that you see, Your Honor. Further sometimes if 22 they're exactly the same values, they're going to overprint 23 unless we do some special, and I didn't do anything to do 24 that. 25 Q. What does the term, outlier, mean in statistics? 45 1/17/01 - BENCH TRIAL - VOLUME II 1 A. Well, there are whole books written on that, none of 2 which I've written, by the way. Outlier means a value 3 that's considered far away from the normal pattern or range 4 of the data. And this particular, this particular figure 5 is constructed so that these values that are printed 6 outside are termed outliers. That's the term that's used. 7 So they're further away from the bulk of the data than the 8 rest of the values. 9 Q. The lines outside the two brackets at the top and 10 bottom are outliers? 11 A. Well, yes, that's what this plot refers them as, yes, 12 outliers, yes. 13 Q. I'll ask you just summarize what your conclusions were 14 based on the analysis you did here for the median GPAs for 15 applicants accepted in 1995? 16 A. Yeah. The GPAs from the plots here, you can see that 17 in looking at these box plots, you can see that the Native 18 Americans box, the African-American box that, they're in 19 the same order as we had for ethnic groups before. And to 20 some extent the Mexican American box, that is the box 21 contains the, between the seventy-fifth and twenty-fifth 22 percentile. And the Puerto Rican box are lower than the 23 other groups. You can see the boxes are all lower. 24 We saw that before, but the boxes themselves are 25 somewhat lower, and Caucasian Americans and Asian Pacific 46 1/17/01 - BENCH TRIAL - VOLUME II 1 Americans have values, and their range of accepted, of 2 accepted applicants is somewhat higher with respect to 3 GPAs. And that's also true of other Hispanic Americans and 4 foreign and unknowns. 5 Q. Now, did you do this similar information, or did you 6 construct similar grids for the later years, 1996 through 7 2000? 8 A. My reports contain plots like this for every year, 9 yes. 10 Q. We don't have all those on your presentation here 11 today, the slide presentation? 12 A. I think it's probably best we don't show all those, 13 but they're all in the reports. 14 Q. Could you summarize in a very conclusionary fashion 15 whether your conclusions were the same, were the same or 16 similar with respect to what you found in these later 17 years, 1996 to 2000? 18 A. There's no substantive difference in the conclusions 19 for each, each of the substantive years. 20 Q. You can find these box plots in your reports? 21 A. Yes. The four for 1995 through '98 are in the first 22 report. And the one for 1999 is in the report referring to 23 1999 analysis. The one for 2000 is in the report referring 24 to the 2000 applicants analysis. 25 Q. Can you just explain why it is that, that your, this 47 1/17/01 - BENCH TRIAL - VOLUME II 1 data is contained in several different reports? Just give 2 is an explanation for that? 3 A. Why it's in several reports? 4 Q. Several different reports for several different years? 5 A. Well, when I constructed the first report, I only had 6 the data for 1995 through '98. Later on I got the data for 7 1999 and conducted a report for that and later I got the 8 data for 2000 and conducted a report for that. 9 Q. Let's go to the next slide. What was the next step in 10 your analysis? 11 THE COURT: Excuse me. Hey, Len, if you guys want 12 to move over to the jury box, too, you're more than welcome 13 to. 14 MR. NIEHOFF: Thank you, sir. I appreciate it. 15 THE COURT: Excuse me. I saw them struggling with 16 the book. 17 MR. KOLBO: We have more copies of this 18 presentation. 19 Q. Can you tell us, Dr. Larntz, what the next step of 20 your analysis was, as reflected by slide fifteen? 21 A. Right. This is a box plot constructing the same way 22 for accepted applicant LSAT scores. And we can see the, 23 the same pattern in the sense that there are four boxes 24 that are lower than the others; those for Native Americans 25 accepted applicants, African-Americans accepted applicants, 48 1/17/01 - BENCH TRIAL - VOLUME II 1 Mexican American accepted applicants and Puerto Rican 2 accepted applicants. 3 And in fact, I guess in this case, the boxes 4 themselves are all at or below the lower end of the boxes 5 for Caucasian Americans and Asian Pacific Island American 6 applicants. That means in terms of what we're talking 7 about, the seventy-fifth percentile, the upper one of the 8 box, the seventy-fifth percentile for these four groups 9 individually are lower than the twenty-fifth percentile of 10 accepted applicants for Caucasian Americans, Asian Pacific 11 Island Americans. 12 Q. And did you construct similar grids and do similar 13 analysis for the later years as well? 14 A. Yes. Lots for 1996, 1997, 1998, 1999 and 2000 are 15 contained in my reports. 16 Q. And the conclusions there, are they the same or 17 similar? 18 A. The substantive conclusions stay the same. Obviously 19 the numbers and the actual plot positions change somewhat, 20 as we'd expect to vary from year to year, but the 21 substantive conclusions remain the same, yes. 22 Q. Okay. Let's go to the next slide. Dr. Larntz, can 23 you tell us whether slide sixteen reflects the next step 24 that you took in your analysis? 25 49 1/17/01 - BENCH TRIAL - VOLUME II 1 (Whereupon an off-the-record 2 discussion was had.) 3 Q. And the question again is if you can tell us what, 4 what your next step in the analysis was and how it's 5 reflected on slide sixteen. 6 A. Well, what I did, this was actually a grid. I call it 7 an admission grid of LSAT and GPAs for all applicants in 8 1995. And this is a, this is actually my construction 9 duplication of a grid that was given to me when I 10 originally got materials for the case of an admission grid 11 that was constructed by, I presume, by the admissions 12 office of a law school. 13 And what the grid gives is for, it cross 14 classifies individuals by LSAT score. And there are a 15 number with no LSAT score in the range that we think of 16 from 120 to 180, and then classifies individuals by LSAT 17 score for categories 120 to 145, out through, well, this 18 doesn't look very good on here, but it's 170 and above. 19 Q. It's the last category. 20 A. So it classifies applicants by their LSAT score in 21 relatively small ranges of LSAT, and their, cross 22 classifies them by grade point average. So the first line 23 for grade point average is 3.75 and above, and then by 24 quarter grade points down until we get to below two. So 25 for instance, and I don't know, I don't know if you can see 50 1/17/01 - BENCH TRIAL - VOLUME II 1 the lower right-hand corner or not. 2 THE COURT: I can see it on my own chart. 3 A. Okay. In the lower-right hand there are all together 4 4,147 applicant. 5 THE COURT: Right. 6 A. And we cross classified the individual applicants, and 7 I also put in this chart the number admitted, and the 8 number right below is the number admitted. So there are 9 1,130 admitted. And for individual cells, we can go at, 10 for instance, at a combination and we highlighted one, 11 individuals with a 3.25 to 3.49 GPAs and LSAT of 161 to 12 163, in that range. There were 198 applicants overall, 13 198, 17 of whom were admitted. 14 So with my charts from before, you could calculate 15 the odds by just looking at, it would be a ratio of 17 16 divided by, I'll do my math, 181, so you could actually 17 calculation the, the observed odds of admission for a case. 18 Q. Can I ask you, Dr. Larntz, how it was you decided to 19 display your materials in this graphic form? I think you 20 may have alluded to it, but could you give a more detailed 21 explanation as to how it is you came to explain the 22 information in this format. 23 A. Thank you. Well, this is exactly the format that I 24 received. I think we, it was referred to me as Exhibit 16 25 from the law school. I'm not sure what exhibit number that 51 1/17/01 - BENCH TRIAL - VOLUME II 1 was or where it came from. 2 Q. Dwayne, could you show the witness Exhibit 16, if he 3 doesn't have that in front of him. 4 A. I have it in front of me here. 5 Q. And do I understand that you did your own work with 6 database to construct a grid in the same fashion as is 7 displayed on Exhibit 16? 8 A. Yes. I used the database, the computerized database 9 that was provided to me and reconstructed Exhibit 16 to my 10 own satisfaction that these numbers were exactly the same. 11 Q. And was it with the use of Exhibit 16 that you decided 12 upon the manner in which the different LSAT combinations 13 and grade point averages should be combined or put together 14 in a cell? 15 A. I used exactly the same categorization as Exhibit 16, 16 yes, I used, that's what I used. 17 Q. If we go to the next slide then, Dwayne. 18 A. Exhibit -- 19 Q. Or tell us what slide 17 represents. 20 A. In the, in Exhibit 16 there are further breakups of 21 the all applicants into various sub-groups, and sub-groups 22 by ethnic group and sub-groups by gender and combinations 23 of ethnic group and residency. And so what, and the next 24 page in Exhibit 16 is the admission grid for Native 25 American applicants only. You can see there are, what, in 52 1/17/01 - BENCH TRIAL - VOLUME II 1 this case, 45 applicants, fourteen of whom were admitted 2 overall. And in the cell we were following, there were two 3 applicants, two of whom were admitted. So both were 4 admitted. 5 Q. And how would you summarize the odds for that 6 particular, that particular cell? 7 A. If I were calculating odds, now we're into 8 mathematics. So it's two versus zero, right, two admitted 9 and none denied. And so technically, the odds in that 10 case, if we divide by zero, they turn out to be infinity. 11 Obviously that's, that's a big number, if I want 12 to say that, in statistical terms, but, in fact, we know 13 that with small numbers that we have here, we're going to 14 see numbers like that, numbers like infinity. I would 15 believe that if there were many more applicants, and there 16 weren't. But if there were many more applicants, we would 17 probably see some accepted and some denied. The odds would 18 be different than infinity. 19 Q. The next slide there. What does slide 18 demonstrate? 20 A. This is the, again, in the next, Exhibit 16, chart was 21 for African-Americans applicants and they're in this order 22 because that's the order at which the variable is in the 23 database. And that's the order in which they were 24 displayed in Exhibit 16. 25 And so we have African-Americans applicants. 53 1/17/01 - BENCH TRIAL - VOLUME II 1 There were 404 applicants, 106 of whom were admitted. And 2 in the cell we're following, there were four applicants, 3 four of whom were admitted. 4 Q. Next slide? 5 A. This is the cell for Caucasian American applicants. 6 Again, remember we had 2,316 applicants, 668 of whom were 7 admitted. And in the cell we're following, there were 126 8 applicants, five of whom were admitted. 9 Q. And what did you calculate the odds for that 10 particular? 11 A. If I calculated the odds, it's five divided by 121, 12 five divided by 121. I could do that with a calculator. 13 I'm not going to try to do in my head. 14 THE COURT: You can use it if you wanted to. 15 Q. Why don't you go ahead and tell us what that number 16 represents. 17 THE COURT: He doesn't have to. I just thought if 18 he wanted to. 19 MR. KOLBO: I'd just as soon, Your Honor. 20 A. Five divided by 121 is 0.041 so in odds of 0.041. 21 Q. Okay. Then if we could proceed to the next slide. 22 What does this information tell us? 23 A. The next slide is for Mexican American applicants. 24 There are 98 total applicants, 41 of whom were admitted. 25 And in the cell that we're just going through to summarize, 54 1/17/01 - BENCH TRIAL - VOLUME II 1 there was one applicant and that applicant was admitted. 2 Q. And the next slide? 3 A. Other Hispanic Americans, as classified in the 4 database, 115 applicants, 15 of whom were admitted. And in 5 the cell that we're looking at, seven applicants, one 6 admitted. 7 Q. And the next slide. 8 A. This is Asian Pacific Island Americans, as tabulated 9 in the database. There was 470 applicants, 111 of whom 10 were admitted. And in the, again, the grid cell we're 11 following, there are twenty applicants, two of whom where 12 admitted. 13 Q. And could you calculate the odds on this? 14 A. This one I can actually calculate, because that's, 15 that corresponds exactly to what we had before, isn't that 16 right, a ten percent chance of, ten percent chance, so two 17 to eighteen, so the odds are one ninth, one ninth. 18 Q. And the next slide. 19 A. This would be for Puerto Rican applicants. And there 20 are a few, twenty, not a lot of individuals classifieds as 21 Puerto Rican Americans. Twenty were, there were twenty 22 applicants, five of whom were admitted. And in this 23 particular cell there weren't any. So as far as 24 information about Puerto Rican American applicants, this 25 cell doesn't tell us anything, because there aren't any. 55 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Okay. And next. 2 A. Foreign applicants, overall twelve applicants, twelve 3 of whom were admitted, and the cell we're following, three 4 applicants, none of whom were admitted. 5 Q. Okay. Let's go to the next cell, or next slide then. 6 A. And finally, this is the last one in this series that 7 give, that give the nine, by ethnic groups. There are 560 8 individuals classifieds as unknown ethnicity, 158 are 9 admitted, and in the cell we're following, 35 applicants, 10 two of whom where admitted. 11 Q. Next, Dwayne. What does this slide summarize? 12 A. This slide is just a summary, looking at that, the 13 particular cell that we happened to follow through in the 14 database. The 3.25, 3.49 GPAs, LSAT range of 161 to 163, 15 and this just summarizes the number of applicants for each 16 of the ethnic groups and the total, and the number of 17 accepted applicants for each of the ethnic groups, among 18 those who applied. 19 Q. Okay. Let's go to slide 27 then. Can you? 20 A. Well, this. 21 Q. Tell us what the next step in your analysis was, Dr. 22 Larntz. 23 A. If we wanted to compare, say, two ethnic groups with 24 respect to their chance of admission with similar 25 credentials; that is, similar GPAs and similar LSAT scores, 56 1/17/01 - BENCH TRIAL - VOLUME II 1 classified in the same way that the law school did in their 2 Exhibit 16, we could, for instance, put the tables side by 3 side, and try to read them, which would be a nice exercise. 4 I don't know how readable they are in the written report. 5 But we could, we could, for instance, look at, 6 say, well, I think we have it set up to bring up the, the, 7 the actual comparison by, say, all the individuals with 8 LSAT scores of 161 to 163 for both groups. 9 So we could, for instance, look at the, at the 10 comparison of African-Americans applicant, and their 11 admission to Caucasian American applicants and their 12 admission. So, for instance, for individuals with that 13 range of LSAT score and grade point averages in the highest 14 category, 3.75 and above, for African-Americans applicants, 15 it was three admitted out of three. And for Caucasian 16 American applicants, it was eight out of 93. 17 In the next cell down, 3.5 to 3.74 grade point 18 average, among African-American applicants that there were 19 six applicants, five of whom were admitted and among 20 Caucasian American applicants, there were 161 applicants, 21 14 of whom where admitted. 22 Going down to the next grade point average for 23 African-Americans, four out of four were admitted. And 24 Caucasian American applicants, five out of 126. That's the 25 cell we were following before, exactly the same, so the 57 1/17/01 - BENCH TRIAL - VOLUME II 1 numbers are the same. 2 To the grade point average range from three to 3 3.24, seven out of eight African-Americans admitted, and 4 two out of 42 caucasian Americans admitted. For 2.75 to 5 2.99, African-Americans, there were four applicants. All 6 four were admitted. Caucasian American applicants, there 7 were 14 in that combination, none of whom were admitted. 8 Grade point average of 2.5 to 2.72, 9 African-Americans admissions were two out of three, 10 Caucasian Americans, zero out of seven. 11 And in the next one, which is 2.25 to 2.49 there 12 was one African-American applicant who wasn't admitted, and 13 five Caucasian American applicants, none of whom were 14 admitted. And so, in fact, in this cell, there's really no 15 ability to compare admission rates in the sense that 16 they're both the same, but there's no, no one was admitted 17 in that particular, no one is admitted in that particular 18 cell. 19 In order to make a comparison, at least in terms 20 of an odds ratio, we have to have individuals that two, 21 each ethnic group in the class, in the cell, and we also 22 have to have some admitted and some denied. And so in this 23 case, both were denied. The next cell has no applicants 24 and no, well, you can see what goes on down below. 25 Q. If you have a, if you have, if you're comparing two 58 1/17/01 - BENCH TRIAL - VOLUME II 1 groups and all are admitted from one cell and all are 2 admitted from that same cell in the other group, can you 3 compute relative odds for that, for those two groups? 4 A. Actually the relative odds is, it could be calculated 5 as being either an infinity, overinfinity in the case 6 you're saying, if all were admitted, and we wouldn't, and 7 we would say that that cell didn't have any comparative 8 information with respect to relative odds. 9 Q. And the same is true if you have cells in which no 10 one's admitted from either group? 11 A. Again, in that case if no one is admitted, the odds 12 are zero for each of the cells. Then it would be a zero 13 over zero. We wouldn't define that either. We would say 14 that also gives no comparative information. 15 Q. Okay. Where there's comparative information, that's 16 when you calculate, when you can calculate relative odds? 17 A. Where there's comparative information, you can 18 calculate relative odds. Although you may wind up as we 19 did in our example with some examples with small counts of 20 infinities or corresponding into zeros. 21 Q. Okay. For example, the first, at the very top there, 22 three over three, and 93 over eight, would that calculate 23 out to infinity because in the case of the African-American 24 applicants one hundred percent of them were admitted? 25 A. That's correct. You'd wind up with three over zero, 59 1/17/01 - BENCH TRIAL - VOLUME II 1 which would be infinity, divided by 91 over 85, which is 2 not, and so we'd wind up with an infinity there, yes. 3 Q. And the cell below that where there are, where it's 4 less than a hundred percent for either category, could you 5 calculate relative odds for the six over five and the 161 6 over 14? 7 A. Yes, you could. 8 Q. Using the same method that you've explained earlier? 9 A. Right. It would be five over one, so there would be, 10 five would be the odds, five would be the odds. They 11 observe odds of African-Americans admits and five over 121 12 would be the odds of admission for Caucasian American 13 applicants. 14 Q. Actually it's the cell above? 15 A. I'm sorry. I'm glad someone's checking me when I read 16 these. I apologize. It's, you're right. It's 14 out of a 17 hundred and, now I lose my train of thought, 149, I 18 believe. 19 Q. All right. Let's go to the next slide then. Can you 20 do the same, you've illustrated these comparisons with 21 respect to a column of LSATs. Can you do the same thing, 22 illustrating a column, a horizontal column with grade point 23 average ranges? 24 A. Exactly. I think we're set to look at those for the 25 same range that we did before. 60 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. And what, what information is summarized here? 2 A. Are we waiting for another one to come up? 3 Q. Sure. 4 A. Thank you. What we've done here is for the grade 5 point average of 3.25, 3.49, we've looked at the, across 6 LSAT score combinations as laid out in the Exhibit 16 7 categories and we've compared African-American admission 8 chances to those of Caucasian applicants with the same 9 grade point average, and then we can see how it changes is 10 a function, or as LSAT changes. 11 So in essence here, if we look across the row for 12 African-Americans applicants, we can see that those 13 African-Americans applicants were very low LSATs 14 relatively. There's 120 to 140. There's, well, none of 15 them were admitted. There were 15 applicants. And 16 similarly in the next category, none were admitted. 17 And as we go across, two out of six in the next 18 cell were admitted, three out of seven in the next one, 19 four out of five. And then actually in this case, once you 20 get to 156 and above, it looks like in this case, all the 21 African-American applicants with LSATs in that range, and 22 that value or above, ten out of ten, three out of three, 23 four out of four, one of one, two out of two, all of the 24 were admitted. 25 So you can see as LSAT increases, the chance of an 61 1/17/01 - BENCH TRIAL - VOLUME II 1 African-American applicant being admitted increases. And 2 that's consistent with, I think, the stated law school 3 policy. And similarly we can look at the same, at the same 4 increase for caucasian applicants. 5 It turns out that, that all Caucasian American 6 applicants below 155 are not admitted except for those for 7 whom there was a no-LSAT category, so at least all of those 8 that had numerical value from 120 to 155, none of those 9 were admitted. And then admissions started in the 156 10 range, exactly the same category in which all 11 African-American applicants then are admitted. 12 And then we had one out of 51 compared to ten out 13 of ten, one out of 61 compared to five out of 126, compared 14 to four out of four, eleven out of 92, compared to one out 15 of one. 38 out of 78, compared to two out of two. And 16 actually the last cell has 55 over 74, but there weren't 17 any African-American applicants in that cell. 18 Q. Let's go to the next slide down. You spent some time 19 taking a look at these cells and some comparisons. If you 20 could just remind us again how relative odds would be 21 calculated looking at those comparisons. 22 A. All right. This should be a summary exactly of the, 23 of what I did on the board earlier; that is, that if we 24 look at calculating relative odds, relative odds comparing 25 a 75 percent probability of admission to a 75 percent 62 1/17/01 - BENCH TRIAL - VOLUME II 1 probability of admission corresponds to a relative odds of 2 nine, just exactly the same way we did before. And 3 similarly, a relative odds of a 90 percent probability of 4 admission, two out of a ten percent probability of 5 admission the relative odds would correspond to 81. 6 Q. You can go to the next slide then, Dwayne. Can you 7 tell us, Dr. Larntz generally what kind of analysis then, 8 perhaps you can draw us some detail about what kind of 9 overall analysis we did to assess the relative odds of 10 acceptance for the different racial groups that you studied 11 or other groups that you studied, as well. 12 A. Well, what we wanted to do next, or what I wanted to 13 do next, and what I did next was to say, try to provide a, 14 what I'll call a composite measure of the relative odds. 15 Many of these cells are very small, relatively small. They 16 have small numbers of counts and small numbers of 17 individuals. Some cases there are cells where there are no 18 admissions. Some cases there are cells where there are a 19 lot of admissions. 20 And so what I did was construct a statistical 21 model that allowed us to calculate a composite relative 22 odds combining all the cells, all the cells in the grids 23 that have comparative information. So in all the cells 24 where there was comparative information, they would 25 contribute to a composite relative odds. 63 1/17/01 - BENCH TRIAL - VOLUME II 1 And what I wanted to do was summarize, I mean, the 2 relative odds that we saw before, some were very, some, a 3 lot were infinity. Some were other numbers. Can we put 4 those together and the statistical method I used which was 5 logistic regression allows us to create an estimate of the 6 relative odds comparing the odds of admission from one 7 ethnic group to another ethnic group. 8 And the goal of that analysis was to summarize 9 the, the weight or the effect in terms of relative odds 10 that are given to members of various ethnic groups with the 11 same grade point average range as we did in Exhibit 16 and 12 the same LSAT range. So individuals that have the same 13 credentials with respect to those two variables. 14 Q. Now, you've mentioned that there are occasions when 15 you get a calculation of infinity because, for example, all 16 applicants from one minority group might be admitted and 17 you have less, you have some number of majority applicants 18 who are denied admission and you'll get an infinity value, 19 correct? 20 A. That's correct. 21 Q. And how is that accounted for in your assessment of 22 trying to summarize in overall fashion the relative odds of 23 different racial groups when you have cells that shows 24 infinity as a value for relative odds? 25 A. Well, those cells contribute information. They're 64 1/17/01 - BENCH TRIAL - VOLUME II 1 not, they're not no-information cells. They certainly 2 contribute information. And so the statistical technique 3 that's used, logistic regression makes those a composite, 4 makes a composite estimate from those cells accounting for 5 all the information, all the cells that have comparative 6 information. And so, for instance, if I might use a very 7 mundane example, and I apologize, Your Honor, if you're not 8 a Chicago Cubs fan, but I go to a Chicago Cubs game every 9 year. 10 I was a graduate of the University of Chicago and 11 they collapsed in 1969 when I was there. And so I followed 12 them from then on. And I have a son who was born in 13 Chicago, doesn't live there any more, lived there only a 14 couple years, but he said that's his hometown, so we have 15 to go back. 16 THE COURT: Oh, yeah. I had a law clerk who 17 wanted to visit every single -- I'm not a baseball fan, and 18 she wanted to visit every single baseball stadium in the 19 country. So we traveled quite a bit hearing cases all 20 over. So I've been to many stadiums. I don't think it was 21 a Cub game. Steve, you want went to a Cub game, didn't 22 you? 23 (Whereupon an off-the-record 24 discussion was had.) 25 THE COURT: But not with Barb. She'd always say I 65 1/17/01 - BENCH TRIAL - VOLUME II 1 need to, assign it here, because I hadn't seen that 2 stadium, and then they started building new stadiums. She 3 just went crazy because, you know, we'd have to go back to 4 cities again. Anyhow, go on. I've been to more games than 5 I wanted to with her. 6 A. Well, I have to do as a father, of course. But, and 7 so I sacrifice myself for that, but I don't -- anyway. 8 At any rate, we've sometimes watched Sandy Sosa 9 play. And we are very curious while he plays each day. 10 And sometimes he does very well. And for instance, he, he 11 bats thousand in some individual games. That's true. I've 12 seen him do that, hit three home runs and bat a thousand. 13 That's a pretty good day for Sandy Sosa. That's a pretty 14 good for most baseball players, right. 15 And so those a thousands are like, are like 16 infinities, in some sense. They're sort of off the scale. 17 And we've also seen Sammy Sosa on other days, I have to 18 admit, strikes out every time at bat, and that's like a 19 zero, nothing, no. And so neither one of those probably 20 represent his batting average overall, right, because, you 21 know, a batting average overall is a composite of a large 22 series of games. 23 And these numbers that I have here, and we 24 certainly would include the games where he, he did very 25 well, and the games where he didn't do very well. And 66 1/17/01 - BENCH TRIAL - VOLUME II 1 these estimated relative odds that we have here are 2 somewhat composites of the relative odds that we see in 3 the, in the individual cells. 4 Q. So in calculating Sammy Sosa's batting average, you 5 don't throw out the games where he bats a thousand, do you? 6 A. No. 7 Q. You don't throw out the games where he bats a zero? 8 A. No, you don't. 9 Q. You include those and they get factored into the 10 analysis? 11 A. That's correct. 12 Q. Is that the same here with respect to the analysis 13 that you did? 14 A. In some sense that's the same. We've got an infinity 15 and a zero, corresponding to the same kind of thing in our 16 case. They correspond to boundary cases, that is, in some 17 cases, a minority sub-cell where we had a hundred percent 18 admitted. That would correspond to an odds ratio of 19 infinity. We certainly include all that information in 20 this composite. 21 Q. Now, you've provided some information about how 22 relative odds are calculated. And we've seen some 23 information about that on the board. 24 Can you tell us how, outside the context of a case 25 like this, how would the other areas where you have 67 1/17/01 - BENCH TRIAL - VOLUME II 1 performed statistical work, whether it's regulatory work or 2 medical work, medical devices, how are relative odds used 3 in those fields, generally speaking? 4 A. Well, actually, in a clinical study, say of a new 5 medical device or a new drug where we would be comparing 6 the new drug or new device to a standard therapy or a, a 7 drug case may be a placebo pill or a current active pill, 8 we would compare the results in, in exactly the same way. 9 We would use our individual cells, in this case 10 would be hospitals, hospitals, for each hospital. If we 11 did a multi-center study of many hospitals, and I've been 12 involved in studies that include, you know, fifty or a 13 hundred hospitals, each hospital contributes a, some number 14 of cases to the study, maybe not very many, just like a 15 cell contributes some number of cases. And we would look 16 at, in each hospital, the number of individuals in the, 17 that took part in the study, that received, say, the 18 standard treatment and seeing what happened to them, 19 whether or not they were cured or had some complication or 20 something like that. 21 And we'd also look at the number with a new 22 device, the number that took part in that particular 23 hospital, and the number that were, say, cured, or had a 24 complication. And many times, many times these studies 25 result in a number of fairly small cells with not very 68 1/17/01 - BENCH TRIAL - VOLUME II 1 large numbers, very similar to what we have here. 2 And we put together, using this, the technique I'm 3 using here is a technique we use to get a composite 4 relative odds of how useful the new device or the new drug 5 is compared to the standard. And we, we often summarize 6 that in terms of relative odds. 7 Q. Can you give some examples of what are, in, say, in 8 the field of working with medical devices or new medicines, 9 what kind of odds are deemed significant? What kind of 10 relative odds are deemed significant in those fields? 11 A. Well, I mean, if I were designing a study, if I were 12 designing a study, we'd often make, you know, some kind of 13 estimate of what was an important relative odds with 14 respect to patients. 15 And depending on the area, relative odds, I'd 16 certainly design a study where we're looking for a relative 17 odds of say, two, or one and a half. Or, I read about Dick 18 Cheney's angioplasty, using a stint. And it's an area I 19 work in, actually, coronary stints, and there the, the 20 relative odds of heart disease versus no heart disease for 21 someone, say, receiving Aspirin is about 1.3, 1.4. 22 So relative odds that might be small numbers 23 greater than one are common. Odds -- I can think of a 24 medical example I had once, if I might continue, where we 25 were looking at historical data of people who were at very 69 1/17/01 - BENCH TRIAL - VOLUME II 1 low blood pressure, very low cholesterol, and their chance 2 of a heart attack, or heart disease, showing itself, to 3 those with high cholesterol and high blood pressure. And I 4 calculated relative odds in a case like that, the actual 5 observed relative odds was, well, if I remember right, 6 about nineteen. We used that as an example in my classes, 7 because that's such a big number. 8 Nineteen's an enormous relative odds, and so that 9 would. To me, it gave me advice on how to handle my diet, 10 of course, and things like that, but it was some of the, 11 some of the odds we're seeing, if you change relative odds 12 by factors of one and a half, two, twenty would be 13 incredibly large value in medical studies I've dealt with. 14 Q. Well, in the relative odds analysis that you performed 15 in this case with respect to the University of Michigan Law 16 School, can you summarize what your conclusions and 17 findings were for comparisons to the different racial 18 groups? 19 A. This slide 30 is a result in my statistical analysis, 20 using logistic regression, using the technique that I would 21 use in standard medical studies to, to compute composite 22 relative odds or odds ratios. And when you're comparing, 23 when you're comparing two probabilities, you have, you 24 compare one probability one odds to another. And you have 25 to have a baseline of comparison. And so I chose as my 70 1/17/01 - BENCH TRIAL - VOLUME II 1 baseline for comparison, it's typical of the analysis we 2 do, we choose, say, the largest, the largest group as a 3 baseline. 4 So what I did is largest group which is Caucasian 5 Americans I use as a baseline, and they would be given a 6 relative odds to themselves of one. That's by definition. 7 So the one that's given here is just our baseline for 8 comparison. And what I've done is computed the composite, 9 estimated, the composite relative odds for Native Americans 10 to Caucasian Americans as about 61. And African-Americans, 11 the estimated relative odds, this is looking at the 12 relative odds of acceptance, a composite measure across all 13 these grid cells. So we're controlling for, we're looking 14 at combining the information from each combination of GPAs 15 and LSAT. 16 So putting those together for African-Americans, 17 the number is 257. For Mexican American, the estimated 18 relative odds is 81. Other Hispanic Americans it's 1.03. 19 One corresponds to basically a relative odds, 20 which is similar probabilities. One is what you get for a 21 relative odds that are, that they're basically the same. 22 1.35 is the number for Asian Pacific Island Americans, 37 23 for Puerto Rican Americans. 0.5, less than one seems to 24 indicate that foreign applicants, although it's not a very 25 big amount less than one, would be, have a smaller chance, 71 1/17/01 - BENCH TRIAL - VOLUME II 1 compared to caucasian Americans. If the relative odds is 2 below one, then it would indicate that there would be a 3 smaller chance compared to Caucasian Americans. 4 And those unknown the estimated relative odds is 5 1.18. So what I see here in this, in this example, 1995, I 6 see that four of these values are very, very large, very 7 large by all my experience. And it just, how do I say 8 this. These are in non-technical terms, enormous values. 9 There's four that are very, very large. Those for Native 10 Americans, African-Americans, Mexican Americans and Puerto 11 Rican Americans. 12 And so given the same credentials, given the same 13 credentials, that is the same GPAs, LSAT cell, it, there is 14 a tremendous advantage, or allowance made. I'm reporting 15 just what the admissions committees have done with respect 16 to admission. There's a tremendous advantage given to 17 Native American, African-American, Mexican American, Puerto 18 Rican American applicants, compared to, well, Caucasian 19 American, other Hispanic, Asian Pacific Island Americans 20 and foreign and unknown ethnicity. 21 Q. Dr. Larntz, can you give us, can you state your 22 opinion as to how the size of these relative odds compares 23 to generally other kinds of odds you've seen in the kinds 24 of work you've done, either as a consultant or as an 25 academic in your thirty years or more as a statistician? 72 1/17/01 - BENCH TRIAL - VOLUME II 1 A. These numbers are just big, I mean they're giant. I 2 can't recall any particular examples of any data set of any 3 size that, where I found relative odds of this magnitude. 4 They're large. 5 Q. Okay. Incidentally, is the data that, size that 6 you're working with here, would you consider it a large 7 sample, small sample, how, for each of these years, and 8 then for the years overall? 9 A. I should let my computer answer that question. We 10 have a lot of data. We have a large number of applicants. 11 The amount of comparative information we have, although the 12 cells themselves are small, the amount of comparative 13 information we have, for instance, for African-American 14 applicants, in particular, is considered a large data cell. 15 Q. Now, you report on slide 30, to the right there, some 16 information on standard deviations. Can you tell us what 17 the significance of those values are here? 18 A. Well, as I said, I tried to explain earlier, we want 19 to see, there's always a chance of variation in the world. 20 That's life. I said it earlier. I'd be out of business as 21 a statistician if there weren't chance of variation, if 22 everybody responded exactly the same. So I like chance 23 variation, okay. It gives me a livelihood. 24 And now what we want to do is say, these relative 25 odds that we see here, if we redid them for a different 73 1/17/01 - BENCH TRIAL - VOLUME II 1 year, or we redid them with the new group, they would 2 change. There's no question about that. We'd get 3 different numbers. That's -- there's variation. And we 4 wouldn't expect them to be the same. And the question is, 5 is there evidence that these are bigger than the baseline 6 of one. Are they really bigger. 7 And so we can do a test to compare what's the, 8 what's the likelihood, what's the chance that we would get 9 odds this big if, in fact, chance alone were operating, if 10 chance alone were operating. So could we get numbers like 11 this. Sure, I mean, you can always get any number. Now, 12 how likely is it that chance alone is operating. 13 So, what I did in the standard deviation column is 14 summarize the degree of statistical evidence in terms of 15 standard deviations, so we're thinking, something 16 statistically significant, if it's two or three, or larger 17 in magnitude, and so what I did is summarize the degree of 18 statistical evidence that, concerning each of the relative 19 odds and how likely it is that chance alone was creating 20 this. 21 Q. And what were your conclusions? 22 A. Well, it's clear there are, in this particular slide, 23 there are four values that are bigger than the usual two or 24 three, and we're talking about. There are four, and those 25 are for Native American, African-American, Mexican American 74 1/17/01 - BENCH TRIAL - VOLUME II 1 and Puerto Rican applicants. And so that's, in fact, the 2 smallest one of those is 4.95, the largest is fourteen. 3 Those are, again, sort of strong, strong indications that 4 it's not just chance alone that's, that's causing these, 5 this difference in relative odds. 6 Q. Now, again, did you perform the same kinds of analysis 7 here of relative odds for acceptance controlling for GPAs 8 and LSAT grid cell for the later years, 1996 to 2000? 9 A. Yes. 10 Q. And we don't have those here. 11 A. I think we do actually have the '96, '97, '98. 12 THE COURT: Let me just ask you a question. In 13 terms of relative odds, can that be translated into how 14 many times more likely it is that one group will be 15 admitted compared to another group? 16 A. It's directly in terms of odds, Your Honor. So if, in 17 fact, so, if -- let's just take the example that we had. 18 So if, say -- 19 THE COURT: Using the figures that are on the 20 board. 21 A. Okay. Okay. Sure. Can I use the Mexican American 22 figure? 23 THE COURT: Whichever one you want, just. 24 A. Well, that number there, you see what number that is? 25 THE COURT: Sure. 75 1/17/01 - BENCH TRIAL - VOLUME II 1 A. It's 81. And we've worked with 81 already. So I can, 2 I can do it in that terms, and I think it makes it easier 3 for me to explain. So, for instance, if, if we, if a 4 Caucasian American applicant in a particular combination of 5 GPAs and LSA had a ten percent chance, ten percent chance 6 of being admitted, then the odds, relative odds of 81 for 7 Mexican American applicants would take Mexican American 8 applicants in the same combination of LSAT and grade point 9 average. They would be boosted from ten percent chance to 10 90 percent chance. Okay. Okay. And so that's -- 11 Q. Excuse me. That's where the 81 comes in? 12 A. Numbers that are smaller than that would have a 13 smaller change, but, you know, still, I mean, numbers, 14 changing odds by tens, I consider that pretty big, more 15 than pretty, big, that's, that's enormous. 16 THE COURT: So when you get to a figure of the 17 African-Americans up there, it would be -- 18 A. Two fifty seven is such they would change, you know, 19 in the same kind of context. And I'm just going to be, I'm 20 not going to be exactly precise, because I don't want to 21 take out my calculator but that would change probably a six 22 or seven percent chance to 93, 94. 23 THE COURT: And you think that that really 24 translates in the real world, is that, these figures? 25 A. These are a summary of the information from the 76 1/17/01 - BENCH TRIAL - VOLUME II 1 admission grids. So these are, these are. 2 THE COURT: Is this the real world? 3 A. Well -- 4 THE COURT: Assuming the grids are correct? 5 A. It's from the database. These are, these are not 6 anything, other than me trying to summarize the information 7 in what I consider an appropriate statistical way for the 8 advantage given to what are terms selected minority groups 9 compared to other ethnic groups. 10 THE COURT: Okay. Thank you. You may proceed. 11 Q. Is there a way, Dr. Larntz, that it would be helpful 12 to illustrate how one would illustrate the change from, 13 say, having a ten percent chance of probability to, to a 90 14 percent chance of probability in terms of relative odds? 15 Is there way to illustrate that with a drawing, or, or 16 would that be helpful? 17 A. Sure. I could do it if you want me to. I mean, 18 again, if you think it would be useful to reiterate the, 19 the point. I could probably do it with a drawing, 20 something I do when I teach. 21 Q. Why don't you just briefly do that, if you could. 22 A. Okay. 23 Q. Give us a physical illustration of what it means to 24 have relative odds of 81, for example? 25 A. Okay. I'll do that and use the flip chart. 77 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: Why don't we put the flip chart over 2 here and then everybody. 3 MR. KOLBO: That's fine. 4 A. And this is an example I use in class, just to, it 5 sometimes helps to have physical models. And in class when 6 I'm teaching, and I'm retired from teaching, but I enjoy 7 teaching, believe it or not. 8 THE COURT: I can tell. 9 A. Thank you. I appreciate that. So what I would do in 10 class with my students is I would actually bring in M & M's 11 in the class. That's actually quite effective. It gets 12 their attention and we use the demonstration, and then they 13 get to eat the results. How's that, fair enough. 14 THE COURT: Yeah. 15 A. And so they, they enjoyed this. And this is, this is 16 an actual illustration that I used in class. Now, is this 17 one, this is red. Okay. So, for instance, let me get the 18 black. I would demonstrate this particular case. And it's 19 just a demonstration of probabilities and relative odds, 20 which I think you probably understand. But I'll just 21 demonstrate it. I've got, say a big jar. And in class I 22 would have an actual jar I'd put M & M's in. 23 And I would say ten percent, ten percent chance of 24 admission, if I were using that in this example. Ten 25 percent chance would be, I would have in the, in the jar an 78 1/17/01 - BENCH TRIAL - VOLUME II 1 M & M. One, fair enough. And what's ten percent. 2 Let's see, I got to make sure I do this right, 3 one, two, three, four, five, six, seven, eight, nine, if I 4 counted right, okay. Then what we would do is we would 5 take this jar, we'd mix them up and say, well, a ten 6 percent chance which is, is, is a chance of drawing in this 7 jar and picking out an M & M. And if it's green, then the 8 event occurred, say admission. If it's red, then the 9 student was denied. Fair enough. And so, so we do that. 10 Now, we want to compare that to a relative odds of 11 81, which corresponds to 90 percent which, as you know 12 already. What does that correspond to? How many M & M's 13 do I have to dump in to make that, make that the relative 14 odds of 81. Well, I've already got one in here, right. So 15 in order to get a relative odds of 81, what I have to do is 16 I have to dump in -- well, I've got to dump in 80 more. 17 And so what I would do then in class, I would take 18 the 80 M & M's, I prepared then, dumped them in and then 19 show them. That's the increase in relative odds by going 20 from one to 81, going from ten percent to 90 percent. And 21 I won't draw 80 here, but the idea is I wind up with many, 22 many, whatever in this thing, and think of, think the jar 23 now containing 81 green ones and nine red ones. That 24 actually corresponds to 90 percent, 81 out of ninety. So, 25 so that's a demonstration of how I would do it in teaching, 79 1/17/01 - BENCH TRIAL - VOLUME II 1 to get people to think about the physical, the physical 2 value of relative odds of 81. 3 THE COURT: Okay. 4 A. Okay. 5 Q. I think Dr. Larntz reminded me that you do have 6 analysis power point recitation as well as in your report? 7 for later years of relative odds acceptance? 8 A. Yes. That's given for, this particular analysis is 9 given for 1996, is the next slide. 10 Q. Slide 31? 11 A. Yes, and '97, and '98, and '99 and 2000. 12 Q. Okay. And again, the numbers change somewhat, but can 13 you see what your overall conclusions are with respect to 14 the relative odds analysis that you did controlling for GPA 15 and LSAT over the course of these, of this six-year period? 16 A. Yes. The numbers do change. And that's what I would 17 expect. It's not the same set of applicants every year. 18 But, in fact, for all four years the, the relative odds for 19 Native American, African-America, Mexican American, Puerto 20 Rican Americans, are all large, and given, very large, 21 compared to those of, in particular, other large groups, 22 Caucasian Americans and Asian Pacific Island Americans. 23 Q. Let's go to the next slide. What was the next step in 24 your analysis then, your next method of analysis then, Dr. 25 Larntz? 80 1/17/01 - BENCH TRIAL - VOLUME II 1 A. Well, in doing a statistical analysis, that, that was 2 a summary and provided a summary, looking at the 3 information of grade point average and LSAT. 4 Now, I do know and, that there are other factors 5 that the law school says in their policy that they would 6 consider using, and so I want to do an analysis that 7 accounted for those other factors in some way. 8 In particular in statistics, if you can find other 9 factors that change or, again, this is not a statistical 10 term, but washes away, takes away the effect, you want to 11 know if there are other factors that explain the effect. 12 And so, and in the case I have here, I want to look at 13 factors that would, that were in the database. 14 I have to say I have to work with, as best as I 15 can, with objective information that's in the database and 16 I, I chose some of the those factors for a further 17 analysis. 18 Q. Dr. Larntz, could you purport to study all of the 19 factors you thought might be considered in the admissions 20 decision-making process? 21 A. I surely didn't do use all the factors, no. I mean, I 22 didn't have the quantitative information on all of the 23 factors. And I chose the ones I thought that I could get 24 reliable information on and had, had in the data. I'm sure 25 there are other factors. I mean, I don't perfectly explain 81 1/17/01 - BENCH TRIAL - VOLUME II 1 the admissions process. The probabilities that we get are 2 not a hundred percent or zero percent. So I, there's 3 surely other factors, yes. 4 Q. How was it, we've seen some focus here on GPAs and 5 LSAT scores. How was it you came to part of your analysis, 6 at least, on the study of those factors in your relative 7 odds analysis? 8 A. GPAs and LSAT? 9 Q. Yes. 10 A. Well, it was clear from the admissions policies that I 11 looked at that those were credentials that were considered 12 very important, and useful for each, in making a decision. 13 It was clear from looking at the data itself. It was clear 14 from looking at the data itself that four all ethnic 15 groups, that the higher you were, an individual was, with 16 respect to GPAs and LSAT that the better their chance of 17 admission. So LSAT and GPAs were stated criteria, and 18 there's strong statistical evidence that, in fact, both 19 LSAT and GPAs were used in, as a part of the admissions 20 decisions for all ethnic groups. 21 Q. Did you make an assumption that some factors, other 22 than LSAT scores and GPAs were factors in the admissions 23 decision-making process and that could be dispositive 24 factors in the admissions process? 25 A. You mean, were there other factors beyond? 82 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Did you assume that there were? 2 A. Well, I mean, I don't, I don't know that I assumed 3 there were. I presumed that that might be different, 4 statistically. I didn't assume there were other factors, 5 but I assume, I presume there probably were, yes. 6 Q. Well, if we could take a look at slide 36, can you 7 summarize what information you sought to present there? 8 A. Well, I found, or constructed variables on other 9 factors I thought might be somewhat explanatory with 10 respect to the admissions process. The first one I looked 11 at is Michigan residency, and that's clearly stated in the 12 admissions policy, that Michigan residents should be given 13 preference over other non-residents. And that's clearly 14 stated, and so I wanted to include Michigan residency, 15 control for it in the statistical sense, that is included 16 in the analysis. 17 Now, I don't believe I saw anyplace in the policy 18 where they said that they would use gender as another 19 factor, but my experience in other aspects of doing 20 litigation is that gender sometimes is a factor, whether 21 it's explicit or implicit. And so I decided I would 22 include gender, and that's certainly in, if I might say an 23 aside, in a non-statistical expert role, in my own 24 university, gender was an important factor in some of the 25 decisions that were made with respect to admissions. 83 1/17/01 - BENCH TRIAL - VOLUME II 1 I also was looking in some sense for another 2 factor that would somehow indicate something about the, 3 what do I want to say, economic status of the individuals. 4 And there was a variable in the, in the variable in the 5 database called fee waiver. And I have to admit that I 6 have no justification, other than as title for using that 7 variable. But I thought that I would, individuals that had 8 received a fee waiver in the admissions process, I thought 9 that there might be some, there might be some explanatory 10 natures, if they took advantage of, if they gave any kind 11 of preference to individuals that, that received fee 12 waivers. 13 Also, if you recall the cell grids, they had, 14 well, the LSAT grid was like 161 to 163, and the GPAs was 15 3.25 to 3.49. I mean, their boundaries, that's how Exhibit 16 16 was constructed, and, you know, I said to myself, I have 17 to say, what if one someone were at the upper levels of 18 those, if they were at the higher end of the LSAT and the 19 higher end of grade point, would they make have a better 20 chance than someone at the lower boundaries of those. 21 And so what I did, I included within, within cell 22 GPAs and within cell LSAT, that's, you know, where 23 individuals were relatively within the cells. And I 24 included that as a possible explanatory factor in, in doing 25 that in doing a logistical regression analysis. 84 1/17/01 - BENCH TRIAL - VOLUME II 1 (Whereupon an off-the-record 2 discussion was had.) 3 Q. I think, Dr. Larntz, you were now talking about within 4 cell GPAs and why you did that analysis. 5 A. I think I completed the explanation that I thought in 6 deposition within the grid cell, you know, with the upper 7 boundaries they probably had a slight advantage over 8 individuals that were at the lower boundaries. And that's 9 why I included those. Those are the fine expert factors 10 that I wanted to allow for, allow for in doing the logistic 11 regression analysis. 12 Q. And what conclusions did you draw from your analysis? 13 A. Well, you can see the results here, this is from 1995. 14 Again, with respect to what we're looking at now is the 15 effect of ethnic group, allowing for these other factors, 16 so allowing for whether or not a person was a resident, 17 allowing for whether or not an individual, a female or 18 male, whether they'd received a fee waiver, and whether 19 their relative position was within the cells. What we can 20 see, actually I think it might be instructive just to see 21 what these other effects were. 22 The relative odds for Michigan residency that we 23 estimated was 6.5. That's big. Okay. That's a big 24 relative odds. That's to say that Michigan residents at 25 the same level of these other factors are given an 85 1/17/01 - BENCH TRIAL - VOLUME II 1 advantage. And so that's, and you can see the statistical 2 significance, the standard deviations correspond to ten. 3 So, in fact, the data showed that residency is a big 4 factor. That is a relative odds of 6.5. 5 If we look at being female, in fact, the relative 6 odds there is 1.9. Now, that's -- that's not giant 7 compared to the numbers we're seeing, but that's of the 8 same size I'm used to seeing as far as a factor that would 9 be considered an important factor. 10 So, in fact, it looks to be, at least for 1995, 11 and the statistical significance backs that up, standard 12 deviations apply, a relative odds of 1.9 indicates that, at 13 least the admissions decisions that were made in 1995 for 14 females seemed to be favoring females. 15 And just to say, just to make sure, Your Honor, a 16 relative odds of two would change, say a 50 percent chance 17 to a 66 percent chance. Just, that's a relative odds of 18 two. We go from fifty to 66, actually 66 and two thirds, 19 right. So that's what it would do. 20 For fee waiver, the relative odds turned out to be 21 very close to one and not statistically significant, which 22 indicated that, at least the variable I chose, whatever it 23 indicated, it didn't seem to be taken into account with 24 respect to the admissions statistics made. 25 And the, within cell GPAs and LSAT, these are 86 1/17/01 - BENCH TRIAL - VOLUME II 1 scaled a little differently, so the numbers look small, but 2 they're both on the order of showing some significance, 3 which is to say there's some advantage to being at the 4 upper corners of these cell grids, compared to the lower 5 corners. That's basically what it's saying. 6 So the extra factors I put in, except for fee 7 waiver were explaining, helping explain the, the admissions 8 decisions. And then once we've done that, we also then can 9 see the corresponding estimated relative odds for the 10 ethnic groups that are included in the analysis. 11 And, again, the ones for Native American, 12 African-American, Mexican American and Puerto Rican 13 Americans are all large. They're all large. And compared 14 to those for Caucasian American -- well, Caucasian 15 Americans automatically won because of the baseline but 16 Asian Pacific Island, 1.56, may be an indication of a 17 slight advantage, but not a lot given these other factors 18 and the other groups. 19 Q. So do I understand correctly that then, Dr. Larntz, 20 here what you've done is, in addition to holding constant 21 in the way that you have, GPAs and LSAT when comparing the 22 relative odds of, say, one of the minority groups to one of 23 the majority groups, what you've also done is controlled or 24 held constant these other factors like residency at the 25 same time you're controlling for GPAs and LSAT. Is that 87 1/17/01 - BENCH TRIAL - VOLUME II 1 fair? 2 A. That's correct. It's what I call a statistical 3 control. Instead of having each individual cell grid, we 4 use a statistical variable to negate that control, but 5 that's what we're doing is, the best we can, to hold 6 constant residency gender, fee waiver status and position 7 within the cell grid. 8 Q. Okay. Can we go to slide 37. What was your next step 9 of the analysis here? 10 A. Well, this, this slide just compares the relative odds 11 for 1995 only for, on the left side we're doing the 12 original analysis, just controlling for GPA position and 13 LSAT grid cell. And on the right we're looking at the 14 controlling, after we've controlled for these additional 15 five factors. 16 So what I did is just compare the estimated 17 relative odds in those two cases, and we can see, you can 18 see the numbers that, the numbers there are big in both and 19 they intend to be big in each. The numbers are big in the 20 original analysis, are big in the analysis adjusting for 21 the other factors. In fact, they look a little bigger. 22 You know, that's, the 80, the 61 became 116. 257 became 23 513. 81 became 183. That's for the Mexican American 24 applicants, and Puerto Rican applicants, 37 became 72. 25 They got larger. 88 1/17/01 - BENCH TRIAL - VOLUME II 1 I don't want, I would not say that that's a big 2 change in the way these things go around. They, you know, 3 once we get up to a high number, there isn't a great big 4 effect, but they all look a little larger in this example. 5 Q. Okay. What is the significance of that, the fact that 6 they are larger? 7 A. Well, if, and I didn't test this per se, but if it 8 were consistently larger, that would say that, that when we 9 take account of these additional factors, we take account 10 of residency, we take account of gender, we take account of 11 fee waiver status, although it didn't have much of an 12 affect when we take a account of position within the cell. 13 When we take an account of these additional factors 14 decisions were made if they were consistently larger for 15 the selected minority groups, say, that would say that as 16 we made the, the, the comparison, you know, finer and 17 finer; that is, we got more and more similar credentials, 18 then it looks like ethnicity is actually a larger factor 19 than it was apparent, just from taking a count of the two 20 the GPAs and LSAT. 21 Q. Now, did you do this -- I don't think we have slides 22 for the later years of this analysis, correct? 23 A. I don't think we do. They're all in the reports, and 24 similar, similar -- the nice thing about doing the 25 computing is that once you've programmed it for one year, 89 1/17/01 - BENCH TRIAL - VOLUME II 1 you can do the other years and we did that. 2 Q. And can you tell us whether the conclusions you drew 3 for the later years for 1996 or 2000, whether they're 4 substantially different from the conclusions you've 5 testified here to this morning with respect to 1995? 6 A. No. The conclusion, the conclusions are the same. 7 There's, they, the allowance I see, for, if I call it 8 selected minority status, which is one of the, the Exhibit 9 16 categories for Native American, African-American, 10 Mexican American, Puerto Rican applicants, there's a very, 11 very large allowance given with respect to admissions 12 decisions for individuals with similar credentials. 13 Q. Let's go to the next slide then. Can you tell us what 14 slide -- I can't read it, 38, represents. 15 A. In the admissions policy and in the database there is 16 a reference to an index, which is -- well, that's what it 17 is for 1995. It's an equation. And the, the index, or, I 18 guess I refer to it as a selection index. But the index is 19 a summary measure that, of GPAs, undergraduate GPA and 20 LSAT. So it's a summary measure that is in the database. 21 It's interesting. I guess I wasn't, I wasn't too 22 clever in all my work. I, this formula that we have here, 23 I may have been given that materials, but I didn't find it 24 in the materials, and so I used the database itself to 25 derive this formula and then was quite gratified when the 90 1/17/01 - BENCH TRIAL - VOLUME II 1 interrogatory answer came back and said this is the formula 2 for the index. So I shouldn't be proud of things, but I 3 was kind of proud that I, at least agreed on the index 4 formula. 5 But what the index formula does, if you look at 6 it, it's just an equation, and -- just an equation, he 7 says. Well, he's a statistician. That's why he says just 8 an equation. 9 It's an equation, and the equation, the 10 co-efficients for the equation, the co-efficients for the 11 equation are the key for understanding the equation. The 12 co-efficient for GPAs is .324. So that means this index 13 value goes up .324 for every grade point average up. So 14 someone that had a grade point average of 2.5 compared to 15 someone that had a grade point average of 3.5, their index 16 score would be .324 higher, case. 17 And similarly for LSAT, the co-efficient, it's a 18 decimal, right. It's 0.0320. This decimal says for every 19 point of GPA, then this index score goes up by .03. That 20 doesn't sound, looks a lot, but, in fact, ten points it 21 goes up ten times that, all right. 22 Q. Dr. Larntz, you said GPA. Did you mean LSAT? 23 A. Did I? I'm sorry. I misspoke. I'm surprised I 24 didn't do it earlier. Okay. So with respect to LSAT, ten 25 points of LSAT, an increase of ten points of LSAT would 91 1/17/01 - BENCH TRIAL - VOLUME II 1 increase about the same amount, .30, ten points of LSAT and 2 one point of GPA would each increase this about .32. 3 That's what the formula, that's what the formula does, and 4 so individuals are scored on this, and it's, it was in the 5 database. 6 Q. All right. Apart from the relative odds, the 7 different relative odds analysis you did, did you do some 8 additional kinds of comparative statistical analysis with 9 respect to the issues you were looking at in this case? 10 A. With respect to selection index, I wanted to see, and 11 I, I assumed, and I assumed from admissions documents that 12 it was intended that individuals with large, higher index 13 scores would have a higher probability of admission. And 14 what I did is I tried to calculate what the probability of 15 acceptance for an individual would be as a function of; 16 that is, as index score changed. 17 So what I tried to do, what I did was construct 18 graphical display that showed an estimated probability 19 using a statistical technique that would allow me to get a, 20 call it the maximum likelihood estimate for the probability 21 function of admission versus selection index. What I did, 22 it's just a curve, it's actually a step-function curve that 23 shows us what the probability of admission is versus 24 selection index for each ethnic group. 25 Q. Can we see the next slide, Dwayne. 92 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: Why don't we take a break here. 2 (Whereupon a recess is had.) 3 MS. MASSIE: I've talked to both Mr. Payton and 4 Mr. Kolbo about this. We're going to file a response to 5 the plaintiff's motions in limine, hopefully this 6 afternoon, but maybe tomorrow morning. 7 THE COURT: That's fine. 8 MS. MASSIE: And if we could maybe argue it 9 Friday, we were all thinking -- 10 THE COURT: That's fine. I just, you know, didn't 11 want to rush you. I know you're in trial. As soon as you 12 can get it, you know get it to us. Friday is fine. 13 MS. MASSIE: Great. Thanks. 14 MR. KOLBO: Before we jump -- 15 THE COURT: Okay. Can we do the lights? 16 MR. KOLBO: Before we jump back into the 17 presentation, I want to cover a couple of housekeeping 18 matters, too. At this time I would offer Exhibit 16, which 19 was the 1995 law school grid. 20 THE COURT: I think. Didn't you already? 21 MR. KOLBO: I think fifteen. Fifteen was -- 22 they're very similar. Sixteen today. 23 THE COURT: Any objection? 24 MR. WASHINGTON: No. 25 MR. DELERY: No, Your Honor. 93 1/17/01 - BENCH TRIAL - VOLUME II 1 THE COURT: Received. 2 Q. And before getting back to the power point 3 presentation, Dr. Larntz, apart from the work that you did 4 with respect to the reports that you generated, was there 5 anything else that you were asked to do in connection with 6 your work on this case, that's not reflected in the actual 7 reports themselves? 8 A. Oh, right. I was, there was, I was requested by 9 attorneys to make a selection of cases, I believe, 10 applicant cases in order to request cases from the 11 admissions files, so I, I did that work. 12 Q. If I could have the witness shown Exhibit 120 through 13 125, and if I could explain briefly. 14 MR. KOLBO: Your Honor, I've had a conversation 15 with Mr. Payton about this, or Mr. Goldblatt, I think this 16 morning, and Mr. Delery. Exhibits 120 to 125 have been, 17 basically, reserved in the event that we decide later on to 18 produce to the Court a number of application files that 19 were requested in the course of discovery in this case. 20 There were, approximately, a hundred files for each year, 21 from nineteen, I think 19 -- I'm not sure of the first, 22 perhaps 1995 all the way to 1997. Those files, we have not 23 decided yet whether to offer them. They're not actually in 24 the exhibit books yet. 25 THE COURT: Well, I see. I'm looking. Okay. Go 94 1/17/01 - BENCH TRIAL - VOLUME II 1 on had. 2 MR. KOLBO: I'm going to ask Dr. Larntz to 3 authenticate the documents that were used to select those 4 files in the event. 5 THE COURT: In the event you need it. Okay. 6 MR. KOLBO: Exactly. 7 THE COURT: Just to make a record at this point. 8 MR. KOLBO: Exactly. 9 THE COURT: Fine. 10 MR. KOLBO: My understanding is there's no 11 objection to that. 12 Q. So Dr. Larntz, could you describe what Exhibits 120 13 through 125 or the portions that you have of those 14 exhibits? 15 A. They're lists of numbers. And they, I actually can't 16 verify that these are numbers I selected because I don't 17 remember the numbers of the cases. But they're lists of 18 numbers that are of the form that I gave to indicate random 19 selection, selections I made from accepted minority 20 applicant's files and non-selected majority applicant 21 files. 22 Q. Okay. And did you provide that list to me? 23 A. I provided lists, again, I can't tell you that these 24 are the exact numbers, but I provided lists like this to 25 you. 95 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Provided a list like that to me? 2 A. Yes. 3 Q. And what do the numbers represent, as far as you know? 4 A. The numbers represented are the ID numbers. I think 5 they're substitute Social Security ID numbers that were in 6 the database for applicant files. 7 Q. They're fictional files that correspond with an 8 applicant? 9 A. That's my understanding, yes. 10 Q. And how did you go about, first of all, deciding to 11 select a hundred files for each of the years? How did you 12 decide upon that number? 13 A. Well, I guess it was a mixture of how many should we 14 look at and how many, how many should we go for. And 15 statistically, I probably said that you needed a 16 representative number in each year. I don't think I did an 17 exact calculation. I think it was a practical number of 18 what would be a reasonable number to select. 19 Q. And how were the -- can you tell me something about 20 the manner in which these hundred files were selected? 21 A. Yes. I can tell you exactly how I did it. In the 22 sense I took a random sample of, of selected minority 23 applicants that, who were accepted. And then I matched, in 24 the sense of chose a candidate from Caucasian or Asian 25 Pacific Island Americans with the same grid cell of LSAT 96 1/17/01 - BENCH TRIAL - VOLUME II 1 and GPA who was not accepted, if that was possible to do. 2 Q. So a hundred files all together, fifty majority 3 students, or fifty students from selected minority groups 4 and fifty students from other racial groups, is that right? 5 A. Yes, fifty, fifty each. 6 Q. Okay. And you do the same thing for each of those 7 years? 8 A. I did it actually for all six years, 1995 through 9 2000. 10 MR. KOLBO: Okay? I would offer then, Your Honor, 11 that portion of Exhibits 120 to 125 that Dr. Larntz has in 12 front of him. 13 THE COURT: Any objection for the limited purpose, 14 should they desire to get those in at a later time? 15 MR. DELERY: No, Your Honor. The exhibit list and 16 the pre-trial order includes a note about how the parties 17 have agreed. Will proceed with the files if we get that 18 far. And subject to that we have no objection. 19 THE COURT: I suspect the same. 20 MS. MASSIE: Neither do I. 21 THE COURT: Very well. 22 Q. Okay. Dr. Larntz. Let's continue. I think you were 23 on slide 39 of your presentation. And I don't think we got 24 to the point of you describing exactly what this, what this 25 kind of a graphic represents, if you could do that with 97 1/17/01 - BENCH TRIAL - VOLUME II 1 using the example of slide 39. 2 A. Yes. If you recall, we put up the formula for a 3 selection index. That was the previous slide. And what 4 we, for 1995. And what we've done here is, on the 5 horizontal axis, we have selection index. On the vertical 6 axis we have probability of acceptance. 7 And what I've done is for each ethnic group, I 8 estimated a, the probability of acceptance as a function of 9 selection index. So as we see down here at the, at the 10 selection index scores that are around two, we'll see that 11 there, that the probability is right at zero. That means 12 individuals with selection indicis that low, actually quite 13 less than 2.3, none of those individuals were admitted, so 14 that that probability of acceptance is zero. 15 The curve, I'm tracing the curve with the asterisk 16 on it. The curve that's estimated, and I say curve, it's a 17 step function. But that actually estimates the probability 18 of acceptance for individuals with the corresponding 19 selection index value, so, for instance, up at the top 20 then, in the curve with the asterisk for selection index 21 values, if I can see it. It's hard to tell. I think 22 that's a value, selection index of about three there. 23 And then from that point on, this is for Native 24 American applicants. Native American applicants with 25 selection indicis; that is a combination of LSAT and GPA 98 1/17/01 - BENCH TRIAL - VOLUME II 1 that corresponded to an index value of three, a hundred 2 percent of those individuals were selected. Now, the 3 values go up, as we see. That's an indication that 4 acceptance probability increases as selection index 5 increases, just as we might expect because selection index 6 is a function of GPA and LSAT. 7 The other curve that's given on the plot, and I've 8 used the same baseline curve. The other curve that's given 9 on the plot, the one without the asterisk, that is the 10 corresponding curve function for Caucasian American 11 applicants. So that's, that's the curve that's to the 12 right. And, and so for Caucasian American applicants, 13 again this angle's hard, but certainly somewhere beyond 3.5 14 or so, then we have a hundred percent selection. And 15 certainly down here, less than selection index of about 16 2.7, 2.8. That's a zero. There's no, no probability. So 17 this curve tries to estimate directly the acceptance 18 probability for each ethnic group. 19 I took the Caucasian American curve on each of 20 these for comparison purposes, so rather than just draw 21 each one and try to compare it, I've drawn the Caucasian 22 American curve on with the Native American curve in this 23 case. 24 And we can see, let's just say if you look at the, 25 at the probability of acceptance of .5, let's just say, you 99 1/17/01 - BENCH TRIAL - VOLUME II 1 could go across and see that that's a value 38. 2 Well, it turns out that if you go down, it's about 3 an index of three for Native American applicants. For 4 Caucasian American applicants it's about 3.2. So this 5 distance here would say that this, this distance here 6 corresponds to the difference, the gaps correspond to the 7 difference in selection index at each probability of 8 acceptance. 9 So this distance here is about, I'd say it's 10 approximately two-tenths of a point in the selection index. 11 And if you recall from what we did last, looked at just 12 before the break, one grade point average is about three 13 tenths of a point or ten LSAT points is about three tenths 14 of a point, so that tells the amount of advantage that's 15 given in terms of, at least those LSAT and GPA points for 16 comparing Native American acceptance probabilities to 17 Caucasian American acceptance probabilities. 18 Q. Can we go to the next slide, please? 19 A. This is the same comparison now done with 20 African-American applicants to Caucasian American 21 applicants. And it has the same structure and the same 22 form. I guess the main substantive difference is that the 23 gap is bigger than it was for the difference between 24 American Indian, for Native American and Caucasian American 25 applicants. And so the gap is bigger, which means that 100 1/17/01 - BENCH TRIAL - VOLUME II 1 with respect to the selection index, we've wound up with 2 similar acceptance probabilities with larger differences in 3 selection index that would correspond to larger differences 4 in GPA and LSAT. 5 You could also look in down here, for instance, 6 say in a value of three, say, you could go across and get 7 the probability, I guess. And again, I can't read it very 8 well, but it looks like it's somewhere between five and ten 9 percent probability for a selection index of three for the 10 chance of Caucasian Americans being accepted. For 11 African-Americans, going up then to three, it looks like 12 the chance is somewhere between 90 and 95 percent. 13 So we actually see our relative odds can be 14 displayed without making any assumptions, at least with 15 respect to selection index. You can see that there's a 16 large gap in the acceptance probability for similar 17 selection index values. 18 Q. When you say you can make, draw some conclusions here 19 without making any assumptions, what do you mean by that? 20 A. Well, these acceptance probability curves, these 21 acceptance probability curves, are computed without making 22 any, any assumption about relative odds, or that there's 23 any, it's just, essentially, a descriptive summary of the, 24 of the, how acceptance probability was a function of 25 selection index for each ethnic group. 101 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. And you did one of these for each one of the ethnic 2 groups for 1995? 3 A. For each ethnic group for 1995 we did the same graph 4 comparing Caucasian American to each other ethnic group. 5 So the next slide, for instance, would be Mexican 6 American applicants. Again we see a gap of the same 7 nature, indicating that for given selection index values, 8 there's a higher probability for Mexican American 9 applicants to be accepted. 10 And the next line I think is other Hispanic 11 American applicants. As, other Hispanic American, just to 12 be clear, all these categories are exactly the categories 13 that were spelled out in the database that was given. So 14 that this categorization was given, we can see -- actually, 15 we see a fair bit of overlap in this one. This one has a 16 fair bit of overlap which indicates that there isn't a 17 great deal of difference in the acceptance probabilities. 18 Q. The database you saw contained a designation for 19 Mexican Americans which we looked at earlier, I think? 20 A. Yes. 21 Q. And then this is a separate one for other Hispanic 22 Americans, and those are designations that you found in the 23 database yourself? 24 A. Those were directly the categories that were in the 25 database in all the years, yes. 102 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. Okay. There was a code, in other words, you were able 2 to identify each applicant by one of those ethnic 3 designations? 4 A. That's right. 5 Q. Okay. Can you tell me what, if one were to apply 6 every combination of LSAT score and, or I should say in the 7 case of these graphics, every selection index for every 8 applicant, would it always be a straight line? Would there 9 be any variation at all along those lines in the positive 10 direction that they're headed? 11 A. Would it always be? Would it always go up and to the 12 right? 13 Q. Yes. 14 A. Well, the technique I used to estimate these, these, 15 the probability function assumes and was estimated under 16 the statistical assumption that it was, that probability of 17 acceptance would be increasing as selection index 18 increases. That's, that's the way the technique was, 19 that's -- this is the best fitting curve under that 20 assumption, and it's clear. There are -- it's clear that 21 for the most part, that's exactly how the decisions were 22 made. 23 Q. There's some variation around that? 24 A. Well, sure, these slides don't go straight out and 25 then straight up. So they aren't using just selection 103 1/17/01 - BENCH TRIAL - VOLUME II 1 index. And in fact, the variation in decisions of course 2 is indicated by the fact that not everyone at a particular 3 value, until you get to a certain point is admitted, and 4 certainly not everyone is denied. So the curves, if only 5 selection index were used, then the curve would go across 6 to one point and go straight up to one and go across, but 7 there's clearly variation, yes. 8 Q. Let's see the next slight then? 9 A. This is the slide comparing Asian Pacific Island 10 Americans to Caucasian Americans, and, well, there's 11 considerable overlap, indicating no particular degree of 12 preference. 13 Q. And slide 44? 14 A. Puerto Rican applicants versus Caucasian American 15 applicants. And again we see considerable separation to 16 indicate that Puerto Rican applicants with the same 17 selection index have higher probabilities of acceptance 18 compared to caucasian applicants with the same selection 19 index value. 20 Q. And I think there's one more slide here? 21 A. I think there are two more, but the next one is for 22 foreign applicants versus Caucasian Americans. This is 23 interesting. Remember the, whatever group we have is with 24 an asterisk. And in this particular case, the asterisk is, 25 at least the top the line of the asterisk is to the right 104 1/17/01 - BENCH TRIAL - VOLUME II 1 instead of to the left of the Caucasian Americans which is 2 an indication of, at least in these higher selection index 3 values, there may be -- it's not that significant, but 4 there may be some preference for Caucasian American 5 applicants compared to foreign applicants. 6 Q. Okay. And one more? 7 A. I think the last one is for unknown. And, well, 8 there's certainly considerable overlap. 9 Q. Okay. Now, again, did you do the same kind of 10 analysis and display the same kind of graphics with respect 11 to the year 1996 through 2000? 12 A. For the years 1996, 1997, 1998, 1999, 2000. There's a 13 plot. There are plots like this for each year, yes. 14 Q. In -- 15 A. In my reports, in their respective reports for the 16 analysis of those years. 17 Q. Can you just summarize whether there are any 18 significant or substantive difference in terms of your 19 overall conclusions, with respect to those years compared 20 to 1995? 21 A. With respect to, looking at these plots, it's clear 22 that in all the years there was considerable preference, 23 and allowance made, that is higher acceptance probability 24 for given selection index values for Native American, 25 African-American, Mexican American and Puerto Rican 105 1/17/01 - BENCH TRIAL - VOLUME II 1 applicants, compared to Caucasian American applicants and 2 Asian Pacific Island American applicants and, in fact, the 3 other categories as well. 4 Q. Okay. And did you do any further statistical analysis 5 with respect to the issues that you looked at? 6 A. Yes. 7 Q. Can we go to the next slide? Can you tell us what the 8 next step in your analysis was, the next method of your 9 analysis? 10 A. This, this grid is actually from my, the supplemental 11 report to my analysis, I mean, original analysis. 12 What we've talked about already is what was 13 contained in my original set of reports. This particular 14 grid was constructed to allow comparison of selected 15 minority applicants. And selected minority applicants 16 consisted of, in Exhibit 16 there was a grid for selected 17 minority applicants. And in selected minority applicants 18 in constructing this, there was four groups. Ethnic groups 19 that were included in that were Native American applicants, 20 African-American applicants, Mexican American applicants, 21 and Puerto Rican applicants. And so this is a comparison 22 of selected minority applicants. This is a grid. 23 But broken down, this time it wasn't in the 24 original grid, but broken down by residency, and so this is 25 actually a 1995 Michigan non-residents. This is the grid 106 1/17/01 - BENCH TRIAL - VOLUME II 1 for Michigan non-residents for selected minority 2 applicants. 3 And the grid was constructed in the same way as 4 the grids for Exhibit 16 was before. There were 474 5 applicants, selected minority applicants, in 1995 Michigan 6 non-residents, 142 of whom were admitted. 7 Q. Can I just ask you, with respect to the categorization 8 of selected minority applicants that you found in the 9 database, selected minority applicants did not include in 10 the law school's categorization of a database Asian 11 Americans? 12 A. That's correct. 13 Q. Did it include the category that's also identified 14 there as, other Hispanic? 15 A. Well, I had the Exhibit 16 for 1995 and that, they 16 were not included in that. In order to replicate, let me 17 just say, in order to replicate the results, it was clear 18 that the four ethnic groups that I named were the ones that 19 corresponded to selected minority applicants. 20 Q. Okay. Any further information from this slide that 21 needs to be summarized? 22 A. Well, they're just in the same way. There's other 23 grids, except now we're looking at non-residents and not 24 just, not combining the two, so for instance, in this we 25 highlighted the cell, just for comparison, 3.5 to 3.74 GPA, 107 1/17/01 - BENCH TRIAL - VOLUME II 1 154, 155. And in this case, selected minority applicants 2 there were eight of eight selected minority applicants, 3 four of whom where admitted. 4 Q. Next slide? 5 A. We did the same thing for comparison purposes. For 6 comparison purposes we did the same thing for non-resident 7 majority and non-selected minority applicants. That's 8 everybody else, essentially. There were 3,008 9 applications, 795 were admitted. And in the cell that we 10 were looking, at the highlighted cell, there were forty 11 applicants, three of whom were admitted. 12 Q. Next slide, Dwayne? 13 A. And this just compares side by side those two grid 14 cells, and so we can bring those up just for comparison 15 purposes. We could compute odds ratios and relative odds 16 for these cells, if we so desire. Okay. 17 Q. And is this the next slide? 18 A. Right. And what we're doing here is in this next 19 slide is we just wanted to highlight other cells that we 20 just, we picked out those. In fact, these two cells that I 21 originally talked about are in my report. And so just to 22 show you other cell combinations. For instance, the 3.5 to 23 3.74 GPA and LSAT of 161 to 163, among selected minority 24 applicants, eleven out of twelve were admitted, compared to 25 other applicants was twelve out of 207. 108 1/17/01 - BENCH TRIAL - VOLUME II 1 And similarly, for the other, for the other cell 2 comparison, I think it's clear, fourteen out of sixteen, in 3 this particular cell. I'm not going to read the 4 designation, compared to one out of 57. And similarly, in 5 this other one, nine out of nine compared to one out of 51. 6 This is, this is a, just highlighting the, if we, no matter 7 where we pick the cells, at least in the range where there 8 are decisions being made, we can do comparisons like this. 9 Q. Next slide, Dwayne? And what does this slide 10 summarize? 11 A. Well, we've seen, we've seen already that residency is 12 a factor in admission, and it's a specified factor. And 13 Michigan residents are supposed to, and do receive 14 preference with respect to law school admission. 15 And so this particular slide actually looks at 16 1995 selected minority non-residents; that is individuals 17 who wouldn't receive the residential preference and 18 compares them to other, that is majority non-selected 19 minority resident applicants to see how they, how resident 20 majority applicants compare to non-resident minority 21 applicants. And so these grids, we can look at that same 22 cell. It was four out of eight for selected minority 23 non-residents. Among residents who are not selected 24 minorities, it turned out in that figure cell to be zero 25 out of twelve accepted. 109 1/17/01 - BENCH TRIAL - VOLUME II 1 Q. All right. Dwayne are there some more examples there, 2 yes. 3 A. Yes. And we've highlighted four other cell 4 comparisons the same cells. And so for instance in this 5 particular cell, which is, there were eleven out of twelve 6 selected minority non-residents accepted compared to seven 7 out of twenty-four majority residents, residents who were 8 accepted. And similarly the comparisons here, sixteen, 9 fourteen out of sixteen versus zero out of eighteen, nine 10 out of nine versus two out of thirteen. 11 Q. And again one could compute if we took the time of 12 that relative odds of these different cells? 13 A. Yes, we could compute the relative odds, yes. 14 Q. Could we go to the next slide. What was the next 15 method of your analysis, Dr. Larntz, in comparing the 16 relative odds? 17 A. Well, in a, in a report that was critical of my 18 original report, and there was one. The, it was mentioned 19 that, that it would be inappropriate to combine the 20 relative odds for the individual cells into a combined or 21 composite relative odds. And so the reason I actually did 22 this, and in that report the categories compared were 23 selected minority and versus majority of non-selected 24 minority, in exactly the same way we're using here. 25 And I was criticized, and the report said that, in 110 1/17/01 - BENCH TRIAL - VOLUME II 1 fact, the relative odds would vary, would change as a 2 function of LSAT and GPA, that, in fact, there would be 3 some shift. And so what I did was I actually went cell by 4 cell in this analysis, and for each cell, each combination, 5 for instance, that cell we were looking at the highlight to 6 begin with, the 3.5 to 3.74, non-residents to 1995 with 7 LSAT of 154 to 155, there were eight minority applicants 8 and four were admitted. 9 There were forty majority applicants, three of 10 whom where admitted. And so I just computed all the odds 11 ratios or relative odds. In fact, the odds ratio there is 12 12.33. That's if we did the math. And so I computed 13 those, because I wanted to see, I wanted to see if there 14 was a pattern, a pattern, that showed that this odds ratio 15 was different as a function of LSAT score and GPA. 16 And what I found when I did in my summaries here, 17 and is that, well, a lot of these are small cells, so we're 18 going to get infinities and other numbers like that because 19 we're dealing with small numbers. But there was no 20 consistent pattern. And so I wanted, I wanted to convince 21 myself that there was no consistent pattern, and there 22 wasn't any consistent pattern. 23 In addition, I have to say, statisticians faced 24 with data like this will immediately jump to do a test of 25 significance to see whether or not there, it could have 111 1/17/01 - BENCH TRIAL - VOLUME II 1 arisen by chance that this discrepancy, even these small 2 cells. I didn't originally do any tests with respect to 3 the individual cells. 4 But I, I, when I was looking at this, I decided, 5 well, why don't I see if there's a statistically 6 significant difference between the admission rates, even in 7 these cells, these small cells. I didn't expect to find a 8 lot, but are there statistically significant differences 9 here. And so I reported the P value, the value that we 10 talked about before. What's the chance that this 11 difference, this odds ratio would have arisen, that one or 12 a bigger one, if chance alone where operating. 13 And in this particular cell would be highlighted, 14 the P value is .01. So that would be considered 15 statistically significant evidence of higher probabilities 16 of acceptance, even for, for minority applicants, even for 17 this cell alone. So this P value summarizes the 18 information for this cell alone, this small cell, and, and, 19 frankly, I was surprised to get a series of significant P 20 values in this particular case because the cells are small. 21 Q. Okay. Can we go to the next slide then? 22 A. And what I did, what we've got here, the slides that 23 we're reporting here is we're reporting all the odds ratios 24 for all the cells that we have. I notice some of them are 25 blank. Some of them are blank down here. And those are 112 1/17/01 - BENCH TRIAL - VOLUME II 1 cases where we have -- well, in this particular one, one 2 minority applicant and one majority applicant, but no one 3 was admitted, so there's no comparative information there. 4 For every cell that has comparative information we 5 compute the odds ratios and the corresponding P values, and 6 there you can see, there are, you know, not everyone is 7 significant, but ones that are statistically significant, 8 at the five percent level of significance, we, I put in 9 bold. So I think we have more pages of these in the slide, 10 don't we, for LSAT, 159 and 160 non-residents for 1995. 11 And we probably have some more slides of these, which show 12 the same, the same kind of patterns, and in the reports we 13 have table after table of this for 1995, 1996, 1997, 1998, 14 1999, 2000 for both residents and non-residents. 15 Q. Can you just summarize what conclusions you drew, 16 based upon the cell by cell analysis that you did for each 17 of these years? 18 A. Well, what I did was I actually just did a very simple 19 thing. And to me it was simple. I just looked at the 20 number of these comparisons that are statistically 21 significant. So I just looked to see. I didn't expect to 22 find very many, but I looked at the number, and for each 23 year it turned out to be -- well, I don't remember the 24 exact numbers, year by year, but between 17 and 22 of the 25 comparisons were actually statistically significant, even 113 1/17/01 - BENCH TRIAL - VOLUME II 1 though we're working with small cells, okay. So I looked 2 at those. And, and then what I did is I looked at the odds 3 ratios associated with those that were statistically 4 significant. 5 And I looked at the odds ratios for each of those 6 that were significant. And it turned out in every year for 7 every comparison that it was statistically significant, if, 8 if the decisions were made, if they were being made so that 9 there were, approximately equal probabilities of acceptance 10 we'd exceed, about half of the odds ratio would be small, 11 less than one, about half would be greater than one. But 12 in this case every single cell, every single cell that 13 showed statistical significance, every cell-by-cell 14 comparison that showed significance showed preference for 15 minority over majority applicants. 16 Q. Is that the last slide there? 17 A. I think we might have more of the same. 18 Q. This is just illustrative for the, for 1995? 19 A. Yes. And as I said, these are done for every other 20 year, and every combination of residents and non-residents. 21 Q. From all of the analysis that you've done, Dr. Larntz, 22 and you've shown us examples of here this morning, and 23 based on all the information that you looked at, in 24 connection with the work that you were asked to do, could 25 you provide us your overall conclusions and summary with 114 1/17/01 - BENCH TRIAL - VOLUME II 1 respect to what your findings were? 2 A. Yes. What I've concluded in looking at all of the 3 analysis I've done, and looking at applicants with similar 4 credentials is, that is GPA and LSAT scores that are 5 approximately the same, residents, or not. 6 What I found is that there is, how do I say this 7 in a non-statistical way; there's an incredibly large 8 allowance given to selected minority applicants, 9 particularly African-American, Native American, Mexican 10 American and Puerto Rican applicants are given an 11 incredibly large allowance when it comes to admissions 12 decisions for individuals with similar credentials compared 13 to, in particular, Caucasian Americans and Asian Pacific 14 Island Americans. So there's a very, very large preference 15 given. 16 And to be honest, you can see that from the 17 original grids. I mean, the original grids show that 18 preference if you look at the cell by cell. You can look 19 individually at the grids and see exactly the same effect. 20 My statistical analysis basically has tried to quantify 21 what was in those grids, but, in fact, the original grids 22 show that exactly the same conclusion. 23 MR. KOLBO: I have nothing further, Your Honor. I 24 would offer Exhibit 143 at this time. 25 THE COURT: Any objections? 115 1/17/01 - BENCH TRIAL - VOLUME II 1 MR. DELERY: What is exhibit 143? 2 MR. KOLBO: That's the policy. 3 MR. DELERY: No objection. 4 THE COURT: Received. Why don't we break for 5 lunch and then we'll give you an opportunity to wait. I 6 will be happy to go now. You know how you're all excited 7 to start your cross examination. I would be if I were you, 8 but let's break for lunch and then we'll reconvene at, by 9 1:20, 1:30. 10 (Whereupon a recess is had.) 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 116 1 _ _ _ 2 (Court back in session.) 3 THE COURT: We've got everybody back, 4 we waiting for anybody. 5 MR. DELERY: I don't think so, your 6 Honor. 7 THE COURT: Doctor, you can take the 8 stand if you would like to. You've obviously been 9 in court before, you know the next thing is 10 cross-examination. Not always as easy as direct. 11 A. Yes, sir. 12 13 CROSS-EXAMINATION 14 BY MR. DELERY: 15 Q. Good afternoon, Dr. Larntz? 16 A. Good afternoon. 17 Q. We have met before, isn't that right? 18 A. That's correct. 19 Q. I took your deposition in this case back in February 20 of 1999? A long time ago? 21 A. I'm sure that's right. 22 Q. You said this morning, I believe that the purpose of 23 your analysis was to look at the role that race 24 plays in admission to the Law School, is that a fair 25 statement of your approach? GRUTTER -vs- BOLLINGER, ET AL 117 1 A. Role in a statistical sense, yes. 2 Q. In your view, is that the same as extent to which 3 race is considered in admission? 4 A. Well, I'm not sure I understand an awful lot of 5 difference. What I'm saying is that what I did was 6 try to understand and describe Admissions decisions 7 and understand to the best that I could, I call it 8 role. 9 And I'm not sure, I guess I should be 10 an English major to understand the extent of 11 difference. 12 Q. The reason I ask is, you maybe you haven't been told 13 this, it's one of the questions the Court has put to 14 the parties for trial here, is the extent to which 15 race is used in the Law School admission process, 16 are you aware of that? 17 A. I wasn't aware of the specific questions, no. 18 Q. All right. You believe that your analysis 19 quantifies the role that race plays in the 20 Admissions process, is that right? 21 A. That's exactly what I think I did, was to try to 22 describe the Admissions decisions, and the role in a 23 statistical sense, and again I'm doing this in a 24 statistical sense. 25 The statistical sense describe the GRUTTER -vs- BOLLINGER, ET AL 118 1 effect when called a statistical purpose calls the 2 effect of race on the chance of admission, that's 3 right. 4 Q. Okay. You said that's what you tried to do. In 5 your view, in your odds ratio analysis actually does 6 that? 7 A. I think it does a good job of summarizing the data 8 that we have. 9 Q. You also said this morning that one of the things 10 that you tried to do is analyze how likely it is the 11 results that you found would occur by chance, is 12 that right? 13 A. With respect to particular odds ratios, and I 14 calculated P values for numbers of standard 15 negations. And they specifically addressed--the 16 question is, how likely is this outcome that we see 17 compared to an outcome that was due to chance alone. 18 Q. That was the purpose of your inferential analysis? 19 A. That certainly summarizes what is given by the 20 standard negations. 21 Q. You're aware that the Law School has a written 22 admissions policy, is that right? 23 A. I think I was given a policy, yes. 24 Q. And you're aware that that policy says that race is 25 considered as a factor in the Admissions process, GRUTTER -vs- BOLLINGER, ET AL 119 1 correct? 2 A. That's correct. 3 Q. So the policy itself indicates that these results 4 are not occurring by chance? 5 A. The policy indicates that these results are not 6 intended to be occurring by chance. 7 Q. So, all you've done is, in this respect, is confirm 8 that the policy is actually having the effect that 9 it says that it intends to have? 10 A. In the sense that race is a factor and if it was 11 intended to be that, then certainly my analysis does 12 show that race is a factor. 13 Q. I believe your reports, certainly your first report, 14 says that the odds ratio analysis--let me rephrase 15 this. 16 The odds ratio was the main tool for 17 comparison in your analysis, is that a fair 18 statement? 19 A. I think the odds ratio is a tool for comparison. It 20 summarizes the individual's odds ratio, yes. 21 Q. You believe that your odds ratios give an accurate 22 picture of the role that race is playing in 23 admission? 24 A. With respect to the individuals with similar 25 credentials, GPA and LSAT and residency and gender GRUTTER -vs- BOLLINGER, ET AL 120 1 and fee waiver status, I think the odds ratio gives 2 a good composite measure which summarizes the extent 3 to which--extent, I use your word, to which race is 4 being used, yes. 5 Q. Is it your opinion that the odds ratio does not give 6 an accurate picture for some group of students other 7 than the ones that you just mentioned? 8 A. No. 9 Q. So, for all students in the applicant pool, you 10 believe that your odds ratios are giving an accurate 11 picture of the role that race is playing in 12 admission? 13 A. I'd better make sure I understand the question. Are 14 you saying for each individual does the odds ratio 15 apply? I'm trying to understand, and I really do 16 understand what the question is. 17 Q. Okay. Let's take the individual first. Is it your 18 view that the odds ratios that you reported give an 19 accurate view of any particular individuals relative 20 on to admissions? 21 A. The odds ratio is a summary of what happens to a 22 group of individuals. And individuals themselves 23 will, in fact, have all kinds of factors and affect 24 with respect to them. But it summarizes for the 25 group as a whole what the affect was. GRUTTER -vs- BOLLINGER, ET AL 121 1 Q. And as a summary statistic, you believe that it's a 2 fair and accurate picture of the role that race is 3 playing for all applicants, as opposed to only some 4 subset of the applicants, that's what I'm asking? 5 A. The analysis I did was for the group of all 6 applicants. And so I think it does summarize the 7 measure of the role of race for that group of 8 applicants, yes. 9 Q. Now, this morning Judge Friedman asked 10 whether--actually if we could put up slide 30 from 11 this morning. I think that's what was up when the 12 question came. 13 This morning Judge Friedman asked 14 whether this 81 odds ratio for Mexican Americans 15 here, meant that Mexican Americans were 81 times 16 more likely to be admitted than white students, do 17 you remember that discussion? 18 A. Yes. 19 Q. And, in fact, the answer is no, that does not mean 20 that the Mexican Americans are 81 times more likely 21 to be admitted than the white students, isn't that 22 correct? 23 A. If we mean times likely in terms of 81 times 24 probability if that's what you're meaning, then 25 that's correct. It's, in fact, an odds multiplier GRUTTER -vs- BOLLINGER, ET AL 122 1 and so it, in fact, multiplies the odds depending on 2 what the base line odds would be. 3 Q. Okay. Let's go back to this, your drawing from this 4 morning, if we could. I think you said that this 81 5 for Mexican Americans approximates basically the 6 lower half here of what you sketched out, is that 7 right? 8 A. That's correct. 81, yes. 9 Q. So, the 81 odds ratios translates to a ratio of 10 probabilities of .9 to .1, is that right? 11 A. Ten percent probability using an odds ratio of 81 12 would become 91 probability, that's true. 13 Q. So, to plus that information into the sentence that 14 we used before, for these numbers we would say that 15 the group with the 90 percent probability of 16 admission is nine times as likely to be admitted as 17 the group with the ten percent probability, is that 18 right? 19 A. If you're using it in terms of probability, that's 20 in terms of probability. 21 Q. Okay. So, the likely language refers to 22 probability, in your view, as commonly used by 23 statisticians? 24 A. No, as commonly used by statisticians, statisticians 25 will work in terms of odds. If I might say, I'm GRUTTER -vs- BOLLINGER, ET AL 123 1 sorry. With respect to analysis of binary response 2 we work in terms of odds and odd multipliers or odds 3 ratios. 4 Q. You work in terms of odds, but let me go back to the 5 sentence earlier. Is it fair to say that this 6 example here with the 81 odds ratio, indicates that 7 the group with the 90 percent probability is nine 8 times as likely to be admitted, say, the group with 9 the ten percent probability? 10 A. What I would say it's nine times the probability, 11 that's what I would say. We particularly don't use 12 likely, because it's subject to all kinds of 13 misinterpretation. 14 Q. All right. And just to take another example, the 15 one that you have up here at the top half, the 75 16 percent chance or 75 percent probability versus 25 17 percent, the odds ratio comes out to nine. 18 But we would say that the probability 19 is only three times greater for the group of 75 20 percent chance, isn't that right? 21 A. In terms of probability, that's right. 22 Q. Okay. I think I just want to get this clear, 23 because as you say there's room for 24 misinterpretation. And you would agree that we 25 should try to be precise in the language that we GRUTTER -vs- BOLLINGER, ET AL 124 1 use. 2 So when we're talking about odds, we 3 should use odds type of language? 4 A. As best I can, I will try to be specific as that, 5 yes. 6 Q. Let me do another example just to get a sense of the 7 relationship between probabilities and odds. If one 8 group has a .99 probability of getting admitted, the 9 99 percent, and a second group has a 90 percent 10 probability of getting admitted, so .99 versus .90. 11 Am I correct that the odds ratio for 12 that is eleven, or about eleven? 13 A. Well, we can do the math. 14 Q. Let me get a pen here. .99 versus .90, let me just 15 make sure I have this right. The formula would be 16 for the odds ratio .99 over .01 all divided by .90 17 over .10, is that right? 18 A. That looks good. 19 Q. Okay. And this comes out to about eleven, doesn't 20 it? You can check me with your calculator, if you 21 like? 22 A. I think eleven is a good number. 23 Q. So, if we get an odds ratio of eleven even though 24 the probabilities are very close, right? 25 A. The probability of acceptance is close, and the GRUTTER -vs- BOLLINGER, ET AL 125 1 probability of denial is quite different. 2 Q. The probability of acceptance is--let me just put it 3 this way. Both groups are highly likely to be 4 admitted in that case? 5 A. Highly likely to be admitted? 6 Q. Yes. .90 versus .99? 7 A. Yes. 8 Q. Okay. And yet you end up with an odds ratio of 9 eleven? 10 A. That's true. 11 Q. When you indicated that earlier two or three in your 12 experience is a very large odds ratio? 13 A. Yes. And if the Court permits I can explain. The 14 reason, of course, is that you also can look at the 15 chance of denial. And the chances of being denied 16 admission for these are one percent versus ten 17 percent. 18 And so there's a symmetry with 19 respect to that, and that's why we use odds in 20 statistics. And so if you look at the chances of 21 denial, it's one percent versus ten percent which is 22 quite discrepant. 23 If we were in a situation in a 24 medical study, a ten percent complication is greater 25 than one percent would be quite big. GRUTTER -vs- BOLLINGER, ET AL 126 1 Q. Am I correct that your opinions concerning how large 2 a particular odds ratio number is, is based on your 3 experience with this technique, correct? 4 A. That's correct. 5 Q. So, if in your experience you encounter, and the 6 kinds of work that you do, you encounter odds ratios 7 of two or three in contacts where that would be 8 large, that forms the basis for your opinion about 9 what constitutes a large odds ratio? 10 A. It certainly informs from my experience as a 11 statistician over the period of time I've been a 12 statistician. 13 Q. Okay. Am I correct that none of your prior 14 publications relate to work on issues on higher 15 education? 16 A. I don't have my--in front of me, but I don't recall 17 any particular applications of higher education in 18 my publication list. 19 Q. And before your work on this case, you had never 20 worked with admissions data? 21 A. In a litigation setting, that is true. 22 Q. In a statistical setting? 23 A. As examples for classes and things like that, sure 24 we have used them. 25 Q. Before your work in this case, you had never GRUTTER -vs- BOLLINGER, ET AL 127 1 designed regression models concerning professional 2 school admissions, is that right? 3 A. That's correct. 4 Q. And you had never given expert testimony on 5 Admissions in any other content? 6 A. That's correct. 7 Q. So, your approach to this case and your opinions 8 about the sizes of the effects that you found, based 9 on your experience which did not include experience 10 with admissions data? 11 A. I think I said I hadn't had experience with 12 Admissions data with respect to working in the 13 litigation setting, or with respect to 14 administrative work. 15 I've certainly worked at Admissions 16 data with respect to examples that we used in 17 classes and so on that. So, I certainly have seen 18 Admissions data examples, and I have looked at those 19 before, yes. 20 Q. But not in litigation, not in this context? 21 A. Not in litigation context, that's true. 22 Q. If you look back at the slide, which I think again 23 is 30 from your presentation this morning, the odds 24 ratio for African Americans there is given as 25 257.93. I just want to get a sense of what that GRUTTER -vs- BOLLINGER, ET AL 128 1 means, going back to our earlier discussion. 2 Am I correct that you're not saying 3 that any African American student has 257 times the 4 odds of admission than any white student? 5 A. On an individual basis? 6 Q. Yes. 7 A. No, this is on an aggregate basis for the whole 8 group. 9 Q. And that has to be the case, right, because not all 10 of the African American students were admitted? 11 A. If they were all admitted it would have been 12 infinity. 13 Q. And if close to all of them were admitted--well, let 14 me put it this way. 15 You agree that the Law School only 16 admits about a third of all of its students, of all 17 of its applicants, right? Somewhere in that range? 18 A. We saw numbers this morning, yes. 19 Q. And, in fact, you understand that the proportion of 20 the so-called majority students who were admitted, 21 is actually larger than the proportion of minority 22 students that are admitted, am I correct? 23 A. I don't have the numbers in front of me. 24 Q. You don't know one way or another? 25 A. I don't know one way or the other. Specifically I GRUTTER -vs- BOLLINGER, ET AL 129 1 don't recall. 2 Q. When you were approaching your work in this case, 3 did you consider those basic probabilities of 4 admission in evaluating your results? 5 A. Did I consider those basic probabilities of 6 admission? I considered looking at comparing 7 individuals with similar credentials, and that's 8 what I would do as a statistician. 9 Q. But you didn't use the overall probability of 10 admission for majority and minority students as a 11 check on the reasonableness of your odds ratio 12 estimate? 13 A. I think I--I didn't do it, I would only do that if 14 they had similar credentials. 15 Q. But as a general matter for the applicant pool as a 16 whole, you did not do that, you didn't use the 17 applicant admission? 18 A. I did not. 19 Q. Now, you have mentioned a couple of times, and 20 certainly many times this morning, the idea that you 21 wanted to look at applicants with similar 22 credentials. 23 Is it fair to say that that was one 24 of the basic principals of your analysis, you try to 25 identify the students with similar credentials and GRUTTER -vs- BOLLINGER, ET AL 130 1 compare those? 2 A. I think I took as a basic principal of my analysis 3 that I would use, the groupings as set by the 4 Law School itself to define groups like that, and 5 that's where I started from. 6 Q. And when you say groupings as defined by the 7 Law School itself, you mean in the grids in 8 Exhibit 16 that you were given? 9 A. That's correct. 10 Q. You didn't get those groupings from any other 11 source? 12 A. The groupings came directly out of Exhibit 16. 13 Q. For example, you didn't review the deposition 14 testimony of the Admissions officers who actually 15 makes the decision when deciding on the structure 16 for analysis, did you? 17 A. I don't recall such review. 18 Q. And when you use the term credentials here today, 19 you generally mean GPA and LSAT scores, isn't that 20 right? 21 A. For the most of our analysis GPA, LSAT. In some 22 sense we did analysis involving residents, gender, 23 fee waiver. 24 Q. Did you consider residents or gender or fee waiver 25 to be credentials when you used the term? GRUTTER -vs- BOLLINGER, ET AL 131 1 A. They're characteristics of the applicants. 2 Q. But when you talk about credentials, you're talking 3 about grades and test scores, isn't that right? 4 A. That's what I'm talking about. 5 Q. And you chose those credentials, to use your term, 6 because that's the data you got, right? 7 A. That certainly was the data I have. 8 Q. You didn't have statistical data on the quality of 9 the essays, or the quality of the letter of 10 recommendation and so forth? 11 A. That's correct. 12 Q. But you have examined the Law School's admission 13 policy, right? 14 A. Yes. 15 Q. So, you understand that the policy contemplates 16 consideration of many factors, other than grades and 17 test scores? 18 A. Of course. 19 Q. It considered things like the essays and letters of 20 recommendation and strength of curriculum and the 21 like? 22 A. Of course. 23 Q. Those factors while they're considered by the 24 Admissions Office, are not included in your model, 25 is that right? GRUTTER -vs- BOLLINGER, ET AL 132 1 A. I didn't have the data, I didn't include it in my 2 model, that's correct. 3 Q. I would like to look now at your odds ratio analysis 4 in a little bit more detail for some background. 5 Am I right that you you basically 6 did--you computed two kinds of odds ratios. You 7 computed them cell by cell, and then you computed 8 what you called a composite odds ratio, isn't that 9 right? 10 A. Certainly I did that in various parts of the 11 reports, yes. 12 Q. I want to look at the cell by cell part first where 13 you ended this morning. 14 You computed odds ratios for, I 15 guess, for applicants in small groupings of grades 16 and test scores, right? 17 A. That's correct. 18 Q. In fact, you broke the applicant pool down into a 19 total of 240 cells for each year, isn't that right? 20 A. I believe that's correct. I would have to go back 21 and count, but I believe that's correct. 22 Q. Well, we counted. And assuming that we counted 23 correctly, it's 240. 24 A. I believe you. 25 Q. Okay. So, then you computed an odds ratio for each GRUTTER -vs- BOLLINGER, ET AL 133 1 of those individual cells? 2 A. For each cell for which there was comparative 3 information, that's true. 4 Q. Okay. And you excluded some of the cells from your 5 analysis, is that right? 6 A. I think I just said what I did. Which was to 7 compute an odds ratio from each cell that had 8 comparative information. 9 Q. And so in your view a cell that had no applicants at 10 all in it, didn't have comparative information, 11 right? 12 A. I would agree with that. 13 Q. And in your view, cells that had only applicants, 14 only majority applicants, for example, but no 15 minority applicants, that cell didn't have 16 comparative information? 17 A. That's true. 18 Q. Okay. But you also excluded cells where all of the 19 applicants in both groups were admitted, isn't that 20 right? 21 A. That's true. 22 Q. And you excluded the cells where none of the 23 applicants in either group were admitted? 24 A. That's true. 25 Q. Because in your view, those cells don't have GRUTTER -vs- BOLLINGER, ET AL 134 1 comparative information? 2 A. Based on statistical principals of computing odds 3 ratios and getting comparative information and using 4 standard techniques, that's what we would do. 5 Q. So, from your statistical point of view, they don't 6 have comparative information? 7 A. They don't have comparative information. 8 Q. You'll agree though that one could look at the two 9 cells and compare them, right? 10 A. They're put out in my reports, one can certainly 11 look at them. 12 Q. And it is a form of comparison to say that all of 13 the applicants in both groups with admitted or 14 denied? 15 A. It's a form of comparison. 16 Q. But it didn't fit within the requirements of your 17 statistical approach, right? 18 A. They didn't fit in the requirements with respect to 19 getting a composite odds ratio, because they give no 20 comparative information with respect to the odds 21 ratio. 22 Q. And so because they didn't fit in your approach, you 23 set them aside? 24 A. Because they didn't give comparative information 25 with respect to the odds ratio, they were not GRUTTER -vs- BOLLINGER, ET AL 135 1 computed. 2 Q. Okay. I'd like for you, if you would, to look at 3 Exhibit 138. 4 MR. DELERY: I believe it's in binder 5 five, your Honor. 6 BY MR. DELERY: 7 Q. It's one of the reports that you introduced this 8 morning, I believe that's your February 21, 2000 9 report. 10 So, the reason, just to clarify 11 something from this morning, the reason that there 12 are a series of reports, is that because the case 13 has been pending for a while we've passed through 14 several additional admission seasons, right? 15 A. That's correct. 16 Q. And as the data became available it was provided to 17 you? 18 A. I don't know that, I know I got the data. 19 Q. I'd like for you, if you would, to turn to the last 20 series of tables here in the report that have the 21 heading Cell by Cell Comparison and Admission 22 Rating. 23 There's 24 pages of them and I would 24 like for you to turn to the one that says page one 25 on 24 at the bottom. GRUTTER -vs- BOLLINGER, ET AL 136 1 A. I have that. 2 Q. Do you have that, okay. Pages one through six of 24 3 are the 240 cells for 1995, is that right? 4 A. That's correct. 5 Q. And the slides that we saw at the end of your 6 testimony this morning are some of these cells from 7 these pages, did I have that right? 8 A. That's correct. 9 Q. Would you agree that for the vast majority of cells 10 on these pages, you don't calculate any odds ratio? 11 A. I don't know about vast majority, but there 12 certainly are a good number of cells with a 13 calculation of odds ratios. 14 Q. In fact, you only calculated numbers, numerical odds 15 ratios in twelve cells? 16 A. On the six pages? 17 Q. Yes. 18 A. Numerical values, you mean you're not--you're 19 assuming, you're saying infinity doesn't count. 20 Q. Okay. 21 A. If you're saying infinity doesn't count, then that 22 may be true. That there are twelve that are 23 infinity, there are a good number that are infinity. 24 Q. Okay. Let's just look at the first page and we'll 25 turn further if we have to. On the first page, all GRUTTER -vs- BOLLINGER, ET AL 137 1 of the cells are blank except for four near the 2 bottom, right? 3 A. That's correct. 4 Q. So you calculated four odds ratios on this page and 5 all four were infinities? 6 A. That's correct. 7 Q. Okay. And looking at the first of the infinity 8 cells, you've got one minority applicant who was 9 admitted and ten majority applicants one of whom was 10 admitted? 11 A. That's correct. 12 Q. And so those probabilities of admission gives you 13 this infinity odds ratio? 14 A. The observed odds ratio is infinity, that's correct. 15 Q. That's not the same as saying that the minority 16 student were infinitely more likely to get in than 17 the majority student, is it? 18 A. That's saying this particular cell the individuals 19 that we saw, that's the observed odds ratio. 20 Q. But it's not the same as saying that in this cell a 21 minority student was infinitely more likely to get 22 in than a majority student, is it? 23 A. I don't believe that's true, and statistically I 24 think I've said that before. These infinities 25 represent the observed information. I believe that GRUTTER -vs- BOLLINGER, ET AL 138 1 if we had many more students in these cells, we 2 would not get infinity. 3 Q. So, in some sense the odds ratio what you find are a 4 function of the small pool sizes in a lot of these 5 cells, particularly for minority students? 6 A. I said these are small pool sizes, that's absolutely 7 true. And the infinities are a result of that, yes. 8 Q. Okay. Just so we're clear. The reason that you get 9 infinity, which seems like a big number, is just 10 that you're dividing by zero and that's a 11 mathematical construct? 12 A. In this case the odds of minority admit are 13 infinity, and that's a one divided by zero. And the 14 odds of majority admit are one in nine, that's 15 right. 16 Q. But these infinities are, when we use the term 17 infinity, it's a mathematical construct, it's not an 18 actual infinite likelihood of getting in, right? 19 A. I don't think such exist. 20 Q. I am just trying to make sure that we're all on the 21 same page. Let's turn to the second page. 22 Page two you've got nine infinite 23 odds ratios, right? 24 A. That's correct. 25 Q. And then you got one for which you can actually GRUTTER -vs- BOLLINGER, ET AL 139 1 calculate the odds ratio, near the bottom 4.33? 2 A. I think I would say I got one for which you can give 3 a value other than infinity. Yes, that's true. 4 Q. Do you view the infinite odds ratios as computable 5 odds ratios. 6 A. They certainly are computable, yes. They are the 7 observed odds ratio. 8 Q. And mathematically, you think that that's 9 computable? 10 A. Are we into the foundations of mathematics now? It 11 certainly add a representation of the numbers that 12 are seen. There certainly is comparative 13 information there, and infinity is certainly a 14 representation of that comparative information. 15 Q. Where you have a cell where all of the applicants 16 are admitted in both groups? 17 A. Yes. 18 Q. Do you believe that that is a computable odds ratio? 19 A. Where all are admitted? 20 Q. Yes. 21 A. In fact, in that case you're going to wind up 22 infinity divided by infinity if we're doing the 23 mathematics. And I would say that that's not 24 something that we could compute, because there's no 25 comparative information there. GRUTTER -vs- BOLLINGER, ET AL 140 1 Q. I just want to understand the difference between 2 that what you say is not computable, and a result of 3 infinity which is computable? 4 A. I think the important thing is that we computed 5 value for every cell with comparative information. 6 And that's all. 7 Q. Okay. The first of the infinities, it's a little 8 hard to see on here. I think the first of the 9 infinities on this list is in bold, does that 10 indicate significance? 11 A. We're on page two of the report? 12 Q. Page two, yes. 13 A. It's the third line from the top down? 14 Q. Yes. 15 A. And corresponds the GPA of 3.25, 3.49, LSAT 151 to 16 153? 17 Q. Right. 18 A. And residents in 1995? 19 Q. Yes. 20 A. And minority applicants on two out of two were 21 admitted, and majority of nonselected minority zero 22 out of nine were admitted, that's right? 23 Q. Right. 24 A. And so that would be calculated as infinity. 25 Q. Okay. GRUTTER -vs- BOLLINGER, ET AL 141 1 A. As the observed odds ratio. 2 Q. And am I right that you're indicating that you feel 3 that odds ratio is statistically significant? 4 A. What we're doing here is comparing the rate of 5 admission which for minorities observed rate is a 6 hundred percent. To the rate of admission for 7 majority and non-selected minorities which is zero 8 percent. 9 And even with this small sample size 10 the test that's used to do that comparison indicates 11 that there is a statistically significant difference 12 between those admission rates. 13 Q. Okay. And you have listed in bold in these tables 14 the cells that you find--the cells for which you 15 find a statistically significant odds ratio, 16 correct? 17 A. As best I could I bolded the lines that have P 18 values less than .025. That's what I intended to 19 do, if I didn't do that then I made a mistake. 20 Q. And where P value is less than .05, in your view 21 that indicates statistical significance of the odds 22 ratio that you're reporting? 23 A. In the particular cells that we're looking at. On 24 that cell by cell basis, that's correct. 25 Q. Okay. So, the cells that for which you report odds GRUTTER -vs- BOLLINGER, ET AL 142 1 ratios but they're not in bold, that's an indication 2 that you don't believe that those odds ratios are 3 statistically significant, correct? 4 A. The cells that are not in bold that have P values 5 greater than .025, means that from that cell 6 information alone, that cell information alone, 7 there isn't sufficient information that indicates 8 that it is statistically significant from the 9 information from that cell alone. 10 Q. So, let's continue, look at page three on this 11 table. You've got one numerical odds ratio and six, 12 I believe, infinities? 13 A. That's correct. 14 Q. On the next page you have three more infinities? 15 A. That's correct. 16 Q. And pages five and six you actually have a number of 17 them. I count on page five, eight numerical odds 18 rations and 14 infinities. And two odds ratios on 19 page six and 16 infinities? 20 A. You want me to count to confirm? That's about 21 right. 22 Q. Feel free to count if you want. 23 A. I'll count if you want me to. 24 Q. Okay. So, in some, am I right, that you've 25 calculated in these 240 cells twelve numerical odds GRUTTER -vs- BOLLINGER, ET AL 143 1 ratios and 52 infinities? 2 A. I didn't sum that up, but I presume that's about 3 right. I forgot, you tell me twelve that had values 4 less than infinity? 5 Q. Yes. 6 A. And? 7 Q. 52 infinities? 8 A. 52 that had values of infinity. That would--it's 9 probably about right. 10 Q. Okay. And by my count you found seven of the 11 numerical odds ratios to be statistically 12 significant. And 14 of the infinities to be 13 significant, does that sound about right? 14 A. Well, I think I testified this morning there would 15 be between 17 and 22 for each year. And so you said 16 21 altogether? 17 Q. That's about right, yes, 21? 18 A. That would be consistent, right. 19 Q. So, to sum up, you found statistically significant 20 odds ratio in only 21 of the 240 cells in 1995? 21 A. 21 of the total number of cells. I don't know how 22 many that is of the cells with comparative 23 information. 24 Q. And again you define comparative information to mean 25 that there are more applicants than admits in both GRUTTER -vs- BOLLINGER, ET AL 144 1 groups, that's your definition of comparative 2 information? 3 A. Well, I'll give my definition just to be clear. In 4 order for a cell to have comparative information, 5 there have to be minority applicants in the cell, 6 there have to be majority applicants in the cell. 7 There have to be some applicants that 8 are admitted and some that are not admitted. So 9 that's the condition that's required. 10 Q. Okay. At any point, did you look to see how many of 11 the applicants were in the cells for which you found 12 statistically significant odds ratio? 13 A. Did I look to see how many were in those cells? 14 Q. Sure. 15 A. No, I didn't calculate that. 16 Q. Would you be surprised to learn that in the seven 17 cells was statistically significant in numerical 18 odds ratio, there are a total of 601 applicants in 19 your table? 20 A. Would I be surprised, no. 21 Q. Okay. Or that there were 894 applicants in the 14 22 infinity cells that you find to be statistically 23 significant? 24 A. So that's about 1400 overall in the two groups? 25 Q. Yes. GRUTTER -vs- BOLLINGER, ET AL 145 1 A. No. I would think that would be the size of the 2 group that would have comparative information, yes. 3 Q. And there were just over 4000 applicants this year, 4 right? 5 A. Certainly. 6 Q. By my math, you can check me. If there's 4,147 7 applicants and 1,495 applicants appear in the cells 8 that you find to be statistically significant, that 9 means that your statistically significant odds ratio 10 cover only about 36 percent of the total number of 11 applicants, does that sound about right? 12 A. Those are the applicants that are in the cells with 13 comparative information, I think that's probably 14 right. 15 Q. All right. And so I just want to be clear. You're 16 drawing your conclusions in this odds ratio analysis 17 based on cells with just over a third of the total 18 number of applicants in the pool? 19 A. That's correct. And ones information whom there's 20 comparative information. 21 Q. So that means that nearly two-thirds of the students 22 are in cells that you have excluded from your 23 analysis? 24 A. Two-thirds of the students had credentials where 25 they didn't give any comparative information, that GRUTTER -vs- BOLLINGER, ET AL 146 1 would be correct. 2 Q. When you say that you feel that cells where all the 3 students are admitted don't have comparative 4 information, you're aware that the University or the 5 Law School could decide not to admit all of those 6 students, right? 7 A. I'm sure if they had a larger pool that we couldn't 8 find them all admitted, that's true. 9 Q. So to some extent those cells reflect choices made 10 by the Law School Admissions officers? 11 A. I think that every cell indicates choices made by 12 Law School Admissions officers, that's true. 13 Q. And similarly cells where all of the students were 14 denied admission, the University could have decided 15 to admit some of the minority students, for example, 16 from the cells, right? 17 A. Certainly. These are displaying the Admissions 18 decision. 19 Q. And so the University simply in this year chose not 20 to admit any of those students, correct? 21 A. In those particular cells? 22 Q. Yes. 23 A. Primarily the ones with the low GPA and low LSAT, 24 they may not have admitted any students with those 25 particular combinations of credentials, that's true. GRUTTER -vs- BOLLINGER, ET AL 147 1 Q. So, the cells that you have excluded from your 2 analysis include information about decisions that 3 the Law School is making, correct? 4 A. Certainly. 5 Q. Okay. But you're not considering those decisions 6 when you report your odds ratio? 7 A. Those cells don't give comparative information, so 8 they're not included in the odds ratio calculations. 9 Q. I'd like to turn now to what you call the composite 10 odds ratios, the numbers taking the various cells 11 put together. And I want to talk a little bit about 12 how you got those odds ratios. 13 Am I right that you generated those 14 using the regression analysis? 15 A. I use the technique called logistic regression, 16 that's true. 17 Q. And your regressions used models that represented, 18 in some sense, the admissions process, correct? 19 A. I think the model that I used allowed us to control 20 the cell grids, and took whatever admissions 21 proportions were seen in those cell grids and 22 analyzed those, that's true. 23 Q. I'm asking a more basic question then that. And 24 that's just that your regressions used models to 25 represent the admissions process, right? GRUTTER -vs- BOLLINGER, ET AL 148 1 A. Models in the statistical sense, I did a model term 2 in my regression analysis the answer is yes, of 3 course. 4 Q. Because obviously you can't replicate the actual 5 admissions process, statistically? 6 A. I'm not sure I understand the question. 7 Q. To do a regression analysis, you have to use a model 8 because given the data you had, you can't actually 9 replicate the admissions process that the Law School 10 uses? 11 A. I'm afraid you've used the same words, and since I 12 didn't understand the first time I probably don't 13 understand it now. 14 Q. Okay. Let me try it this way. Your regression 15 analysis was designed in some sense to approximate 16 the admission process that the Law School use, is 17 that fair to say? 18 A. No. 19 Q. You didn't get the individual applicant files and 20 sit down and read them obviously? 21 A. If you mean that I was supposed to go through the 22 files and replicate--that's what you mean a 23 replication, did I read 4,500 files and make 24 decisions on that. The answer is, I did not do 25 that. GRUTTER -vs- BOLLINGER, ET AL 149 1 Q. You tried to use the statistical technique to draw 2 conclusions about the Admissions decisions without 3 actually going through the same process that the 4 Admissions officers had gone through, right? 5 A. If I'm understanding now, did I look at the files 6 and make decisions, and the answer is I did not. 7 And so I don't think I was trying to replicate the 8 process, what I was trying to do is understand the 9 results of the Admissions process and summarize 10 those. And that's what I was doing. 11 Q. And just trying to understand. You're doing that 12 though with something that you have constructed, in 13 other words, using this statistical technique that 14 you call logistic regression? 15 A. I use the standard technique for analyzing data 16 controlling for, in this case, LSAT and GPA. That's 17 what I did. 18 Q. And that's something that you constructed, you 19 didn't get it from the Law School? 20 A. What do you mean, did I decide to do this 21 statistical analysis? I made the decision, yes. 22 Q. And you decided how to do it? 23 A. How to do it? 24 Q. Yes. 25 A. In the sense that I decided I would use logistic GRUTTER -vs- BOLLINGER, ET AL 150 1 regression? 2 Q. Yes. 3 A. I did decide to use logistics regression, that's 4 correct. 5 Q. And in doing your logistic regression, you made 6 various choices along the way? 7 A. Very few choices, but I suppose I must have made 8 some choices. 9 Q. You made some choices. And, in fact, you had to 10 make certain assumptions along the way in designing 11 your models, isn't that right? 12 A. This model makes very few assumptions, but there 13 always are assumptions. There's a saying, it's not 14 a saying, George Bakke the professor says, how can I 15 say this? All models are wrong, some models are 16 useful. 17 So, I certainly made a choice of 18 looking at particular models, and I think I looked 19 at ones that I thought were useful. 20 Q. You'd agree that it's important in order to evaluate 21 your results to understand the assumptions that you 22 made as you designed your models? 23 A. As best we can, sure. 24 Q. Okay. And without a clear sense of the assumptions, 25 we can't evaluate the accuracy or reliability of the GRUTTER -vs- BOLLINGER, ET AL 151 1 results, isn't that right? 2 A. Well, I think we can look at the assumptions, I'll 3 be glad to. 4 Q. All right. One of your assumptions was that the 5 association between race and admissions does not 6 vary with differing levels of grades and test 7 scores, isn't that right? 8 A. I think I said this morning that the composite 9 estimate provides us a value that goes across--that 10 summarizes the individual cell odds ratios. 11 So, in fact, the model that did the 12 computation, then assumes that it would be the same 13 across, that's correct. 14 Q. Okay. And so in order for your results to be 15 useful, that assumption needs to be correct, isn't 16 that right? 17 A. No, that doesn't have to be absolutely correct. No. 18 Q. If it's wildly wrong, you think it doesn't affect 19 your results? 20 A. I think the composite estimates still summarizes the 21 odds ratio for this cells that we have, and it's a 22 summary. 23 Q. You would agree that for applicants with low grades 24 and test scores regardless of race, those applicants 25 are extremely unlikely to get in, correct? GRUTTER -vs- BOLLINGER, ET AL 152 1 A. The evidence is clear that the Law School use grades 2 and test scores. And applicants with low grades and 3 low test scores have very little or no chance to get 4 in. Many of those cells have all of those students 5 denied, that's correct. 6 Q. And that's true without regard to race? 7 A. Absolutely. 8 Q. So, for those applicants whether or not they're a 9 member of a minority group, is really unimportant 10 for deciding or for predicting whether they're going 11 to get in, is that fair to say? 12 A. There certainly is a range of grades and test scores 13 where students are not admitted, that's true. 14 Q. And for those students whether or not they're 15 members of a minority group, is unimportant for 16 predicting whether they'll get in, is that correct? 17 A. That would be true, yes. 18 Q. Let's look at the other end of the scale. At the 19 very upper end of grades and test scores say, you 20 know, 4.0 and LSAT of about 170, applicants in that 21 range are extremely likely to get in regardless of 22 their race, isn't that right? 23 A. That's correct. 24 Q. And for those students whether or not they're 25 members of a minority group, is unimportant for GRUTTER -vs- BOLLINGER, ET AL 153 1 predicting their chances of getting in? 2 A. Actually I think the evidence contradicts that. 3 Because I think if you look at those upper ranges, 4 there are, in fact, still--there's still information 5 in most of those upper ranges. 6 Because there are non-minority 7 students denied in most of those cells, not a 8 hundred percent. And where there's students that 9 are both admitted and denied, we can look for 10 comparative information. 11 Q. But if students in those ranges are extremely likely 12 to get in, we're back to our .99 versus .90, for 13 example. Those probabilities are true without 14 regards to race, they're at the extreme high end? 15 A. The probabilities may be high, but if you look at 16 just in your example you have .99 to .90. If you 17 had .999 to .90, then you would have an odds ratio 18 of 111. 19 And that was still in the sense of 20 comparative information that odds ratio could still 21 be high, but now it's effecting not the admitted 22 cohort but the denied cohort. The ones that are 23 chosen not to be in. 24 Q. So, in that situation where you go from .99 to .999, 25 you're getting ever increasing odds ratios even as a GRUTTER -vs- BOLLINGER, ET AL 154 1 practical matter that the Admissions decisions are 2 coming out and saying, isn't that right? 3 A. I have to say I don't think they're the same. If 4 the students that--you can ask the students who are 5 denied in those cells if they think the decisions 6 are the same. 7 Q. So, in your view as you understand how the process 8 work, do you think that there is a real world 9 difference between a .99 chance of admission and a 10 .999 chance of admission? 11 A. Now, you're switching, I'm sorry. Say it again so I 12 can make sure I've got the numbers right. 13 Q. Do you believe that there is a real world difference 14 for applicants between a .99 chance of admission and 15 a .99 chance of admission? 16 A. You know, I think we're probably in an area of about 17 99 percent, where given the size and samples where 18 we probably wouldn't have much of a difference. 19 But in the examples you gave of 90 20 percent versus 99, I probably think there is a 21 difference, yes. 22 Q. And going back to the issue of levels of grades and 23 test scores, is it your view that the role that race 24 plays is the same at the very high levels of grades 25 and test scores, as opposed to the middle range? GRUTTER -vs- BOLLINGER, ET AL 155 1 A. Well, I'll tell you what I know about that if you 2 want it. What I know is that I was criticized with 3 regard to that issue. And I went back and looked at 4 the odds ratios in all of the cells and the 5 particular case where I was criticized, and I found 6 no pattern, no pattern that related to grades or 7 LSAT. 8 And admittedly the amount of 9 comparative information may be small in those upper 10 cells, so I don't have real information on it. But 11 there is no evidence, statistical evidence I have 12 that there is a strong, strong effect one way or the 13 other. 14 Q. So you don't have a view one way or the other about 15 whether the role is the same in those two? 16 A. I'm saying the information, the data itself, doesn't 17 inform me that there is information. So, I'm saying 18 when I looked, I looked for evidence to see whether 19 the odds ratios varied by grade point average and 20 test score. I looked at that information, because I 21 was criticized for that. 22 And when I looked and looked at the 23 odds ratios, I couldn't find any consistent pattern 24 that showed that it change, for instance, in the way 25 that you said, which was that it would be higher at GRUTTER -vs- BOLLINGER, ET AL 156 1 one area or the other. 2 Q. I want to go back to, I think now, my question a few 3 questions ago. 4 A. Sure. 5 Q. In your opinion, is the role that race plays in the 6 admission process, the same for applicants at the 7 very high levels of grades and test scores, as it is 8 for students in the middle range? 9 A. With respect to odds ratio, we have no statistical 10 evidence that it is different, it's a statistical 11 issue. I can't say it's the same, because we don't 12 have enough information to say it's the same. 13 What I have is in statistics we often 14 do things backward. What I have is, I have no 15 evidence that there are differences of this sort 16 that we're talking about. 17 Q. Am I right that that means you also have no evidence 18 that it's the same, based on your odds ratio 19 analysis? 20 A. My evidence is that the data are consistent with it 21 being the same. 22 Q. All right. So just trying to make sure I understand 23 your opinion. Your best opinion on this matter is 24 that, from what you have seen, the role and based on 25 your analysis, the role that race plays in GRUTTER -vs- BOLLINGER, ET AL 157 1 admissions is the same for students in the middle 2 ranges of grades and test scores as it is for 3 student at the upper range? 4 A. I'll say what I know and I'll try to say it as 5 clearly as I can. What the data itself does not 6 give me evidence that there are differences. I have 7 done statistical tests to try to check that 8 assumption. 9 The statistical test give no evidence 10 of differences, that's what I can say. You can 11 never say, you can never state, and I don't want to 12 be James Bone. 13 You can never say never, but because, 14 in fact, because in fact, the data are consistent 15 with a range of differences in odds ratio. 16 Q. Dr. Larntz, I just want to understand your opinion. 17 You indicated this morning that you were expressing 18 an opinion, attempting to quantify the role that 19 race plays in admissions. 20 And my question to you is, in your 21 opinion, is the role that race plays in admission 22 the same, or approximately the same, for students in 23 the middle ranges of grades and test scores, as it 24 is for students in the upper range? 25 A. And I'll answer it in the same way I can, because GRUTTER -vs- BOLLINGER, ET AL 158 1 I'm going to answer you with what I know. And what 2 I know is that the data are consistent with it being 3 the same. The data are consistent with it being the 4 same. 5 In statistics we can't say that we 6 have proven it when the data are consistent, but we 7 can say that the data don't contradict that 8 assumption, and that's true. 9 Q. And that's as far as you can go with what you have? 10 A. As far as I can go is my composite estimates if they 11 are different, if they are different, my composite 12 estimate gives me a value that if they are different 13 and they are higher or lower in one range or 14 another, then the composite estimate is greater then 15 some of them and less than some others. So the 16 composite does apply as a summary over the entire 17 table. 18 Q. So, the composite then is something like a weighted 19 average? 20 A. The composites--I have to been very careful. The 21 technique used doesn't do weighted averages. It's a 22 technique called maximum likelihood, and what it 23 does is it chooses the value that is the value that 24 is most consistent with the data. 25 Q. Is it fair to think about the weighted average, even GRUTTER -vs- BOLLINGER, ET AL 159 1 if that is not what is computed? 2 A. Well, what the computer did is nothing like a 3 weighted average, and I'm not sure that I 4 would--there's no way I could compute a weighted 5 average. What I think was weighted average and get 6 the numbers. It doesn't work that way. 7 The computer tries different values 8 and sees which one is the most consistent with the 9 data. So this is the value that is most consistent 10 with the data. 11 Q. Data as a whole looking across all the cells? 12 A. Looking across all the cells with comparative 13 information. 14 MR. DELERY: Your Honor, I think this 15 is sort of a logical stopping point, if that is good 16 for you. 17 THE COURT: Okay. We'll take a 18 recess in this case. 19 (A brief recess was taken.) 20 (Court back in session.) 21 THE COURT: Back on the record. 22 Okay, you may proceed. 23 MR. DELERY: Thank you, your Honor. 24 BY MR. DELERY: 25 Q. Dr. Larntz, I was going to take this down to get it GRUTTER -vs- BOLLINGER, ET AL 160 1 out of their way, but I wanted to ask one more 2 questions about this example that you had, I think I 3 learned something earlier. 4 You're saying that if we go from .99 5 to .999 versus .90, the odds ratio goes up to 111 6 from eleven? 7 A. The .999 to .90 it goes to 111, I think that's 8 right. 9 Q. And if we go to .999 more of them, do we add another 10 one here so that it's a 1,011? 11 A. It turns out in that range of odds and probabilities 12 and the corresponding conversions to odds, I think 13 that's correct. 14 Q. So the odds keeps growing? 15 A. The odds ratios keep growing, yes. 16 Q. The odds ratio, correct, I'm sorry. I'd like to go 17 back, if we could, to assumptions that you made 18 during your regression analysis. 19 Am I correct that another assumption 20 that you made is that the variables that you 21 included in your model were not related, not 22 correlated with the variables that were not in your 23 model? 24 In other words, the other admissions 25 factors for which you didn't have any data? GRUTTER -vs- BOLLINGER, ET AL 161 1 A. Did I make that assumption? 2 Q. Yes. 3 A. No. 4 Q. When a factor not in the model is related to the 5 factors in the model, am I right, that's called 6 confounding? 7 A. When a factor--I'm sorry, would you say it again so 8 I make sure I got your situation clear in my mind. 9 Q. Okay. When a factor that's not in a regression 10 model is related to one of the factors that is in 11 the model, am I right that that's called 12 confounding? 13 A. The technical term confounding is when two factors 14 are related. 15 Q. Okay. 16 A. And it either can be in or out of the model where it 17 could be confounding effect. 18 Q. So it's a more general term then just whether it's 19 in or out of the model? 20 A. Sure. 21 Q. Okay. Did you make an assumption that the 22 admissions factors that the Law School considers but 23 for which you didn't have data, were not confounded 24 with the factors that are in the model? 25 A. No. GRUTTER -vs- BOLLINGER, ET AL 162 1 Q. You didn't make any assumption one way or the other 2 about that? 3 A. I didn't make any assumption one way or the other. 4 Q. If it turned out, for example, that minority 5 students were comparatively over represented in the 6 group of applicants with, say, strong leadership 7 experience but leadership experience isn't in your 8 model, isn't the effect of that that your model 9 would overstate the effect that race is playing in 10 admissions? 11 A. What would happen in that case is that the model we 12 have measure the terms that are in there, and if 13 another factor were related to minority status then 14 some of the effect of that other variable would go 15 into the minority status effect, that's true. 16 Q. Okay. So, if that were the case, then the model 17 would be over stating the role that race is playing 18 in admissions? 19 A. It could go either way depending, you know. But the 20 models, the coefficients we have are for the terms 21 that we look at. So, the coefficients we have with 22 respect to looking at the grid cell, GPA and LSAT, 23 those are the effects of minority status given 24 those. 25 The ones looking at the other GRUTTER -vs- BOLLINGER, ET AL 163 1 variables, coefficient change obviously, and so they 2 are representing the effect given those other 3 variables, that's correct. 4 Q. And if it turned out that the minority students were 5 over represented in the group of applicants with 6 better than average leadership experience, then an 7 effect of that would be that your model overstates 8 the role that race plays in admissions? 9 A. The effect of other variables that are used as 10 they're related to race in the model, would mean 11 that if you include those other variables in the 12 model, then you would get a different odds 13 multiplier for race, that's true. 14 Q. And the different odds multiplier would be lower 15 than it is in your estimate? 16 A. It could be, it depends. When we added other terms 17 such as residency, gender and fee waiver, the odds 18 multiplier went up. But it could go either way. 19 Q. Did you do anything to evaluate whether the 20 admissions factors that you couldn't include in your 21 models were, in fact, related in some way to race? 22 A. I had no data to do it that way. 23 Q. So you didn't do any of that? 24 A. That's correct. 25 Q. I believe you said before that you selected your GRUTTER -vs- BOLLINGER, ET AL 164 1 general model, cell by cell model inspired by 2 Exhibit 16, is that right? 3 A. Inspired? 4 Q. Taken from, that's your point of departure? 5 A. It was provided to me, I wanted to understand the 6 effects based on that exhibit. If I'm ever inspired 7 I suppose. Maybe that's true. 8 Q. All right. And I think you said this morning also 9 that you used the model that you chose, or that the 10 model that you chose was the standard approach that 11 you would take in a medical study, is that right? 12 A. It's a standard approach I would take if I were 13 trying to evaluate, as carefully as I could, the 14 effect of the one factor versus another in any 15 study. 16 I have done this kind of study 17 whether it be a medical study, I gave that as an 18 example. This is the area that I did my research 19 in. So, this is the kind of analysis I would do 20 across a broad range of applications. 21 Q. It's the kind of analysis with which you're most 22 familiar, is that fair to say? 23 A. The kind analysis which I'm most familiar? 24 Q. Yes. 25 A. I don't know if I can rate all the analysis I know GRUTTER -vs- BOLLINGER, ET AL 165 1 and call one most familiar. It's one that I 2 certainly have used and feel comfortable using, yes. 3 Q. Am I right that when you created the model, you had 4 the computer create a variable for each of the GPA 5 and LSAT cells? 6 A. We're talking now about the actual computations that 7 are done? 8 Q. Yes. 9 A. The actual computations, the effect of those is that 10 you want to look at the composite estimate across 11 the cells with comparative information. 12 So, if you want to make sure that 13 you're controlling for the cells with comparative 14 information, then you would include what we call 15 indicator variables for each of those combinations. 16 Q. So, you created an indicator variable, well, the 17 computer did, for each of the cells? 18 A. The effect of that in the model is so that you can 19 estimate the odds ratio controlling for, that is 20 taking account without making any particular 21 assumption about the effects of LSATs and GPA. 22 Q. Are these variables sometimes called dummy variables 23 in your field? 24 A. We have cute things for things, yes. Did I use the 25 term indicator variable? GRUTTER -vs- BOLLINGER, ET AL 166 1 Q. Yes. 2 A. And you use the term dummy variable? 3 Q. Yes. 4 A. I suppose since they both indicate the same thing, 5 it would be a matter of one's persuasion to which 6 one you prefer I call it. 7 Q. Okay. I just wanted to make sure that they're the 8 same things, we can use indicator variables if you 9 like that. 10 And the purpose of using the 11 indicator variables was to control for GPA and test 12 scores? 13 A. To allow us only to use the comparative information 14 in getting our composites. That's the only purpose 15 in doing that. 16 Q. Am I right that you ended up with more than a 17 hundred variables in each of your models? 18 A. As many cells as there were in comparative 19 information. 20 Q. Do you know whether it was more than hundred in each 21 of these years? 22 A. Well, in the basic grid there's 120 cells. So, if 23 what you're saying now is I used more than a hundred 24 indicator variables, that mean I used more than a 25 hundred cells, which may be true. GRUTTER -vs- BOLLINGER, ET AL 167 1 Q. You don't remember how many variables were in there? 2 A. Precisely? 3 Q. Yes. 4 A. Of course not. 5 Q. Am I right also that the computer excluded cells 6 from this analysis if they were empty, if there were 7 no applicants? 8 A. It excluded all cells without comparative 9 information. 10 Q. So, those cells included cells with no applicants, 11 am I right? 12 A. Certainly. 13 Q. Okay. It also included the cells where all of the 14 applicants in both groups were treated the same as 15 we discussed earlier? 16 A. I want us to be clear. I think you were mixing up 17 analysis, so I want to make sure the court is clear 18 and everyone is clear. 19 In the models where we use the 20 controlling for LSAT and GPA, the cell grid, okay, 21 and the analysis that compared the various ethnic 22 groups, okay? 23 Q. Yes. 24 A. Then in that particular model we control for the 25 individual cell combinations. So, we did have, as GRUTTER -vs- BOLLINGER, ET AL 168 1 you would say, there are 120 such cells? Is that 2 you counted them, and you were right? 3 Q. I believe on your cell by cell analysis you had 240. 4 120 for Michigan residents and 120 for 5 non-residents? 6 A. Exactly. What you're doing is using the cell by 7 cell analysis when the odds ratios weren't computed 8 from that cell by cell analysis, they were computed 9 from the initial--analysis. 10 And in that one there are actually 11 120 cells. And what you're saying, which I believe, 12 is that of those 120 cells probably a hundred or so 13 contributed to the model. 14 Q. I'm not saying, I'm just asking you whether you 15 remember or know how many variables went into the 16 model? 17 A. Well, certainly if there are indicator variables 18 from each cell for which there were applicants, and 19 for which there were applicants from, at least, two 20 ethnic groups, which is what it would have to be. 21 All the applicants couldn't have been 22 in one ethnic group. And if there were individuals 23 both admitted in the volume cell, I would not be 24 surprised if there were about, in that analysis, at 25 about 80 percent of the cells, a hundred out of 120 GRUTTER -vs- BOLLINGER, ET AL 169 1 would contribute. 2 Q. But just so I'm clear. The computer excluded cells 3 that didn't fit into the three categories that you 4 just mentioned? Or cells that didn't have the three 5 characteristics, I should say, that you just 6 identified? 7 If a cell didn't have applicants from 8 both groups? 9 A. From two groups of the nine. There were nine. 10 Q. Two of the nine? 11 A. So, it had to have at least some representation from 12 two of the nine categories of ethnicity. 13 Q. Okay. And then there had to be both admitted and 14 rejected applicants, and at least two groups, am I 15 right? 16 A. There had to be--no, there had to be admitted and 17 rejected applicants. 18 Q. In the cell? 19 A. In the cell. 20 Q. And for cells where that was not the case, the 21 computer excluded the cells? 22 A. For cells not the case, in that case they did not 23 contribute to the odds multiplier estimation, that's 24 true. 25 Q. And for cells that if you just look at a cell by GRUTTER -vs- BOLLINGER, ET AL 170 1 cell comparison would yield infinity odds ratio, 2 those cells were included by the computer, am I 3 right? 4 A. There's comparative information, so they would 5 certainly be included. 6 Q. So, the computer excluded cells from which you say 7 there was never comparative information, and 8 included, at least, some cells where the odds ratio 9 was infinity, right? 10 A. Where the observed odds ratio was the infinity? 11 Q. Sure. 12 A. Sure. 13 Q. And the computer did that because that's the way you 14 programmed it? 15 A. Yes, that's the way I arrange for the computer to do 16 it. That's true. 17 Q. You didn't have to choose that methodology? 18 A. If I didn't choose that methodology, then I would 19 have had to make some other assumption about the 20 effect of LSAT and GPA on admissions. 21 And I did not, I wanted to do an 22 analysis where I made as few assumptions as possible 23 about LSAT and GPA. Because it was clear from the 24 grid cells that that was a fairly complicated 25 process relatively. GRUTTER -vs- BOLLINGER, ET AL 171 1 Q. But I just wanted to be clear, that the computer did 2 what it did because that's the way that you set it 3 up? 4 A. I do my own computing, I don't have a grad student 5 or anyone else, I do all of my own work. So, if the 6 computer did something it's because I made 7 instructions that would have it do it, that's true. 8 Q. And you could have chosen something else, this is 9 the way you decided to approach the answer? 10 A. Absolutely. It was my decision. 11 Q. Now, earlier this morning you talked about a 12 baseball analogy, of calculating a batting average. 13 And when a batting average is calculated, as I 14 understand it, all of the games are included, isn't 15 that right? 16 A. All of the games. 17 Q. All of the games that a particular person has played 18 are included? 19 A. Sure. 20 Q. Okay. But if some of the cells were not included in 21 your model, that's the equivalent of excluding some 22 of the games from the batting average, correct? 23 A. There's a flaw in my analogy, I agree. 24 THE COURT: In your analogy? 25 A. Well, in the analogy in the sense that-- GRUTTER -vs- BOLLINGER, ET AL 172 1 THE COURT: (Interposing) The 2 analogy in the sense of the baseball game? 3 A. In the sense that this situation is more complicated 4 than the baseball game, okay. And so in this case 5 we have cells of various sorts. There are cells 6 that basically where there's no comparative 7 information, everyone is admitted or everyone is 8 denied, everyone. 9 Here we're including only cells that 10 have comparative information. So, that is cells in 11 which there are, at least, members of two of the 12 ethnic groups. And cells from which some applicants 13 are admitted and some are denied. 14 But it doesn't matter if ones 15 particular group is all admitted, or one particular 16 group is all denied. So it's a more complicated 17 analogy and I agree with that. 18 BY MR. DELERY: 19 Q. Now, in terms of the way the cells were constructed, 20 I believe you said this morning that you took them 21 from Exhibit 16, what is now known as Exhibit 16? 22 A. And I knew it as Exhibit 16 I think then, yes. 23 Q. You could have picked cells that were larger then 24 what is reflected, isn't that right? 25 A. I could have. GRUTTER -vs- BOLLINGER, ET AL 173 1 Q. You could have picked cells that were smaller? 2 A. I could have. 3 Q. And the odds ratios that you report would have been 4 different if you had selected different cells, isn't 5 that right? 6 A. It would really be amazing if we did the 7 computations that came out to be exactly the same if 8 we made different cell boundary choices. That's 9 absolutely correct. 10 Q. So the odds ratio numbers that you report would be 11 different if you used different cells? 12 A. Absolutely. 13 Q. Am I right that as a general matter as you expand 14 cells beyond what you have, you would expect that 15 the observed difference between the racial groups 16 would decrease? 17 A. Please be more specific. 18 Q. Okay. For example, if you looked at the whole 19 applicant pool, and essentially took the whole 20 applicant pool as one being cell, the odds ratio 21 then would, in fact, be below one, isn't that right? 22 A. So, if we ignored credentials, ignored GPA and LSAT 23 and we said, okay, one big cell. Then we get a 24 different odds ratio, and that odds ratio you say 25 would be less than one, it would depend on the GRUTTER -vs- BOLLINGER, ET AL 174 1 admission rate? 2 Q. If the admission rates for minority students were 3 lower than the admission rates for non-minority 4 students, then the odds ratio would be less than 5 one? 6 A. If we made it one big cell, which means we're 7 ignoring LSAT and GPA credentials and calculated, 8 which you could do, then if what you say is true and 9 I don't have any reason to doubt it, then the odds 10 ratio calculated would be less than one, that's 11 true. 12 Q. So, if we start with your cells and expand them in 13 the direction of the whole pool being one cell, then 14 the odds ratio is going to move in the direction of 15 the odds ratio for the whole applicant pool, isn't 16 that right? 17 A. There's no guarantee that you would get exactly that 18 as you step along. But in the end you would 19 obviously wind up there, that's true. 20 Q. And based on your work with these data sets, you 21 don't have a view as to what would happen as you 22 move in that direction? 23 A. Well, actually my other analysis informs us, at 24 least, somewhat on that. The analysis where I 25 accounted for their thin cell GPA. So, as we move GRUTTER -vs- BOLLINGER, ET AL 175 1 to groups that are less specific with respect to 2 credentials, that is wider groups, that is so there 3 is more of a mix of individuals within. 4 It appears that the odds ratio in the 5 case where I control for the grid cell the odds 6 ratios went up. So, if we did the opposite of that, 7 it would seem if you use broader, that is made 8 groups that are of larger groups of credentials, 9 that you'd probably wind up with odds ratio that are 10 smaller. I think that's probably right. 11 But I didn't do that calculation to 12 know for sure that's exactly what would come out in 13 this case. 14 Q. All right. I believe you said a number of times 15 that your goal in this analysis was as much as you 16 could, to look at students who were similarly 17 situated in terms of credentials as you call it, 18 correct? 19 A. That's the goal of statistical, I call it principal 20 statistical fair comparison, that's what I would 21 call it. 22 Q. Okay. And if we could put up slide 37 from this 23 morning. If you look at the African American line, 24 the second one down here on the left, the left side 25 is your model that controlled only for grade point GRUTTER -vs- BOLLINGER, ET AL 176 1 average and LSAT scores, correct? 2 A. That's correct. 3 Q. And if we look at the line for African Americans, 4 your odds ratio there is 257.03, right? 5 A. Well, I'm not going to worry about the decimal, you 6 can't. 7 Q. 257, okay. On the right is your second model where 8 you added in some additional factors in addition to 9 GPA and test scores, right? 10 A. That's correct. 11 Q. And so you were controlling for more factors in the 12 model reflected on the right? 13 A. That's correct. 14 Q. So, in your view, the applicants being compared in 15 the model on the right, were even more similarly 16 situated then the applicants on the model on the 17 left? 18 A. I believe that's true, yes. 19 Q. And the African Americans odds ratio for the model 20 on the right goes up from 257 on the left to 513 on 21 the right, correct? 22 A. Correct. I mean those are both giant numbers and I 23 don't want to say they're very different as far as 24 factors goes. 25 Q. So, when you control for more factors you're getting GRUTTER -vs- BOLLINGER, ET AL 177 1 a larger odds ratio, right? 2 A. In this particular case we control for these 3 additional factors. I got a larger odds ratio 4 in--my expectation was that it may be that they 5 could go down. 6 And in many analysis where we do 7 control for additional factors, they go the other 8 direction. 9 Q. In other context in your experience they go down? 10 A. In statistical context that I have worked on, which 11 is a greater variety of context, yes, they can go in 12 either direction. I didn't say they would go down, 13 I said they could go either direction. 14 Q. In this case in each year the odds ratios go up, is 15 that right? 16 A. I'm not sure that they're in uniformity, I think 17 that's probably the case. 18 Q. At least in 1995? 19 A. Well, certainly n '95 we have that, and we can look 20 at the reports if you want to for the other years. 21 Q. Suppose that you had numerical information or all of 22 the factors that the Admissions Office considers, so 23 that you can bill all of the factors in the model 24 and control for them. 25 In that situation, wouldn't you GRUTTER -vs- BOLLINGER, ET AL 178 1 expect the odds ratio to approach infinity? 2 A. In what sense? I guess if I had additional 3 numerical factors, additional factors that we 4 control for? 5 Q. Yes. 6 A. You know, I just don't know. 7 Q. In other words, you wouldn't expect that in your 8 model as you add in other factors so that you got to 9 the point where you were controlling for everything 10 the Admissions Office considers other than race, 11 that the resulting odds ratios would not be 12 infinity? 13 A. I think the odds ratios may get large in this case. 14 I don't know if I had--you're talking in a very 15 hypothetical way, since we can't do this analysis in 16 only a hypothetical way. 17 But, in fact, boy if you ever have 18 such models, you've described the process perfectly, 19 that's right. So everything would be infinity in 20 sense of odds ratio. 21 Q. And the reason for that, and I'm just trying to 22 understand the way your approach works. The reason 23 for that is if you control for all the other factors 24 and are looking at people who are identical except 25 for the fact that some are minorities and some are GRUTTER -vs- BOLLINGER, ET AL 179 1 not, then any difference in the admission in those 2 two groups, the model contributes to race, isn't 3 that right? 4 A. If that's the only factor left and that's how 5 decisions were made, then that's what would happen. 6 Q. And the resulting odds ratio in that context would 7 be infinity, correct? 8 A. It would be large, it could be infinity. In our 9 hypothetical if we could make all the odds ratio 10 infinity for everything. 11 Q. Okay. And the reason it would be large or infinity, 12 is that the only factor left to explain any of the 13 difference would be race, right? 14 A. If that were the deciding factor, sure. 15 Q. And the odds ratio in that situation would be the 16 same, no matter how much race had been taken into 17 account by the person actually making the decision, 18 isn't that right? 19 A. If we have all the other factors that went into the 20 process, if we had that, which we don't here 21 certainly. But if we did have that, then I think, 22 in this hypothetical example you would wind up in 23 that situation. 24 Q. Okay. And this odds ratio analysis then, can't tell 25 us about how much race is taken into account by the GRUTTER -vs- BOLLINGER, ET AL 180 1 people making the decision, right? 2 A. It can't, is that what you're saying? 3 Q. I'm asking you if whether I'm correct that it 4 cannot? 5 A. It measures with respect to just what we have here. 6 It measures that when we take in account grade point 7 average, LSAT grid cells, how much race is taken 8 into account with respect to explained decisions 9 beyond those. That's what it explains, no more than 10 that. It's a description of the Admissions process. 11 Q. You think that this model is saying something about 12 how heavily race is being weighed by the person who 13 sits down and reads the admission file? 14 A. What I think is the aggregate effect of the 15 decisions made in the Law School with respect to 16 admissions. The aggregate effect is--well, for 17 instance if we look at what we can't see there, the 18 effect of residents, as far as making decisions. 19 That the effect of race, for 20 instance, is much greater and has a stronger effect 21 then Michigan residents. 22 And when we compare non-resident 23 minority applicants to resident majority applicants, 24 you can see that, in fact, decisions were made 25 strongly in favor of the minority applicants. So, GRUTTER -vs- BOLLINGER, ET AL 181 1 with respect to that, which is that factor, I could 2 say. 3 Q. You just said that the odds ratio say something 4 about the effect of taking race into account? 5 A. The effect that we see, right. You asked me a 6 different question, I'm sorry. 7 Q. I'm sorry. I want to make sure that I understand. 8 The effect that you see after the fact, looking back 9 at the decisions that have been made, because that's 10 what you're doing? 11 A. I don't think I prefer to do anything else other 12 than look at and try to understand the decisions 13 made by the Admissions Office, that's correct. 14 That's correct. 15 Q. In looking at the effect of a factor on the 16 decision, looking at them after they have been made 17 is different, you would agree from trying to 18 quantify how much a person sitting down to read a 19 file is weighing race? 20 A. I'm not sure there's a statistical way to quantify 21 what you're talking about. So, different, but 22 nothing that I can measure. 23 Q. So, statistically we cannot measure the weight that 24 a person sitting down to read a file accords race, 25 is that right? GRUTTER -vs- BOLLINGER, ET AL 182 1 A. What we see is the effect of whatever they have 2 done, however they have done their rating, however 3 we see the effect of those decisions. And what I'm 4 reporting is the effect of those decisions. 5 Q. Okay. And therefore, you are not reporting the 6 extent to which a person sitting down to read the 7 file is taking race into account, is that right? 8 A. Well, let me say just to be very clear. To be very 9 clear, and you can stop me if I'm not answering the 10 question appropriately. 11 But what I'm going to say is that, if 12 in fact, what we're seeing is in my estimation an 13 incredibly large effect with respect to race, 14 incredibly large. 15 If decisions were made, if decisions 16 were made by individuals that didn't take race into 17 account, to a great extent if I may use your word, 18 and use it in an English contest, then I can't 19 imagine that we would ever see effects this large. 20 And so what I'm saying is, that in 21 fact, what we're seeing are enormous effects. And 22 these enormous effects would be really difficult to 23 imagine that they were not made--decisions were not 24 made without taking race into account. 25 Q. I don't think anybody's disputing that the GRUTTER -vs- BOLLINGER, ET AL 183 1 University and the Law School take race into 2 account. My question is, whether the odds ratios 3 that you report are reporting the extents to which 4 the people sitting down and reading the admissions 5 file are taking or weighing race, or taking it into 6 account? 7 MR. KOLBO: Your Honor, I think I'm 8 going to lodge an objection. I think that's been 9 asked and answered. 10 THE COURT: He's answered it. You 11 think you have answered it? 12 A. I was willing to repeat the same thing. 13 THE COURT: No, that's all right. 14 BY MR. DELERY: 15 Q. I don't want you to repeat the same thing. Do you 16 have an opinion on how your odds ratios would 17 change, say, if the Admissions Office took race into 18 account half as much as it does right now? 19 A. As you well know, I can't give a numerical answer to 20 that, the odds multipliers would be smaller. I 21 think the way to think about that, and maybe there's 22 lots of ways to think about that and you can decide, 23 is that the range at which there are changes, such 24 as when we saw on the Admissions LSAT for a specific 25 grade point average that I showed you this morning? GRUTTER -vs- BOLLINGER, ET AL 184 1 If they took it into whatever half 2 means, I have to be careful and I'm sure you 3 understand exactly what you mean and I understand 4 exactly what I mean, and they may not exactly match. 5 But what we would have is we would 6 have more overlap in the area in which there 7 were--and the odds ratios would be smaller. 8 There would be more overlap in the 9 area where minority applicants were admitted and 10 rates at which they would be admitted would be 11 closer to the same rates as majority applicants. 12 Q. Let me move to a related point, but not exactly the 13 same point. I think you said this morning that these 14 odds ratios that you reported as a general matter, 15 are about the largest you have seen in your 16 experience? 17 A. They're big, yes. 18 Q. Largest you have seen in your years as a 19 statistician? 20 A. I don't recall a set of consistent analysis that 21 showed odds ratios like that, that's right. In this 22 large setting, that's right. 23 Q. And you think that they're really large, even though 24 only about a third of the minority students are 25 being admitted, isn't that right? GRUTTER -vs- BOLLINGER, ET AL 185 1 A. In the area in which decisions are made, these are 2 large effects, that's right. 3 Q. What would the odds ratio be if, instead of 4 admitting about a third of the minority applicants, 5 the University or the Law School admitted about half 6 of the minority applicants? 7 A. In the sense that they would consistently, and I'm 8 going to ask you, because I want to make sure I 9 understand how you're thinking. 10 That they would take individuals in 11 the grid and then go further down in the grid for 12 admission in the way, and then result in about half 13 of the minority students being admitted? 14 Q. They may have to go to some extent further down the 15 grid, or maybe they would take more minority 16 students from cells, you know, comparatively higher 17 up? 18 A. Sure. And so in general what would happen in that 19 case, I think, and I'll wait to see how you follow 20 up to decide whether I agree with that. In general 21 the odds ratios would go up. 22 Q. Okay. So, they're already as large as you've ever 23 seen, and if we admitted 20 percent more minority 24 students they would get-- 25 A. (Interposing) I think they would get bigger. They GRUTTER -vs- BOLLINGER, ET AL 186 1 would get bigger in the sense that if the same 2 pattern of admission goes on with respect to using 3 grade point average and LSAT which is clearly done, 4 then they would get bigger. 5 Q. Okay. Could we put up slide 36, please. These are 6 the relative odds for the model that included the 7 factors beyond grades and test scores. We talked 8 about this this morning. 9 If an odds ratio of two or three is 10 really large in your experience, what is the 11 significance of the 6.5, it looks like, for Michigan 12 residents? 13 A. I think I said two or three are the kind we design 14 for and think were important. I didn't say that I 15 have not seen a 6.5 before, in fact, I gave you an 16 example of 20, 19 or so in a typical example. 17 So, the 6.5 I think I said this 18 morning, that that is a pretty big effect. 19 Q. Okay. Would you call it a large allowance for 20 Michigan residents? 21 A. Would I? 22 Q. Yes. 23 A. I think it's a good size allowance for Michigan 24 residents, absolutely. 25 Q. I think this morning you mentioned that you GRUTTER -vs- BOLLINGER, ET AL 187 1 calculated composite odds ratio for each of these 2 groups for each of the years, for 1995 through 2000, 3 is that right? 4 A. That's correct. 5 Q. And I think we churned through some of those slides 6 pretty quickly this morning. But if I could, I 7 would like to go back to a couple. 8 Before I do that I should ask you, as 9 you understand it the Admissions policy that you 10 have reviewed adopted in 1992, has been in effect 11 throughout this period, 1995 through 2000, is that 12 right? 13 A. I don't know if anyone told me any different, I have 14 not seen another Admissions policy. So I don't know 15 anything to the contrary. 16 Q. Was that your understanding when you did your 17 analysis? 18 A. I think when I did my analysis I knew that I had 19 this Admissions policy. I didn't know if it was 20 necessarily the effect of Admissions policy, I 21 looked at the data in trying to understand the 22 description of what went done. 23 Q. Did counsel for the Plaintiffs discussed whether 24 they had stipulated that the policy had been in 25 effect during that time? GRUTTER -vs- BOLLINGER, ET AL 188 1 A. I'm sure I never heard that. 2 Q. Would you agree also that the number of applications 3 from members of the various racial and ethnic groups 4 was fairly stable over this period? 5 A. I think it is fairly stable, that's true. 6 Q. There's some fluctuation, but it's fairly stable? 7 A. These are data. If there's not fluctuation, then we 8 wouldn't believe the data. 9 Q. So, over the same time period as you calculate them, 10 the relative odds have varied quite substantially 11 for members of the minority groups? 12 A. I think that's the way I would expect, they would be 13 very substantial. 14 Q. Okay. If we could put up slide, I think it's 35, 15 which is the 2000 Relative Odds of Acceptance. We 16 have for African Americans 443 are the odds that you 17 calculate, is that right? 18 A. That's right. 19 Q. Okay. And if we then go to slide 32 for 1997, 20 instead of 443 you have 53.49? 21 A. That's correct. 22 Q. So, that means that the odds ratio for African 23 Americans in 2000 was about eight time greater than 24 it was in 1997, is that right? 25 A. That's correct. I have to say once odds ratios are GRUTTER -vs- BOLLINGER, ET AL 189 1 high they're high. And 50 is high, 400 is high. 2 And I would have to say very clearly that I think 3 that there's not a substantive difference when odds 4 ratios get as high as those. 5 Q. So you don't think there's a substantive difference 6 between 53 and what was it, 443? 7 A. I think they're both big, okay. And I think the 8 substance which is what we are talking about, is we 9 have to very careful and statisticians don't get 10 bogged down in numbers and, in fact, allow us to 11 understand the substance. In a substantive way, 12 these are both big numbers. 13 Q. Interesting to hear that statisticians say don't get 14 bogged down in numbers. But did you do anything to 15 look at whether the differences in the odds ratios 16 for these two years 443 versus 53, were 17 statistically significant? 18 A. I did not make a formal comparison. 19 Q. There is a formal test that could be done to test 20 the significance of that difference, is that right? 21 A. It's possible to do such a test. 22 Q. And you haven't done that? 23 A. I did not carry that test out. 24 Q. Okay. Would you be surprised to find out that the 25 difference between these two numbers, 443 and 53 is GRUTTER -vs- BOLLINGER, ET AL 190 1 more than eleven times the standard error of the 2 difference? 3 A. Yes, I would be surprised. I can't imagine given 4 the size of these that that difference is there. I 5 can do the calculation, if you would like. 6 Q. If it turned out that the difference between those 7 two odds ratios is more than eleven times the 8 standard error of the difference, would that concern 9 you as a statistician? 10 A. Would it concern me? 11 Q. Yes. 12 A. In saying that there is a statistically significant 13 difference between the Admissions policies in these 14 two years? 15 Q. Yes. 16 A. And thus say if I were concluding that I would say 17 that in 1997 they didn't give so much preference to 18 African Americans, and in 2000 they decided they 19 would give more preference? 20 Q. Exactly. 21 A. Would that concern me? 22 Q. Yes. 23 A. In the substantive part of this case, that doesn't 24 concern me because these are all large preferences. 25 Q. Okay. You don't think that that kind of instability GRUTTER -vs- BOLLINGER, ET AL 191 1 in the odds ratios would call the validity in your 2 model to question? 3 A. Actually the fact that we got large ratios no matter 4 if they're as different as you said, and the 5 consistency of them in the cross years actually 6 makes me feel very comfortable with the substantive 7 conclusions that we have drawn into these models. 8 Q. I believe you said near the end of your testimony 9 this morning, that you thought you had quantified 10 the role that race plays in Admissions, did I hear 11 that right? 12 A. Did I say I had quantified it? 13 Q. Yes. 14 A. I'm not sure I said that as a direct quote, I may 15 have said something to that effect. 16 Q. Do you believe that you have quantified the role 17 that race plays in Admissions? 18 A. I think what I have shown is that race plays--the 19 individual selected minority groups are given a 20 large allowance with respect to Admissions 21 decisions. 22 Q. How big is that allowance? 23 A. How big is that allowance? 24 Q. Yes. Can you put a number on it? 25 A. I can give you examples of odds ratio for similarly GRUTTER -vs- BOLLINGER, ET AL 192 1 situated individuals at various levels of GPA and 2 LSAT. 3 Q. But my question is, is it 443, or is it 53 for 4 African Americans? 5 A. And I would say that that specific number is in the 6 sense not important statistically, because they both 7 represent large allowance. 8 Q. So, we can't hang on any one of these numbers as 9 representing what the allowance that you say you 10 found actually is? 11 A. I mean I think it's difficult from year to year, 12 it's estimated to be different from year to year and 13 I think that's the nature of the process. 14 Q. Did you calculate competence in the roles around the 15 odds ratios for any particular year? 16 A. I probably did at some point look at competence 17 levels, yes. 18 Q. You don't remember what they are? 19 A. Well, I remember they're wide and they're fairly 20 wide and that's to be inspected. 21 Q. So, in other words, you have given us an odds ratio 22 on these charts, and maybe we should put the last 23 one back up for 2000, 443. 24 But really all we can know, or all 25 that your model can tell you is that it's likely to GRUTTER -vs- BOLLINGER, ET AL 193 1 be 443 plus or minus something, right? 2 A. I think there would be a range, of course, that's 3 what we would expect if we get in competence levels 4 we get a range. And that range would be, I think, 5 fairly large. But that range would be far away from 6 one. 7 Q. Okay. The fact that the range is far away from one 8 is important, because that tells you that this is 9 not happening by chance, correct? 10 A. Certainly. 11 Q. Okay. But other than saying that "true odds ratios" 12 is something far away from one so that this effect 13 is not happening by chance, you can't really tell us 14 where around 443 we can be confident with the odds 15 ratio of one? 16 A. We can do calculations and present those, we can do 17 that. We didn't do that in our report. 18 Q. Okay. And you'd would agree that you really need to 19 know the competence intervals in order to know how 20 much weight we can put on a number like 443, 21 correct, in 2000? 22 A. I think that you can calculate competence intervals, 23 I think it's reasonable to look at competence 24 intervals. Yes, I agree with that. 25 Q. But you haven't reported that? GRUTTER -vs- BOLLINGER, ET AL 194 1 A. I did not report that, correct. 2 Q. I'd like to talk for a minute about how the policy 3 of taking race into account in Admissions, affects 4 the non-minority student, the so-called majority 5 student. 6 Are you saying that the current 7 policy has a big effect on the likelihood of 8 admission for majority students? 9 A. Go back to the word likelihood again. 10 Q. Okay. 11 A. So, you're saying does it change the probability of 12 admission for a majority student by a lot? 13 Q. Much better way to ask the question. 14 A. I apologize for correcting, for suggesting. And I 15 think that, in fact, with respect to the aggregate 16 numbers, I think that it would not affect the 17 aggregate numbers a lot if the policy were changed. 18 The aggregate numbers. 19 Q. And that's because there's so few minority 20 applicants that we're talking about, right? 21 A. I mean there are a fair number of minority 22 applicants. I mean there's hundreds of minority 23 applicants. 24 Q. I should say so few of minority students who are 25 admitted? GRUTTER -vs- BOLLINGER, ET AL 195 1 A. Well, it's the simple math is if a majority student 2 is denied admission in favor of a minority student, 3 that affect that majority student. And it's a one 4 for one. 5 And now, when we're talking about 6 percentages with different denominators, which is 7 what you're doing. It's surely true that if you 8 change 50 students, then the group with the larger 9 denominator will have a smaller percentage change. 10 And I think that's all you're saying. 11 Q. I think that's right. But just to make sure, let me 12 do it this way. 13 You're aware that there are only 14 between something like 165 and 190 minority students 15 who are admitted in any of these years, right? 16 A. I don't remember the numbers but that sounds in the 17 right range, that's true. 18 Q. Okay. So, even if every minority student were 19 rejected so they took nine, at most, that would mean 20 say 190 additional majority students who could be 21 admitted? 22 A. I think we're talking the same language. It's one 23 for one, yes. Absolutely. 24 Q. So, given that that's 190 majority students out of 25 2700 or 3000 majority applicants, the possible GRUTTER -vs- BOLLINGER, ET AL 196 1 change and the probability of admission for the 2 majority student is small? 3 A. The proportion of majority students who would be 4 admitted would go up by a smaller amount, that's 5 right. Well, the denominator is bigger so the same 6 number means smaller percentages. 7 Q. Did any of your, I guess, I should do it this way. 8 Your analysis did not look at what would happen if 9 the University did not take race into account in 10 Admissions, is that right? 11 A. That's correct. 12 Q. Well, I think now I have only one more thing to ask 13 you about on odds ratios, and then we'll look at 14 some of your other graphs. But I want to go back to 15 this idea of what these odds ratio numbers really 16 mean. 17 If we look back at the 443 number, is 18 it fair to say based on this number in your view, 19 that an African American had a more than 400 times 20 greater chance of admission than a white student? 21 A. The odds of admission for, the odds of admission in 22 terms of odds, the odds of admission for an African 23 American student situated at the same credential of 24 LSAT cell, GPA cell. 25 That, in fact, there's evidence that GRUTTER -vs- BOLLINGER, ET AL 197 1 there's about 400 times greater observed admission. 2 These are descriptions of what actually occurred, 3 and so that that's apparently what happened in those 4 cells. 5 Q. If the lawyers for the Plaintiff said that based on 6 this number, an African American had a more than 400 7 times greater chance of admission than a white 8 student, would that be a correct statement? 9 A. No. 10 Q. If we could put up, I want to turn now to the 11 curves, the graphs of select index versus 12 probability of admissions that we talked about. If 13 we could get slide 39, I think it is, David. 14 We obviously talked about these 15 earlier, and I'm going to try to be brief about 16 this. 17 The probabilities of admission that 18 you have on the vertical access here, those are not 19 the observed probabilities of admission for people 20 with a given index score, are they? 21 A. No. I described it was an estimation of the 22 function of probability admission versus selection 23 rate, yes. 24 Q. That prediction is from one of your regression 25 models? GRUTTER -vs- BOLLINGER, ET AL 198 1 A. This technique actually has the name regression in 2 it, it's called isotonic regression. So they're 3 estimates from an isotonic regression model. 4 Q. So, the probability of acceptance that you're 5 graphing, I think you said this morning is really 6 the maximum likelihood estimate for the probability 7 function of Admissions, is that right? 8 A. That's actually right, I think. 9 Q. So, you didn't just look at the people who had a 3.0 10 index score and figure out what proportion of those, 11 that's not what this graph reflects? 12 A. This graph estimates the probability of admission as 13 a function of select index in a way that the 14 function will go up and to the right. 15 Q. And it's not the observed data? 16 A. The observed data? 17 Q. Yes. 18 A. Well, I mean, there aren't many applicants with 19 specific scores. So it would not be the observed 20 data. 21 Q. The index score across the horizontal axis, I think 22 you explained this morning is a combination with 23 some coefficient of grades and test scores, right? 24 A. That's correct. 25 Q. Is it your understanding that the Law School GRUTTER -vs- BOLLINGER, ET AL 199 1 Admissions officers actually look at this index 2 score when they're making decisions? 3 A. Is it my--what I know about the index is it 4 certainly referred to an Admissions policy. And 5 it's included in the data base of information I was 6 given. 7 So, that index is in the Admissions 8 policy, and I actually don't know what Admissions 9 officers or decision makers have in front of them, 10 no. 11 Q. You haven't actually looked at the application files 12 that went with those numbers that you showed us 13 earlier? 14 A. I have looked at some application files, that's 15 true. I haven't looked at--for instance, for 1995 I 16 did not look at 4500 application files. 17 Q. But you looked at some of the samples of the files 18 that were produced, is that right? 19 A. I have looked at some files, yes. 20 Q. Did you see the index score reflected anywhere in 21 that sample file? 22 A. I should remember that, sorry. The answer is I 23 don't recall whether it's there or not, I really 24 don't know. 25 Q. I think you said earlier that your understanding, as GRUTTER -vs- BOLLINGER, ET AL 200 1 a general matter, is that as the index score 2 increases, the probability of admission should 3 increase, is that right? 4 A. I think that's actually the Admissions policy. 5 Q. And so these graphs actually reflect that 6 assumption, don't they? 7 A. Sure. Actually these graphs reflect the data which 8 follow that assumption, yes. 9 Q. But you, I think, the phrase is kind of constrained 10 these curves to be non-decreasing, is that right? 11 A. I constrained these curves to be monotonic. That 12 means, if in fact, the policy--if it turned out that 13 fewer people were admitted as a function of selected 14 index, then they would go down into the right 15 consistently, and if more they would go up into the 16 right. 17 And we can see that there's variation 18 in the middle, certainly in the places where there's 19 zero and ones. All individuals beyond those points 20 were either admitted or denied. 21 Q. Okay. Now, I thought you said this morning that, in 22 your view, these graphs don't reflect any 23 assumptions, did I misunderstand that? 24 A. Any assumptions, I see. And you're going to say-- 25 Q. (Interposing) I'm wondering, because monotonicity GRUTTER -vs- BOLLINGER, ET AL 201 1 sounds like an assumption to me. And I thought you 2 had said earlier that the advantage of these graphs 3 is they didn't reflect any assumptions? 4 A. So, if in the sense that I did assume that as the 5 Admissions policy said, that however the admission 6 was related to monotonically, I didn't say how. 7 And so, these are what we call 8 non-parametric curves, they make a minimum of 9 assumptions. But as you have indicated, all of the 10 techniques involved some assumptions and there is 11 one there. 12 Q. And so the effect of your monotonicity strength, if 13 that's the right phrase, is that we don't ever see 14 the curve jog downward, right? 15 A. That's what monotonicity means, yes. 16 Q. But, in fact, from looking at the grids, we know 17 that some students in high cells, comparatively 18 higher cells are denied admission, whereas more 19 students in a lower cell might in one year happen to 20 be granted admission, right? 21 A. That's certainly in the grid cells, that certainly 22 occurs. 23 Q. So, these grids simplify the Admissions pattern in a 24 way that removes a lot of the detail, don't they? 25 A. In fact, the steepness of them indicates there is a GRUTTER -vs- BOLLINGER, ET AL 202 1 monotonicity assumption, but, in fact, they go up 2 relatively sharply. 3 So, there is variation, but that 4 variation is indicated by the lack of straight line 5 up. I think I said that this morning. 6 Q. And doesn't your monotonicity assumption also 7 exaggerate the consistency of the spread between the 8 two curves? 9 A. I don't think that's true. 10 Q. In other words, if you graph the actual observes 11 probabilities of admission, wouldn't you see some 12 situations in which the curves are closer to each 13 other than the ones you have here? 14 A. Further and far apart, yes, both. We would see more 15 variation, see a very jagged curve, yes. 16 Q. We talked earlier about whether the effective race 17 in admissions varied depending on where a particular 18 applicant was, along the range of grades and test 19 scores, do you recall that? 20 A. Sure. 21 Q. Isn't it true that this graph suggests that that 22 effect is greater in the middle than it is on either 23 end? 24 A. Greater in the middle in what sense? 25 Q. In other words, this graph shows as you have created GRUTTER -vs- BOLLINGER, ET AL 203 1 it, a larger gap between the two curves in the 2 middle range of selection index, than in either the 3 low end or the high end, isn't that right? 4 A. I mean at the very extremes there's no gap in the 5 sense that there's a hundred percent. But if we 6 look at the horizontal distance for various 7 probabilities, if you do that, I don't have a laser 8 pointer anymore. 9 But if we looked at the horizontal 10 difference, I mean look at the ones down at the 11 bottom. The horizontal difference, that's a pretty 12 big gap down there. And then there's a jump up and 13 then there's a gap. Actually in the middle it's not 14 very far apart. 15 And then at the high end the 16 horizontal gap gets bigger. So, in respect to the 17 selection index, there's actually, you know, it 18 varies and it's going to vary, because these are 19 real data. 20 But I don't see that their 21 necessarily much closer together at the top than 22 they are in the middle. In fact, it looks to me 23 like they might even be further apart in this 24 particular example. But it's going to look 25 different for different groups. GRUTTER -vs- BOLLINGER, ET AL 204 1 Q. The vertical gap is the one that reflects the 2 difference in the probability of admission, correct? 3 A. The vertical gap sure, with respect in the 4 probability scale. Not in the on scale, but in the 5 probability scale. 6 Q. Correct. So, up until about 2.75 here on the left 7 side of the graph, the probabilities of admission 8 for the two groups are quite similar? 9 A. Up to 2.75? 10 Q. Yes. 11 A. You mean in the sense that there are no Caucasian 12 Americans admitted up to about 2.75? If I may 13 follow with that, there are no Caucasian Americans 14 admitted up to about 2.75. And from 2.3 on there 15 are Natives Americans admitted. 16 So, in fact, the odds ratio in that 17 case is infinity, even though the probabilities are 18 relatively close. Ten percent versus zero percent. 19 Q. So ten percent versus zero percent is the same in 20 this content as in infinite odds? 21 A. That's what we're seeing in this particular plot. 22 Q. And then at the upper end above, say, 3.4 certainly, 23 it's a little hard to tell from here. Both groups 24 have a probability of admission over 90 percent? 25 A. That looks to be true, yes. Correct. GRUTTER -vs- BOLLINGER, ET AL 205 1 Q. Whereas in the middle range there is more of a 2 difference between the probabilities of admission? 3 A. Both are admitted. But, in fact, for Native 4 Americans is a hundred percent admission rate from 5 three on. So, with respect to looking at odds, 6 again we're in a situation where there are no denied 7 Native Americans who is beyond, say, three. 8 Q. Now, this graph doesn't tell us anywhere how many 9 applicants fall at any particular point along the 10 index score, right? 11 A. On the graph? 12 Q. Yes. 13 A. No, that's not displayed, the number. 14 Q. Here we don't know how many applicants are in the 15 range where there's a separation between the two 16 curves, and how many are in the ranges where there's 17 no or very little separation, correct? 18 A. From the graphs, that's true. 19 Q. If you could now, David, I would like to look at 20 slide 14. Which is one of your box plot for grades, 21 is that right? 22 A. Yes. 23 Q. Would you agree, Dr. Larntz, that the grades for 24 admitted students of all ethnic groups are quite 25 high? GRUTTER -vs- BOLLINGER, ET AL 206 1 A. Quite how compared to what? 2 Q. Quite high along the standard 4.0 grading scale? 3 A. I guess I'm not familiar with anything other 4 than--these certainly look like good grades. Yes, I 5 agree with that. 6 Q. And, in fact, the median GPA for the majority 7 students runs in the range of an A minus, doesn't 8 it? Around 3.6 something? 9 A. Is that an A minus? I have to be careful. In 10 Minnesota we didn't use plus and minuses, so I have 11 a limited familiarity with that. 12 But they did put it in the year I 13 left. That wasn't the reason I left. 14 Q. But you would agree that all of these grades were 15 quite high, this is a qualified applicant pool 16 across these ratios? 17 A. That I would agree it's qualified? 18 Q. Yes. 19 A. This is a comparison plot, and what the plot says is 20 how the grades compare. And they compare--the 21 overall level of the grades are what they are, 22 they're numbers. 23 Q. So, you would leave it to the Admissions Office to 24 determine whether, in fact, all of these ranges are 25 what they would consider to be highly qualified? GRUTTER -vs- BOLLINGER, ET AL 207 1 A. Well, I certainly am not going to say that I know 2 that they're highly qualified or not, I don't know 3 that for a fact. 4 Q. So that issue is to be left to others? 5 A. Highly qualified in a statistical sense, I haven't 6 done no analysis to quantify that. 7 Q Fair enough. Let's look at the next one, I think, 8 slide 15. Same thing for LSAT scores, have you 9 looked at where the medians, the various medians 10 that you report fall on the percentile scale for the 11 LSAT? 12 A No. 13 Q So, you have no view, again, about how "high" all of 14 these test scores are? 15 A These are comparative plots, that's true. 16 Q Did you consider how the gaps between the medians or 17 between the boxes compare to the arrow of 18 measurement of the LSAT? 19 A I don't think that's relevant to looking at the 20 comparison. In fact, I understand that there would 21 be a test retake variation in taking the test. But 22 what is summarized here are the coretiles and the 23 medians, in particular, of a large, mostly, for a 24 large group of individuals that were admitted. 25 Q But you understand that at some point fine GRUTTER -vs- BOLLINGER, ET AL 208 1 differences between two applicant's LSAT scores 2 don't mean very much, right? 3 A What I know which is something about this, but not a 4 great deal. What I know is that Admissions 5 decisions are made based on these LSAT scores, I do 6 know that. 7 And I do know that the Admissions 8 decisions seem to be taking account of specific 9 points of the LSAT, we saw that in the analysis. 10 Do I know substantively how much a 11 difference of a certain amount of point needs, the 12 answer is, I don't know. 13 Q Could we go back, David, to slide 14. Did you do 14 anything to look at the absolute numbers that fall 15 within, say, the lower ranges of these box plots? 16 That's a bad question, I'll rephrase it and be more 17 specific. 18 If you look here at the second 19 column, which is for African Americans, I think you 20 said this line is the 25th percentile? 21 A That's correct. 22 Q And this bracket reflects the lower range, the lower 23 end of the range of normal value, is that right? 24 A That's how I describe it, yes. 25 Q And is that about another 15 percent of the GRUTTER -vs- BOLLINGER, ET AL 209 1 students? 2 A Well, I think it's probably closer to another 25 3 percent. 4 Q Well, I guess I should ask it this way. There's a 5 total of 50 percent of the admitted students in the 6 colored part of your chart, is that correct? 7 A That's right. The box represents a group that 8 corresponds to the 75th percentile and 25th 9 percentile. So, half of the students are within the 10 box, and thus half of the students are outside the 11 box. 12 Q Okay. And the other half of the students, for the 13 most part, are within the brackets, correct? 14 A Oh, sure, absolutely. 15 Q Except for the occasional outlier, I think? 16 A And the outliers are defined relative to be actual 17 box themselves. So, once an outlier in one column 18 differs from once an outlier in the other. 19 Q So basically the entire group of accepted applicants 20 is between these two brackets? 21 A It looks like with one exception. Again, let me be 22 very clear. It's possible there were two or three 23 at that particular level, but it's doubtful. It's 24 doubtful, it's probably wrong. 25 Q Did you do anything to compare, say, the number of GRUTTER -vs- BOLLINGER, ET AL 210 1 applicants in this 25 percent range here, as against 2 the number of outliers over here, for example? 3 A The wrong number? 4 Q Yes, the wrong number. 5 A And I'm going to have to--my recollection is there 6 are about 450 African American applicants in this 7 year, so there would be about 25--I can't do the 8 math, sorry. About a hundred in that range down 9 there, about a quarter of them, 25 percent. 10 And then you count the number 11 of--now, of course, the number of applicants of, 12 Caucasian applicants, is over 2000. It was like 13 2300 or something, I think that's right. 14 And so did I do any comparison of the 15 actual numbers of Caucasian applicants that are in 16 that same range with the hundred lowest African 17 American GPA? 18 Q Right. 19 A No. 20 Q You would agree that based on your analysis 21 undergraduate GPA and LSAT scores are very important 22 in the Admissions process? 23 A Certainly. 24 Q And that's what you've concluded based on your 25 models, right? GRUTTER -vs- BOLLINGER, ET AL 211 1 A Actually I concluded that based on looking at the 2 Admissions policy and also looking at the grids to 3 begin with, I think the grids basically tell the 4 story. They may tell all the story. But it's 5 certainly clear from the grids. 6 Q Your models tell the same story, don't they? 7 A With respect to those? 8 Q To the importance of grades and test scores in the 9 Admissions process? 10 A The models make virtually no assumption about that. 11 So I don't want to say that my models say that. I 12 do know from things that weren't presented that LSAT 13 and GPA are very important in the Admissions 14 process. But not from what you have presented 15 today. 16 Q But for the work that you did for this case, your 17 models leads you to conclude that grades and test 18 scores are very important in the Admissions process, 19 right? 20 A It's very clear they were. 21 Q And, in fact, maybe not in what you presented here 22 today, but in the models that you report, your 23 models show that grades and test scores have the 24 strongest assocation with Admissions decisions, 25 isn't that right? Strongest association of any GRUTTER -vs- BOLLINGER, ET AL 212 1 factor? 2 A I'm sure that that's true, even without quantifying 3 it specifically that their grades and LSAT are 4 important for all the applicants, yes. 5 Q For all applicants, regardless of racial-- 6 A (Interposing) Certainly if you look at the grids, 7 you can see within each racial group, you can see, 8 each ethnic group, you can see the effects and the 9 effects are very strong within each group. Although 10 the point at which admissions are done are different 11 for the two groups. 12 Q And, in fact, you believe from your models, don't 13 you, that grids and test scores have stronger 14 associations with admissions than an applicant's 15 race, is that right? 16 A It's certainly true if you look at the grids and 17 from my models, that individuals that are at very 18 low grades and very low LSAT will not be admitted, 19 no matter what. 20 Individuals with high grades and high 21 LSATs will almost certainly be admitted if they are 22 members of a selected minority group. And have a 23 higher chance, but not the same certainty as an 24 admission for a majority non-selected minority 25 group, that's right. GRUTTER -vs- BOLLINGER, ET AL 213 1 Q Would you agree that based on your models, grades 2 and test scores are the most important factor in 3 Admissions? 4 A The most important factors? 5 Q Yes. Is that what you conclude from your model? 6 A I don't know that I quantified it specifically, but 7 I think they are very important. 8 Q In fact, didn't you conclude based on your models 9 that grades and test scores are more important than 10 race in the Admissions process? 11 A Grades and test scores are more important than race? 12 Q In the Admissions process? 13 A In a statistical measure? 14 Q Yes. Based on your models? 15 A I would think that probably if I did a quantitative 16 measure which I didn't do particularly, that I would 17 find that that's true. In the sense that what my 18 models are doing is looking at the effected race 19 beyond the grid cells, and I assume that the grid 20 cells were important to begin with. 21 Q So, the results of your statistical analysis are 22 that grades and test scores are more important in 23 the Admissions process than race, is that right? 24 A I would say in my statistical analysis starts with 25 the assumption that we want to look at the GRUTTER -vs- BOLLINGER, ET AL 214 1 individuals with the same grades and test scores. 2 That premise does say that grades and test scores 3 are, in fact, important. And I believe they're 4 important. 5 To say they're more important in given 6 situations, I looked at comparisons of ethnic groups 7 by combination of grades and test scores. 8 I did not look--I did also look at 9 grades and test scores by ethic groups, but I don't 10 want to say if I ever looked at the Admissions 11 process without taking grades and test scores into 12 account. 13 Q I just want to make sure that I'm understanding you. 14 Are you saying that grades and test scores, according 15 to your models are, are not more important than race 16 in the Admissions process? 17 MR. KOLBO: Your Honor, I need to 18 pose an objection. This has been asked and answered 19 THE COURT: Sustained, it has been. 20 BY MR. DELERY: 21 Q I think, Dr. Larntz, you ended your testimony this 22 morning by saying that you conclude that you found an 23 incredibly large allowance given to selected minority 24 applicants in the Admission process; is that a fair 25 summary? GRUTTER -vs- BOLLINGER, ET AL 215 1 A In comparison to other applicants, that's true. 2 Q And you think you have quantified that allowance? 3 A I think that allowance is there, it's large, and I 4 have quantified it in a statistical sense year by 5 year. And the substance of the conclusions remain 6 year by year, yes. 7 Q And you quantified it based on your models? 8 A Yes, based on my models and the examination of the 9 grids and the whole composite of that. The whole 10 set of analysis I did, yes. 11 Q Do you think you quantified it even though your 12 models don't include all the factors the Admissions 13 Office take into account? 14 A Even though we don't include all factors--which by 15 the way, all statistical analysis there--I don't 16 know of any statistical analysis that account for all 17 possible factors. 18 Q You think that it is a quantification of the role 19 that race places into admission, even though you have 20 excluded the number of cell from the model? 21 A I think what I did is and an appropriate statistical 22 analysis for the comparison. Most of the cells that 23 were included, most of the individuals that we talked 24 about are in very low combination of the LSAT and 25 GPA. GRUTTER -vs- BOLLINGER, ET AL 216 1 There are very few at the upper end 2 that are included, most are at the low end and where 3 there were no students admitted. So there was no 4 comparison made. 5 Q And do you think that you have quantified the role 6 that race plays in admission, even though the number 7 that you give for that roll varies from year to year? 8 A Certainly. I think that the degree which varies 9 from year to year is not surprising. And certainly 10 I think that the fact that it's substantively the same 11 from year to year is confirming. 12 Q And that's true despite the fact that the Admission 13 policy hasn't changed in these years? 14 A The Admission policy hasn't changed. But, in fact, 15 what I'm trying to describe is admission to the curve 16 during those years regardless of what the policy was. 17 Q And you have done that? 18 A I have done the best I could, yes. 19 MR. DELERY: Thank you. I have no 20 further questions, your Honor. 21 THE COURT: Okay. You want to start 22 today or you want to do it in the morning? 23 MS. MASSIE: The morning. 24 MR. KOLBO: Your Honor, I do want to 25 alert the Court to the fact that Dr. Larntz has a GRUTTER -vs- BOLLINGER, ET AL 217 1 plane he has to get tomorrow at 1:00. And I assume 2 that's not going to be a problem. 3 We're happy to continue tonight if the 4 Court wants to do that. But I didn't want to 5 surprise the court tomorrow. 6 THE COURT: I'd love to do it tonight, 7 I thought we would finish him today. I have made 8 some plans and it's too late to cancel right now. 9 Let's start at 8:30 tomorrow morning. 10 MR. KOLBO: Is it my understanding, 11 your Honor, that Dr. Larntz will be able to leave at 12 least by noon tomorrow? 13 THE COURT: Yes, so he can get to the 14 airport on time. If there is some reason we don't 15 finish with him, we'll do it by video conference 16 somewhere during the trial, since we have had a 17 chance to see him. 18 We can give him the exhibit books, we 19 can work it out. I think the federal court--where 20 do you live? 21 A I will be traveling a good deal for the next few 22 weeks. 23 THE COURT: Okay, we will set it up 24 for a video conference. All of the federal courts 25 have them now. We can start him at 8:30 tomorrow and GRUTTER -vs- BOLLINGER, ET AL 218 1 then we'll make sure that we can get him out. 2 MR. KOLBO: I appreciate that, your 3 Honor. 4 THE COURT: I will be here, so it's up 5 to you guys. 6 (Court adjourned at 4:45 p.m.) 7 - - - 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 GRUTTER -vs- BOLLINGER, ET AL