The 2nd meeting of the Education Assessment and Accountability Review Subcommittee was held on Friday, November 17, 2000, at 10:00 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator David K. Karem, Co-Chair; Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, and Tim Shaughnessy; Representatives Mary Lou Marzian, Frank Rasche, and Mark Treesh.
Guests: Dr. John Poggio, and Dr. James Catterall(by phone), National Technical Advisory Panel; Gene Wilhoit, Bonnie Brinly, Helen Mountjoy, Scott Trimble, and Hunt Helm, KDE; Dudley Cotton, LRC; Clyde Caudill, KASA; Libby Marshall, Alicia Sells, KSBA; Bob Shimer, OPM; Ken Henry, Jerry Lunney, OEA; Cindy Heine, Prichard Committee; Mary Ellen Wielderwohl; Carol Grissett; Sam Crawford, Ky Farm Bureau, and Cindy Rausche.
LRC Staff: Sandy Deaton, Audrey Carr, Ethel Alston, Evelyn Gibson, Anne Armstrong, Kelley McQuerry, and Lisa Phelps.
Senator David Karem moved approval of the October 12, 2000 minutes and Representative Marzian seconded the motion. The motion carried.
Representative Moberly introduced the guests and then turned the meeting over to Helen Mountjoy, chair, Kentucky Board of Education. Ms. Mountjoy reviewed the past year of the assessment and accountability program. The assessment system is in place; the biennial scores have been received and there were very few questions about the process that was used. This time last year there were concerns with the interim accountability system and how it might fail. She said Senator Shaunessy was clear as to how it should be explained, and it has been accepted in the field as a one time interim accountability model. The long term accountability system is ready to be put in place. There is also an audit process for the low performing schools. The audit process has been piloted and two-thirds of the audits for this year have been completed.
Ms. Mountjoy also commented on a major standards setting process that has included 1700 teachers to make sure that everyone understands and agrees on what is meant by novice, apprentice, proficient, and distinguished in all subject matters, and at all grade levels. The Kentucky Department of Education (KDE) can move forward knowing that everyone is committed to the idea that proficiency is not just a goal for the school, but is an achievable goal, one that people can understand and work toward.
Ms. Mountjoy said there has been work on the longitudinal model. KDE, along with the national panel, has done research on models that might be more effective and have begun piloting those processes. KDE is ready to tell the committee where they are and how they got where they are. HB 53 is in place and the department is ready for advice.
Commissioner Wilhoit explained the longitudinal model and the two different studies. The first model takes the student results from a component of the norm reference assessment, goes through a process of setting a predictable level of performance on the core content portion of the assessment, and forms a comparison of those two scores to make a judgement of where students are in terms of where they should be. The second model uses the Kentucky core content portion of the test and re-administers it to a portion of low performing students the next year to see if they have advanced or declined.
Commissioner Wilhoit said that the first model, by using the norm reference test to predict scores on the core content portion of the test is efficient and can be carried out very easily without more testing, but it has a validity problem because it is difficult to equate across the two tests.
The second model is technically more sound and the technical panel can defend it better than the first. The problem with the second model is feasibility, it is more expensive, and readministering of the core content portion is more expensive than re-administering the norm reference. There were concerns in the field with the difficulty in administering the test. Other concerns related to singling out low performing students. There is also the question that if one re-administers the fourth grade exam in the fifth grade, is the curriculum being pushed to a fourth grade level rather than a fifth grade level.
Commissioner Wilhoit said that neither of the models has produced support in the field or produced the validity that is wanted. He said they need advice on where to go from here.
Representative Treesh asked if the method on the first model was to take the norm reference portion and let it be a predictor of the core content, so that it is not really testing at two different times, but finding correlation between the two.
Commissioner Wilhoit agreed and said that the advantage of that model is that it does not require additional assessments.
Ms. Mountjoy added that it is at two different points, the norm reference test would be given at grade three and the core content test at grade four.
Commissioner Wilhoit said that both models would meet the test of one year and repeating an assessment the next year to make some judgement of the progress of students.
Representative Treesh said there is not enough overlap between the two tests to be able to say that there is real high correlation.
Dr. Poggio added that one ends up with a model that does not do a good job of evaluating school effectiveness. When the National Technical Panel reviewed the data, the conclusion was that if a model was desired to be implemented right away then it would work, but it should not count in the accountability system.
Representative Treesh said that in second model the test is being given at two different grade levels and at two different points. He asked if the longitudinal model would consist of just the fourth and fifth grades or other grades.
Commissioner Wilhoit said that the application is possible, but could be limited to the elementary level, the middle school level or the high school level using the same format. There are things that are being done about student progression through the grades. The state is in the process of developing annual benchmarks and content benchmarks for classes and the grade level performance. It is a policy decision about whether to limit the longitudinal study to an elementary sequence or if to re-administer it to a elementary-middle or middle-high school sequence.
Dr. John Poggio said that if one were to go for retesting at the higher grade levels for students that performed poorly or the retesting of all students, then there are a number of variations at the high school level on both models. The problem is that the content changes at the middle school level and problems occur. Because of the curriculum change, the panel has suggested that to continue the pursuit of testing and retesting of poor performing students or all students in a subsequent grade, that testing should begin at the elementary level and stay focused on reading and mathematics and work through the feasibility management problems before carrying it on to the high school level. At the higher levels the relationships will diminish and the ability to evaluate effects of schooling will be set aside.
Dr. James Catterall said that the panel sees a great number of trade offs involved in the decision making. A predicted model has a longitudinal feel to it, and adds a symbolic longitudinal component or testing and retesting using the Kentucky core content battery in given subjects. The more that is done of that causes problems with retesting at the second tested grade. There are difficulties at every angle and the panel has been trying to be sure that we are trying to assess the possibilities. It seems to come down to a trade off of getting more valid information vs the burdens that are placed on the system. One of the things the panel has explored is why longitudinal aspects should be added to the process.
Dr. Catterall said that the panel wants to do things to get the system in a stable and constant form so that the challenges in the testing system will not be issues for the schools as we continue throughout the years. The other challenge is getting something in place relatively quickly with stable scores. The key question is what the committee believes the purpose of a longitudinal component is and if one should be added to the test.
Representative Treesh stated that by re-administering the core content test, the same form shall be used but it not have as much correlation as before.
Dr Catterall said that there would be a mixer form used by different students that would be given out randomly, and that it is not an issue.
Dr. Poggio added that the forms are built to be parallel and it would make no difference as to what form is given, if all are brought to a common scale.
Representative Treesh said that we should look at success factors to determine how students are doing. He said that reading at grade level is one key success factor, as well as mathematics. Something that needs to be focused on is tracking how well students are reading at grade level. It may be first through eighth grade and it might give better results than anything else.
Dr Poggio said it was suggested that another year be spent to study what instruction has to do to guarantee the success of the child. Given another year, the model could be evaluated so that it creates an intervention that leads to success, rather than giving the test again and seeing what happens this time.
Dr. Catterall added that the panel would agree that reading is one of the most saleable ideas. The reading curriculum is fairly similar across the grades. Reading is always a focus because of its feasibility and has always been high on the list because it is critical to succeed anywhere in the curriculum and anywhere in life. A focus on reading makes sense. The panel needs to know from the committee if the intent of the legislation is met by limiting the longitudinal study to either a limited number of subjects or grade levels. The panel finds more technical feasibility in limiting it to reading and there are fewer technical questions if it is focused only on one part of the curriculum.
Commissioner Wilhoit said the department would like to challenge the panel to think about the benefit of doing the annual assessment as opposed to looking at intermittent assessments. The intermittent model would solve the concerns from the field of over-testing and the cost factors that it will entail in the annual repeating of the assessments. He asked if there could be a period without testing and move to a time period of sixth and seventh grade assessment. The department may be able to do something in the curriculum, to prepare the schools better for incremental programs of study, and to emphasize the role of responsibility of every grade level without having to do an assessment every year.
Representative Moberly, remembering testimony before the 1997 Task Force on Public Education, said teachers wanted the opportunity to show how they moved students along and that request has been what has sparked the committee’s interest in a longitudinal model. He said that the model should be limited to just one or two subjects or limited to elementary.
Representative Rasche asked why one could not use a reading exam like the CTBS.
Commissioner Wilhoit said that he did not think an off the shelf test is a clear and direct measure of Kentucky’s curriculum.
Representative Rasche asked about administering a test in the fall and again in the spring.
Dr. Catterall said that he had been involved in that type of testing process before and that the tests given in the fall have higher scores because of the lack of energy given to the spring testing. That is when one sees motivational decline in students so that is why tests from one school year to the same time in the next school year is a better model.
Representative Rasche said that the longitudinal model should also be used as an evaluation of the Extended School Services (ESS) program. The major point of ESS was to make a difference with the lower performing students. There should be some accountability of that program as well. The longitudinal instrument would give you something to go along with that, in the basic skills, which was what the committee was interested in along the way.
After a general discussion of testing, Representative Moberly asked if there would ever be an instrument that could it be used for instruction and accountability.
Dr. Poggio said not in the panel’s opinion.
Dr. Catterall said the purpose of the longitudinal study is to extract information that is being generated in some format to some procedures that give you a sense of some longitudinal readings. The goal is to have consistent definition of categories. One of the things that would be on a district level is percentages of students in each category, in a school district, that is at the elementary level in these subjects. Then there are percentage distributions of the students at the middle school level. If the standards are set well, one could look at a distribution and see the changes. This kind of model can be used with existing data.
Representative Moberly stopped the discussion to review 703 KAR 5:140, school and district report cards.
Commissioner Wilhoit said the Department is moving ahead with the changes on the regulation and asked if it could be deferred until the next meeting.
Representative Rasche made the motion that 703 KAR 5:140 be deferred, and Representative Treesh seconded. The motion was adopted.
Representative Moberly returned to the discussion and said that there is a question on “social promotion.” We need to look at the ESS program and see what needs to be done to measure the students who don’t have the skills and knowledge they need, and how the ESS program can be improved. The committee has heard from the panel that it would be a burden on the accountability system to do both. He said that if we are going to design something to evaluate ESS and measure remediation for our students, it should be done outside the accountability system.
Senator Casebier asked about narrowing the focus to reading in the third and fifth grade, and maybe in the sixth grade. The students have discipline problems because they can’t read; if they are failing in reading they are not going to succeed anywhere. He said that maybe too much emphasis has been put on every other level and what is needed is too focus on the front end.
Dr. Catterall said that the system will allow us to know the percentage of students reading at grade level. A similar model is a percentage of students reading at grade level at grade six, the test is being given at three, five, and nine. Understanding of longitudinal progress is the percentage of students reading at grade level at one grade vs the percentage of the students reading at grade level down the line. The scores could be reported that way, and could be considered a longitudinal component.
Dr Catterall said that there has been discussion about looking at reading as a longitudinal study. There are a couple of data points in the system that speak to the longitudinal questions, and the question is how valid are any inferences that might be draw from them.
Representative Treesch said that trying to solve the multitude of social and educational problems with the longitudinal model might not be what needs to be done. He said that the focus should be on reading. Teachers want to be given credit for bringing students along. Things that are not valid at a student level, could be valid at the school level. If there is validity at the student level, then it can be done outside the longitudinal model and outside of the accountability model also.
Dr. Catterall said that technical questions are different at the student level vs the school level, and things valid at one, might not be at the other. The two simple models would be more valid at the school level, the problem would be the mobility issue.
Representative Moberly said that there would be no support for the first model because of the validity issues. There should be concentration on the second model.
Commissioner Wilhoit said that it was adding one more assessment in the time frame and adding additional testing time, but the Department could prepare the schools better for the additional assessment.
Representative Moberly asked what would happen to the accountability system if schools could do some longitudinal measure in a form like the second model on an optional basis and have it included in their accountability score in some form or fashion if they wanted to.
Dr. Poggio said that it could be done technically, but Kentucky has never afforded the schools local option to evaluate.
Dr. Catterall asked that if the panel comes back with a model that focuses on reading in the elementary grades, would that in fact be delivering on the legislation.
Representative Moberly said that would be within what the committee intended. He said that as the pilots have been done, the committee’s thinking has evolved. He said that would be a reasonable approach.
Representative Rasche said that it has come full circle back to the original intention of school assessment. As the committee looks at the longitudinal model, it needs to be very careful that there are incentives to teach every child.
Representative Treesh said that the model should be limited to the most critical success factors, reading being one of those, and maybe even mathematics. He said the focus should be in early grades.
Representative Moberly said that the only other thing on the agenda was the annual report of the Office of Education Accountability. It was agreed to have it first on the agenda at the next meeting.
With no further business the meeting adjourned at 11:40 a.m.