The fourth meeting of the Education Assessment and Accountability Review Subcommittee was held on Wednesday, May 16, 2001, at 10:00 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator David K. Karem, Co-Chair; Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, and Tim Shaughnessy; Representatives Frank Rasche, and Mark Treesh.
Guests: Bill Doll, Kentucky Medical Association; Cindy Heine, Prichard Committee; Bob Blackburn, CFC; Wayne Young, KASA; Martin Cothran, Family Foundation; Tomy Scholar, Kentucky Chamber of Commerce; Judith Gambill, KEA.
LRC Staff: Sandy Deaton, Ethel Alston, Audrey Carr, Evelyn Gibson, and Kelley McQuerry.
Senator Karem moved approval of the minutes from the December 16, 2000, meeting and Representative Rasche seconded the motion. The motion carried by voice vote.
Representative Moberly told the members that a copy of Senate Bill 280 was in their folders and asked Senator Karem to make comments about the reason for the legislation.
Senator Karem said the legislature thought there needed to be more oversight of the Office of Education Accountability (OEA) and its work should be monitored by the committee.
Representative Moberly said OEA was created by the 1990 Kentucky Education Reform Act to respond to the language of the court decision that said it was the General Assembly’s responsibility to monitor the public education system.
Dr. Ken Henry, Deputy Director, OEA, spoke about some of the issues that OEA is reviewing, some management issues, and an update on some of the statistics.
Representative Moberly asked if most of the procedures that are mentioned in the legislation are being used.
Dr. Henry said that the categorical program reviews are being done to collect data for the annual reports.
Representative Moberly asked if the language of this legislation provided a more in-depth look at the accountability testing system.
Dr. Henry said such a review would require going outside the Commonwealth Accountability Testing System (CATS) for other quality indicators. This would be matched against the test results to validate them and to determine the accuracy in light of the other indicators.
Representative Moberly stated that the committee needed to help OEA focus on the processes that are permitted in the legislation. Dr. Henry said if the committee wants OEA to go forward in specific areas, there should be dialog set up for direction to OEA.
Senator Karem said the statute speaks clearly about the responsibility of OEA, and the two critical parts of the office are to provide research on finance and to provide redress for people. The concept of informing the public about OEA is important.
Dr. Henry said the majority of the school personnel is aware of OEA, but perhaps parents and the general public are not.
Representative Moberly said OEA has done a large volume of investigative work and is required to provide the committee with summaries of this activity.
Representative Moberly said the next order of business was a discussion on setting the standards for the CATS. Two opposing editorials concerning the issues relating to setting the standards were distributed. Commissioner Gene Wilhoit from the Kentucky Department of Education and Ms. Helen Mountjoy, Chair, Kentucky Board of Education, were invited to testify before the committee.
Ms. Mountjoy said she would talk about the standards setting process for the CATS test. She said that in 1990 the General Assembly created a public system that included an education accountability system designed to increase student achievement and focus on instruction at higher levels so that no child in Kentucky would fall behind. In 1998 the General Assembly required changes in the testing program and created the CATS test. Other mechanisms were established for the state board to receive advice and guidance from committees such as the National Technical Advisory Panel on Assessment and Accountability, the School Curriculum Assessment and Accountability Council and the Office of Education Accountability. These groups have been helpful to ensure that the state board carries out its statutory responsibilities making sure there is widespread involvement in the testing and accountability system.
Ms. Mountjoy said that in the last 18 months, 1651 teachers have been involved in the process of setting standards. Commissioner Wilhoit sent an e-mail to 40,000 teachers across the state for a response on the standards-setting process and the descriptors which are part of the standards. The public can also go to the website to look at the descriptors that have been recommended to the board and give input. Over 2654 people have responded in the past week.
Ms. Mountjoy said the standards are definitions of what novice, apprentice, distinguished, and proficient should mean. These descriptors are more sophisticated, complex, and comprehensive than the descriptors used under the old system. When the descriptors have been agreed upon then the cutpoints can be established that will show where a student stops being novice and becomes an apprentice, where a student stops being apprentice and becomes proficient, etc. These cutpoints will be used to make performance judgements for the individual schools. The advantage of the expanded definition is that everyone across the state will have an agreed upon understanding of what novice, apprentice, proficient, and distinguished means, and teachers will be able to use the information to improve their instructional practices to assist students in becoming more proficient.
Ms. Mountjoy stated that the state board will take action to establish the standards. Each school will be given a baseline and growth chart to show what needs to happen between now and the year 2014 when the Commonwealth of Kentucky determines that schools in the state have reached proficiency.
Commissioner Wilhoit described the process that was used to come up with the recommendations for the state board on the new standards. This will enhance the process by being more precise and detailed in the descriptors given to teachers and will allow them to have a meaningful instructional process. New standards were necessary because multiple choice items and norm reference test components were added at all levels, the length of the test components were changed, some test questions were revised or eliminated, the method for equating tests was revised, the accountability system was changed, and there is a need to revisit standards periodically.
Commissioner Wilhoit said three methodologies were used, first being the Contrasting Groups, which basically has teachers using their own students to make judgements against those performance descriptions. The second method is Jaeger-Mills that used actual student work and placed that work in a category at points. The third method was the CTB Bookmark process where items were lined up in order of difficulty. The last step was a synthesis step that brought teachers who had been engaged in one of the other processes together to look at the recommendations and consider all of them and then go back to the descriptors and find out which of these truly represent student performance that they could recommend.
Representative Moberly introduced Ms. Linda Montgomery, East Jessamine Middle School, and Mr. Brian Robinson, Highlands High School.
Ms. Montgomery said she was involved in the CTB Bookmark standard-setting process. She explained how the group took the test and then determined from the descriptors and core content what knowledge was required in order to do a novice performance on the questions. Through discussions it was determined what the cutpoints should be for the middle school levels at that time.
Ms. Montgomery said the next step was the synthesis step that was made-up of a group of nine elementary, middle, and high school teachers who reviewed all the cutpoints. There was discussion across the curriculum, the instructional rationale for recommendations, and the impact on the percent of students at novice, apprentice, proficient, and distinguished.
Mr. Robinson said he was involved in the Jaegar-Mills part of the process along with the synthesis step. He said there were extensive discussions about the descriptors within each group to determine where each cutpoint should be. The next step was to practice evaluations. There were 60 assessments to evaluate and to see how each descriptor would be put in place. There were a variety of performances on the evaluations.
Mr. Robinson said it was very impressive to see 1600 teachers determine what they believe the standards should be. He said his experience would be beneficial to his district.
Representative Moberly asked Mr. Robinson and Ms. Montgomery if any of the teachers who participated in the program were dissatisfied with their involvement or with the final results.
Mr. Robinson said there were short discussions on the different descriptors and for the most part that everyone made professional decisions.
Ms. Montgomery said she had worked with many of the participants before. They are all very professional teachers who finished with the feeling that for the most part it was a good system.
Representative Moberly asked both the teachers if they thought the process lowered standards or if they thought they were appropriate. Mr. Robinson said that by having three steps and including many teachers the bias was avoided.
Senator Karem asked Commissioner Wilhoit about the perception of the standards that they are setting.
Commissioner Wilhoit said there is no pattern because some of the standards are more rigorous and some have been adjusted so the index scores would be lowered. The standards are very different and much more precise; it has been a solid process and it is a true representation of what we want for our students. The descriptors are not minimum standards or recall of basic facts. The common language in Kentucky is that all children can learn at higher levels and it is our responsibility as adults to carry this out. This goal can be made if every school district takes this seriously and the goal can be made before 2014.
Senator Karem said the most disturbing thing is that educators thought students could not learn and large numbers could not learn at a higher level. He urged Commissioner Wilhoit and Ms. Mountjoy to continue to push educators to meet the higher goals.
Representative Treesch complimented the program and said that a lot has been learned over the last 11 years. One of the main concerns is how many students are proficient now and should the number have been the same under the old testing system.
Dr. Catterall, by phone, said that you do not start by saying there are more or fewer students on a proficient level, but should take the position of reviewing and asking what is really meant by proficient. It is about applying questions asked against student work and that leads to the definition of proficient. The percentage of students will grow out of the proficient level. Once all the evidence is reviewed in steps one through five based on student performance and how it is assessed against those standards, there is a distribution of novice, apprentice, proficient, and distinguished.
Dr. Catterall said that if some of the schools are not progressing toward an overall proficient level of performance, then there need to be questions about why the students are not where they need to be and then how the school is going to get there.
Dr. John Poggio said that the comparison of the Kentucky Instructional Results Information System (KIRIS) and the CATS tests are two totally different tests. The test structures are different as KIRIS was an open response test and the CATS test is half multiple choice and the other half performance based assessment. He also said we need to acknowledge there have been changes, but we should be comfortable with the system that is in place now and move forward from this point. The system says that all children can learn and this is a creditable process.
Representative Treesch said the standards need to be looked at prospectively. He asked if the number of proficient students ended up at the same level under the new system as the old system, would it then be coincidental.
Dr. Poggio said it has come to a point that the recommended standards that will be reviewed by the board are sound and trustworthy and will set the new baseline.
Senator Casebier said he participated in the bookmarking process and that he was impressed with the diversity of the group of teachers. The process did what the stakeholders wanted it to do. The part of the equation that has not been conquered is the student accountability component. All students can learn, but they have to want to learn.
Dr. Poggio said that in 1992 there were six teachers in a content area covering all grade levels, and that is how the standards were set. In this process there were three panels of eight people each looking at a particular grade level. Then there were 24 teachers at a grade level in a content area working in three separate panels who also came together to resolve their differences. He said if this process does not work, he was not sure what process would work.
Senator Karem said he agreed with Senator Casebier that the majority of teachers do a good job, but there is also a collective mentality that thinks that the new system will not work. If data were taken from two schools in Kentucky that had the same number of students, one would make it happen and the other would not. The system does work and there are schools that make it happen.
Commissioner Wilhoit said that the belief in children is a necessity. He also said he had visited schools that are blaming the students, the parents, race, and economic incomes as the cause of the problem instead of looking internally. He said to have this process work it takes a principal who is focused on the curriculum and instruction, a curriculum that is in line with the core content, a set of instructional practices that are geared toward the needs of children, a constant diligence toward identifying the needs of children, and strong reading programs at the elementary grade levels. It is an issue of making sure all the schools come together in a district supportive network.
Senator Karem said there needs to be a way to help the schools achieve these goals. He asked if the new money that may flow from the federal government will help these districts.
Commissioner Wilhoit said the fact that they are tying federal money to results and student performance is very encouraging. Title I is tied to a particular program that works with the neediest schools to help turn them around with professional development and that is also encouraging.
Ms. Mountjoy said another point is that through House Bill 58 (1998), there was money allocated for audits of low performing schools. The assessment and accountability system is to ensure that every child has an opportunity to learn what they need to be successful in the future.
Representative Rasche asked if the CATS test will need another transitional formula in the next few years.
Dr. Poggio said there would be no need for transitions, but what would happen is the need for a baseline looking at the 1999 and the 2000 tests scores against whatever cutpoints the state board of education sets. When the board takes its action, the department will send to all schools their biennial accountability index and what the growth chart of their goals will be between now and the year 2014.
Ms. Mountjoy said it is necessary to insure there is an opportunity for a student to score at the proficient or the distinguished level on individual test questions. There was previous discussion that perhaps some of the questions did not allow students to demonstrate the high level of knowledge necessary for a distinguished score. The state board has been assured by the department and the testing company that this concern will be taken into consideration in developing test questions.
Dr. Catterall said CATS is an element of the process to get schools where you want them to be with some rewards for performance and motivation. The test becomes a comparable diagnosis about each individual school by grade level and content level. It is important to remind ourselves that standards are good over periods of time, but need to be revisited at some point.
Senator Karem asked what the teachers think of the 2014 date.
Mr. Robinson said that it depends; some of the veteran teachers have the discussion about the changing of standards every few years. The mentality as a school is, “What can we do next year to improve, and then the year after that?” The question should be, “Can we improve from what we did the year before?” and not “What can we do in 2014?”
Representative Moberly thanked everyone on the panel for being at the meeting. The final item of business was consideration of the work plan for the study required by HCR 88. A motion was made to accept the work plan by Senator Shaughnessy and seconded by Representative Treesh. The motion passed by voice vote.
With no further business the meeting adjourned at 12:15 p.m.