The Education Assessment and Accountability Review Subcommittee met Monday, August 25, 2003, at 10:12 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, Daniel Kelly, and Ed Worley; Representatives Jon Draud, Mary Lou Marzian, and Frank Rasche.
Guests: Scott Trimble, Cindy Owen, Linda France, Johnnie Grissom, Bonnie Brinly, Kevin Noland, and Gene Wilhoit, Kentucky Department of Education; Cindy Heine, Prichard Committee; Wayne Young and Mike Carr, Kentucky Association of School Administrators; Alicia Sells, Kentucky School Boards Association; and Clyde Caudill, Jefferson County Public Schools.
LRC Staff: Sandy Deaton, Audrey Carr, Janet Stevens, and Kelley McQuerry.
Representative Marzian moved for approval of the minutes of the June 23, 2003 meeting and Representative Rasche seconded the motion. The motion was passed by voice vote.
Representative Moberly introduced Ms. Helen Mountjoy, Chair, Kentucky Board of Education; and Commissioner Gene Wilhoit and Kevin Noland, Kentucky Department of Education; and Benny Lile, Chair, School Accountability, Assessment and Curriculum Council; and Dr. James Catterall, Chair, National Technical Advisory Panel on Assessment and Accountability (attending via speaker phone) to speak about the policies to implement the federal No Child Left Behind Act.
Ms. Mountjoy said that the state board has spent a great deal of time in the past fifteen to eighteen months looking at the No Child Left Behind (NCLB) legislation. She said that two special full day sessions were devoted to this legislation. She said that in looking at ways that Kentucky might comply with this legislation, there were four guiding principles that the board followed. She said that the first principle is to comply with the NCLB, and the goals contained in this legislation. She said that proficiency by 2014 and progress for every child are the same as the goals in Kentucky.
Ms. Mountjoy said that the second principle is to ensure that Kentucky continues to make progress toward the learning goals that are outlined in state statute. She said that there are several things that need to be done to ensure high learning and performance standards that are important to measuring academic progress across the curriculum. She said that reading and math are fundamentals, but science, social studies, writing, practical living and vocational studies, the arts and humanities are also important elements in preparing students for the future.
Ms. Mountjoy said that the third principle is to build on the assessment system that has been developed in Kentucky by making changes that enhance the process. She said the state board will add reading and math assessments at those grades between three and eight where they are currently missing. She said that next spring they will look at the types of items that should be included on the assessments and make sure they will reflect the standards and performance that is expected. She said there will be a shift to using graduation rates, instead of the dropout rates.
Ms. Mountjoy said that the fourth principle is to ensure the continued validity and reliability of the assessment and accountability system. She said that wholesale changes in an assessment and accountability system are damaging to the validity and the reliability. She said it is important to have a system that is valid and reliable so there is confidence in the performance judgements. She said that whether a school is deemed not making yearly progress or deemed to be successful, the school needs to be correctly identified.
Ms. Mountjoy said that these principles were used as the state board has looked at a wide range of decisions that need to be made. She said there is not a mandated test for No Child Left Behind or a specific mandated process for use in all states. She said that what NCLB calls for is each state to use an assessment that contains multiple measures that can gauge student success toward the state’s learning standards. She said that the Commonwealth Accountability Testing System (CATS) has been designed to measure progress toward the learning standards that have been developed and adopted in Kentucky. She said that the issue of various subpopulations scoring as high as the total school on the tests had to be addressed. She said that the subpopulations are students with English as a second language, students that have identified disabilities, students who are in racial or ethnic minorities, and students that are in low socioeconomic status. She said that each of the groups are to be compared and measured against the same standards of high performance as the rest of the school. She said that this is an expansion of some of the ideas that were contained in Senate Bill 168 (2002), giving attention to achievement gaps that exist in student performance.
Ms. Mountjoy said that a process had to be developed to give a student an opportunity to transfer out of a consistently low performing school. She said that real challenges were presented by the federal timelines for reporting scores. She said that the federal legislation requires the use of multiple measures, which means it is not just “a one size fits all” short answer test, but multiple ways to gauge student success. She said that the federal statute requires that states’ scores be back to the school before school starts in the fall. She said that it takes more time to score items that are assessed in ways other than by norm reference or other multiple choice type of assessments.
Ms. Mountjoy said that at the last state board meeting, the board spent time looking at a range of decisions that needed to be made for implementing NCLB. She said that some of the actions that were taken were new ones, and others confirmed the discussions that had taken place over the last eighteen months. She said that all the decisions are predicated on the four guiding principles discussed earlier. She said that the most important thing is to positively affect student learning in Kentucky.
Mr. Kevin Noland said that the department and the state board were present to get guidance and to brief the subcommittee on the NCLB Act. He said that the last time he was before the subcommittee, the plan had just been approved by the United States Department of Education. He said this meant that Kentucky school districts will receive $330 million dollars of federal funds for elementary and secondary schools, which upon average is seven percent of the school district’s budget. He said that all states had conditions of approval that had to be met in order to be in compliance. He said that when the State Board of Education amends an assessment or accountability regulation, it must be reviewed and approved by the Education Assessment and Accountability Review Subcommittee (EAARS). He said that the department has always brought all items at every stage of the process to the subcommittee to get guidance and direction. He said that the state board thought that with so many complex issues it would be too soon to amend a regulation. He said that at the last meeting there was guidance on some of the key decision points that apply to the 2003 spring CATS testing calculations such as what proficiency is and what the minimum number of students for a subpopulation is going to be. He said that the calculations will apply to the scores that will be made public around October 2, 2003.
Mr. Noland said that there are rules to comply with the NCLB Act in the spring of 2004 and thereafter. He said that they are going through the regulation amendment process and at the October board meeting, they will give the proposals a first reading. He said that at the December meeting they will take action on the amendments and then it will be filed with the Legislative Research Commission. He said that the timeline is that there will be a public hearing in January and then in February it will be ready for official review and action by the EAARS subcommittee.
Mr. Noland said that one of the decision points that has been identified for the NCLB components is adding a reading and mathematics assessment to grades three through eight. He said that the state board has decided to do this by adding a norm referenced test augmented by open response items. He said that federal law requires this to be in place by the 2005–2006 school year. He said that the plan is to test this no later than 2004–2005 and have it count for the 2006 school year. He said the next item is getting all students to proficiency by 2014. He said that Kentucky went through a process with input from teachers and content specialists to define proficiency.
Mr. Noland said that the state board is seeing results from those standards that were put in place. He said that in addition to a school’s whole population, it is required that the subpopulations have at least ten of that type of student per grade and at least thirty in the school to be counted in the assessment. He said that testing experts say that for validity and reliability, ten should the lowest number of students.
Mr. Noland said that in order to make Adequate Yearly Progress (AYP) and to reach the goal under the federal law, there must be a ninety-five percent participation rate on the test and by using one testing system it minimizes the confusion.
Mr. Noland said that the state board decided to have a confidence interval, which is a school that is considered meeting its annual measurable objective in reading and mathematics if the percent of students scoring proficient or above in a school or district meets or exceeds the annual measurable objective in reading or mathematics, or the annual measurable objective falls within the ninety-nine percent confidence interval placed around the school’s or the district’s percent of students scoring proficient or above. He said that the board is exploring ways to meet the federal reporting timeline that requires test scores to be provided to schools before the beginning of the school year so that the federal AYP decisions can be rendered at that time.
Mr. Noland said that the state board has deferred action on determining the length of a full academic year to give time for advisory groups and others to review and provide input. He said that the definition for a full academic year will determine which students are included in calculations for federal accountability purposes. He said that this year the United States Department of Education has agreed to let Kentucky use the current definition during the transition to implement the NCLB requirements.
Mr. Noland said that the board has agreed to comply with NCLB relative to Limited English Proficiency (LEP) students which requires administering the CATS test to any student who has attended a school in the district and is enrolled on the first day of the testing window. He said that the student’s test results will only be part of the calculations to determine if the school has met federal AYP if the student has been in the school or schools within the district for a full academic year. The school will be held accountable if the student has attended only one school and the district will be held accountable if the student has attended multiple schools within the district.
Mr. Noland said that state funds is another challenge. He said that with the NCLB Act there is additional testing that has to be paid for along with the data collection, reporting, and the professional development of teachers to get all teachers to meet the criteria of a highly qualified teacher required by federal law. He said that another area is the assistance to the schools. He said that last fall when the last performance judgements under CATS were made, under 100 schools in Kentucky were identified as not reaching their goals, either levels one, two, or three. He said that as a result, scholastic audit teams went to those schools and highly skilled educators were provided. He said that the simulation with the federal system shows there could be 400 schools identified and that triggers assistance if they do not reach the goal for two years. This creates a resource problem on how to do this.
Mr. Noland said that each state is responsible for determining an annual measurable objective in reading and in math to meet goals that get students to proficiency by 2014. He said that after all states filed plans in June, twenty states proposed a graduated approach. He said that in the early years one jump is made so that schools can get used to the federal system and accountability for subpopulations. He said that equal intervals of improvement start thereafter. He said that the plateaus in the early years will allow schools to adjust to the requirements of NCLB without immediately expecting annual increases.
Mr. Noland said that there is total agreement with the goals of the NCLB Act, but there are some challenges on how to get to those goals. He said that Kentucky values two years of data being more reliable than one year for making judgements on schools. He said that by telling a school in September that the goal has not been met and there needs to be improvement by the next testing in April, does not provide enough time for significant changes. He said that by giving schools two years to make the adequate changes is a fair system. He said that another challenge is the federal law that requires every school to have the same baseline, which is twenty percent up from the bottom. He said that in Kentucky each school has an individualized baseline based upon where the starting point is. He said the other issue is the narrowing of the curriculum to math and reading only. He said that in Kentucky the teachers and everyone involved in the input of this program values science, social studies, arts and humanities, practical living, vocational studies, and writing, and these have been included.
Ms. Mountjoy said that the system that has been in place since 1990 and was re-affirmed in 1998 values progress toward the goals that have been set. She said that under NCLB schools are graded strictly on the percentage of the students that have attained proficiency and does not look at the progress that students have made toward proficiency.
Mr. Noland said that Kentucky does not have to identify any new schools that need improvement this year. He said that last fall 27 schools were identified as schools that were level three schools that had to offer school choice and supplemental school services. He said that those schools have to continue through this year because under the federal system it takes two years to get out of trouble, even if the AYP is reached that year.
Representative Moberly said that the media presented the account of the last state board meeting in a way that it appeared the board was upset because of the possibility of having consequences at some of the schools making adequate progress and meeting the goals of CATS, but not meeting the goals of the AYP. Ms. Mountjoy said that it is a balancing act and there will be challenges in explaining it to people, but by staying with the high goals that have been set and keeping the view on student learning, the unintended consequences will be avoided.
Representative Draud said that he thought that communicating the tests results will be difficult. He said that this is an example of the federal government getting involved in something they should not be involved in. He said that under the tenth amendment of the United States Constitution, the right for education has been reserved to the states. He said he feels that this is an intrusion that has created complexity and he sees no advantage of NCLB for the students of Kentucky. He said that it will be difficult to communicate the tests results and once it is started more people will be confused.
Ms. Mountjoy said that the media reports after the last board meeting stated that the board had adopted a dual assessment system, which is not true. She said there is only one test, which is the CATS test and those results will be used to fulfill both the accountability requirements of NCLB and the accountability requirements of state statute.
Representative Moberly asked whether there will be adjustments made with the NCLB Act in the future or whether this Act was set in stone. Ms. Mountjoy said that when the Kentucky Education Reform Act was passed in 1990, some people thought that it would not be changed and that each element outlined would never be altered. She said that there have been improvements made when there was evidence that it would benefit the system. She said that hopefully the people at the national level would also make improvements if evidence is presented.
Commissioner Gene Wilhoit said that some of the people that were involved in development of this legislation indicated that some of the products that ended up in specific language were not a part of the original thinking of NCLB. He said as the process began, different people had different thoughts. He said that there is a growing awareness in the conflicts that are coming from multiple states and that level of frustration has received the attention of the federal department. He said there are now five amendments being offered. He predicted that Congress will provide future latitude.
Representative Moberly said there have been suggestions that Kentucky do what is necessary under NCLB and have that as the primary testing and accountability system in Kentucky, which would save money that is being spent on CATS. Commissioner Wilhoit said that the immediate difficulty is the confusion that would be created in the state. He said that if you adopt a system that has a lower standard, that would send a message to the education community that Kentucky would be pursuing to a less rigorous goal than before. He said that in looking at the language in NCLB, about what the system should measure, it does ask that the state reflect the core content in that it have multiple measures and that it measures a higher level of learning. He said that this can not be done with an off the shelf test because it would not meet the federal challenge.
Mr. Lile said that assessment drives the curriculum. He said that the council is made up of school personnel, business people, and parents that are faced with issues that confront narrowing of the curriculum. He said that there is no doubt that by narrowing the curriculum, the choices for students would also be narrowed throughout the early years as well as in high school.
Mr. Noland said that in the 1980’s by narrowing the curriculum in testing math and reading, Kentucky was at the bottom of the barrel compared to other states. He said that in terms of the federal program, it is truly supportive of teaching and learning. He said that changing the system just to change it may not be a good enough reason. He said that it must be thoughtful plan and be better for the students. He said that changes take years to implement.
Senator Kelly said the idea that there might be different designations of schools under CATS versus NCLB does not seem to be problematic. He said that because of this new requirement that is creating challenges, this is the time to continue the on-going evaluation of how things are being done and what could be done differently. He said that he is concerned about the defensive attitude of not being able to change. He said that it seems one of the difficulties in meeting this federal requirement is relying on a single assessment to determine whether there is success in a school or a district. He said that if there is too much emphasis on trying to create a single index, it brings a whole host of complexity. He said it is hard to measure through a single test as to what is happening in the school. He said another problem is dealing with technical difficulty that comes with using open response questions particularly trying to conform to the NCLB requirements for earlier reporting and not adding more testing levels. He said it seems there needs to be some need to maintain the open response in regard to the NCLB and this is a defensive attitude that no ground can be given and everything has to remain the same. He said in this one area it is adding expense and difficulty trying to comply with the federal requirement that the information gets back early. He asked if the board and the department are looking at this requirement with openness and flexibility that every component has to be a mirror image if there is just technical, practical and logistical problems. He said that if you are measuring math and reading to comply with NCLB, there are assessments out there that do this and are very inexpensive. He said that when using open response questions, unless there is enough questions there is a validity problem.
Senator Kelly asked why the open response questions are being used in efforts to comply with the NCLB, when it seems they are creating logistical problems as well as an expense. Ms. Mountjoy said that it is part of the whole system and part of measuring the depth of attainment that students have had.
Senator Kelly said that he is not questioning the importance of having the open response questions on the CATS test. He said that he was curious as to why that component has to be in the annual testing for progress in reading and math to comply with the NCLB requirements when it is creating problems. Mr. Noland said that the United States Department of Education said that since there is a single state assessment that applies to state and federal purposes, if the testing is in math, reading, multiple choice, and open response, then they are not allowing separation of multiple choice items to be used for federal purposes.
Commissioner Wilhoit said that this solution was superior to putting partial results out on what could be scored quickly and then coming back and making changes in the scores. He said this is not a solution for the long term.
Ms. Mountjoy said that the testing companies are being consulted to try to get the math and reading scores back before schools start. She said that there are very few testing companies that have experience with large scale assessment testing programs for states. She said that Kentucky has a contract with a well respected company, but they also contract with other states.
Senator Kelly said that the NCLB will be a continuous struggle to see that there is a testing accountability system in place that works. Ms. Mountjoy said that this is not seen as a threat. Most of the statute is very helpful to Kentucky and is being incorporated. She said that the concern is maintaining a coherent system that keeps the focus where it needs to be and that is on comprehensive learning for students.
Dr. Catterall said that adding a limited number of open response items is important to the national technical panel because the lowering on basic skills test every other year makes for a different definition of proficiency from one year to the next by definition. He said that the state has requirements for schools and districts and the federal government has a set of requirements for progress in reading, language arts, and math. He said the challenge is taking federal law and its multitude of requirements, and the CATS as it exists and blending the two systems in a way that makes sense. He said that the overlays and the indicators seem very clear under this combined system.
Commissioner Wilhoit said that he is concerned about the absence of a solid assessment program at the classroom level and the over-reliance on the state assessment. He said that a number of activities have been put in place this year to shore up the capacity of teachers at the local level to make those decisions and remediate those children as quickly as possible with some high quality programs. He said that the state assessment is a one time a year check to see how well a school is doing. He said that in terms of the instructional process in the schools, and the teachers are not getting the tools they need to make those on-going assessments throughout the year in ways that are consistent with the state-wide assessment. He said that there are developmental programs in place where the children are taken along and there will be more emphasis on this. He said that if teachers put in perspective what each assessment is and start supporting each other, the disconnect that has been in the classroom will be removed.
Senator Casebier said that since 1998 there have been discussions about the elimination of the CATS test. He said that since the budget crunch there have been concerns about saving money and this would be one of the ways. He asked Mr. Noland to explain the cost of CATS per testing cycle. Mr. Noland said that every year the legislature appropriates $8.1 million dollars for CATS, and that includes the whole assessment. He said that the misunderstanding occurred because last September when the contract was renewed, it has a total cost over a four year period. He said that there is additional money from Congress for implementing the NCLB Act that will allow adding math and reading tests in additional grades.
Senator Casebier asked what the cost factor is for an off the shelf type of test. Commissioner Wilhoit said that would be five dollars per child but on the multiple choice test only. Mr. Noland said that would not include the content areas that are presently tested. Commissioner Wilhoit said that assessment is not a cost factor and should be seen as an integral part of the instructional process.
Representative Rasche asked if there is a recommendation that a factor be built into the school funding for Limited English Proficiency (LEP) students. Ms. Mountjoy said there would be and that Kentucky has the fifth largest growing LEP population in the United States. She said that this is something many districts are not prepared for. She said that there are many languages other than Spanish being spoken by LEP students in schools.
Representative Moberly said that adjustments to the concept of continued assessment has been made over the years. He said that assessment does drive curriculum but on a continuous basis and if this is being achieved with the help of intervention strategies, the requirements of NCLB should be accommodated in a positive way.
Senator Kelly indicated that in the handout the cost per pupil on the CATS assessment is thirty-two dollars per pupil. Commissioner Wilhoit said that it is sixteen dollars at the local level and sixteen dollars at the state level.
Representative Moberly thanked everyone and recognized Dr. Ken Henry, Office of Education Accountability, who recently took a position with a law firm and will be leaving OEA.
With no further business, the meeting adjourned at 11:35 a.m.