The Education Assessment and Accountability Review Subcommittee was held on Friday, September 20, 2002, at 1:15 PM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator David K. Karem, Co-Chair; Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, Daniel Kelly, and Tim Shaughnessy; Representatives Mary Lou Marzian, Frank Rasche, and Mark Treesh.
Guest Legislators: Senators Dan Seum, Katie Stein, Johnny Turner, and Jack Westwood. Representatives Larry Belcher, Charles Siler, Arnold Simpson, and Charlie Walton.
Guests: Kevin Noland, Department of Education; Mike Carr, Kentucky Association for School Administrators; Jane C. Lindle, University of Kentucky; Judy Johnson, Spencer County Schools; and Clyde Caudill, Jefferson County Public Schools.
LRC Staff: Sandy Deaton, Audrey Carr, Ethel Alston, and Kelley McQuerry.
Representative Treesh moved for approval of the minutes from June 4, 2002, meeting and Representative Rasche seconded the motion. The motion carried by voice vote.
Representative Moberly introduced Commissioner Gene Wilhoit, Kentucky Department of Education, to speak about the contract renewal for the state testing program with CTB/McGraw-Hill.
Commissioner Wilhoit said that the Kentucky Department of Education (KDE) issued a contract through the Finance and Administration Cabinet to CTB/McGraw-Hill to administer the Commonwealth Assessment Testing System which was awarded in September 1998 and will end September 30, 2002. The contract included a four year renewal option. He said that CTB/McGraw-Hill was awarded the contract as the prime vendor who chose to use two subcontractors, Data Recognition Corporation (DRC) and West Ed. He said the original contract was for $30,500,000. He said that in May 2002, the Finance and Administration Cabinet issued a renewal of the contract for a four year term effective October 2002 with the concurrence of the Department of Education. The KDE wants CTB/McGraw-Hill to begin to use image scoring which will expedite the scoring process on the new contract. Image scoring will help get the test results back sooner. He said that there are no significant changes to the contract at this point. He said that if the contract were to be opened for re-bidding there would be a long delay in the bidding process and the KDE has been very satisfied with the performance of CTB/McGraw-Hill. He said that in July 2002, CTB/McGraw-Hill presented KDE with the price of approximately 45 million dollars for the next four years. The price has been reduced to $39,950,000. He said that the contract will include the use of image scoring, and a ten percent read behind of the scores which will assure high quality in individuals scoring the test. He said that there would be advantages in moving the scoring process to several Kentucky sites. He said that one of the advantages would be that Kentuckians would be paid to do the work and the work would increase the teachers’ knowledge of the types of assessments that are conducted. He said that adjustments to the contract are anticipated to comply with the “No Child Left Behind” implementation.
Representative Moberly said that during the September Interim Joint Committee on Education meeting there was discussion pertaining to the dispute between CTB/McGraw-Hill and the principal subcontractor DRC. Commissioner Wilhoit said that DRC was a subcontractor for a certain part of the prior contract that pertained to actual scoring. He said that in the current proposal CTB/McGraw-Hill will be the prime contractor and there would not be a subcontractor. Representative Moberly asked if CTB/McGraw-Hill could continue the good performance without the subcontractor. Commissioner Wilhoit said that it is imperative that image scoring be done in order to shorten the process and provide quality control, and doing the scoring in Kentucky will be a major benefit. He said that the capacity to do image scoring by both entities was reviewed. CTB/McGraw-Hill does this kind of work in other states and has a positive record.
Representative Treesh asked if DRC would have grounds for a lawsuit if its name was left on the contract. Kevin Noland, KDE, said that the contract only requires two signatures, which are the Commonwealth of Kentucky and CTB/McGraw Hill. He said that CTB/McGraw-Hill then subcontracted with DRC and West Ed Laboratories. He said the only legal commitment has been with CTB/McGraw-Hill. He said that the contract states that the Commonwealth has the right to cancel with a 30 day written cancellation notice at anytime. He said that a renewal clause in the original contract could be exercised in this situation.
Representative Treesh asked about the capacity that CTB/McGraw-Hill has to get scoring finished. Commissioner Wilhoit said that the competition among the states makes it more difficult. All states are looking for contractors to comply with “No Child Left Behind” and the question is whether you take a continuation of the current contract or do you open up the bidding to that type of environment. He said that it is a wiser choice for Kentucky to stay on its current course.
Representative Treesh said that timely results were another concern. He said that in order for the teachers to make new plans for the next school year they have to have the results in a timely manner. He asked if the new contract will shorten the time for reviewing the results. Commissioner Wilhoit said the reason for going to imaging is to shorten the process.
Senator Westwood asked why the cost of the image scoring is more expensive. Commissioner Wilhoit said that the higher cost was part of the original proposal, but was removed during the negotiations. Senator Westwood asked what amounts other states were spending on testing contracts. Commissioner Wilhoit said that Kentucky is within the range of competition with other states. He said that Maryland just re-negotiated its contract and it was $40 million dollars. Senator Westwood asked if image scoring would affect the writing on demand answers. Commissioner Wilhoit said there in no anticipation of changes in format that would cause adjustments. He said that the industry can do image scoring with the same accuracy that is done with traditional means.
Senator Westwood said that one of his concerns is that a four-year contract is expensive and whether there would be any audits performed. Mr. Kevin Noland said that the progress with CTB/McGraw-hill has been good. He said that the KDE has not commissioned an audit. He said that the National Technical Advisory Panel on Assessment and Accountability has provided advice on the assessment system and provided recommendations to the Education Assessment and Accountability Review Subcommittee and the Kentucky Board of Education. He said that there was an audit commissioned by LRC in 1997, but not one since then.
Senator Westwood asked if there will be any federal funds provided to the state in order to comply with the “No Child Left Behind” testing. Commissioner Wilhoit said that there will be funds provided. He said that if the if the scope of work is not changed then the funds can assume the additional developmental costs for the next few years that would include the administration of the tests and the work that has to be done on the longitudinal implementation.
Representative Walton asked about the timeline of the testing results and if there will ever be a testing at the beginning of the first semester and end of the second semester to give a picture of progress throughout the school year. Commissioner Wilhoit said that those issues are being discussed. He said that the test is a way the state can make a one-year assessment and then every two years, an overall judgement. Representative Walton said that the contract that has just been finished is better than the former contract.
Representative Moberly asked if there could be assurance that the advantages and disadvantages of renewal versus the rebidding of the contract. Commissioner Wilhoit said that from the programmatic side of development he feels that there is movement in the right direction with this and will enhance the capacity as we move ahead. He said that from a contractual side of issues, there has been consultation with the Finance and Administration Cabinet throughout the process and the KDE has followed its direction and guidance.
Representative Moberly asked Commissioner Wilhoit to report the results of the accountability cycle. Commissioner Wilhoit said that the results of the Commonwealth Accountability Testing System(CATS) were released as of September 19, 2002. He said that this is the first time that the schools will have new trendlines against the base scores that were set in 1999-2000. He said that every school has a growth chart that will set them toward the goal of proficiency by the 2014 timeline. He said that all schools already have reported to them an actual score for the 2002 line that moves from the baseline. He said this has allowed the creation of an environment in the state that is much more productive than it was in the past. He said the changes that were made by the 1998 General Assembly and the results of the work that was done are now in effect. He said that the schools are reporting that there is a more comfortable feeling with the new system. He said there is predictability in the new system to tell schools where they need to be. He said that under the old system the goal line had to be adjusted after testing. He said that this year disaggregated information is being reported to the schools as a result of SB 168 (2002) that required the department to break out information in more detail than in the past. He said that each of the schools has received information that breaks the performance out by subpopulations of race, gender, social economic status, and disability. He said that information is going to be used as a trigger for the work in the schools.
Commissioner Wilhoit said the new accountability cycle is very positive in terms of providing the type of guidance and direction to help schools make decisions. He said that the results are provided by year in terms of index score and the amount of change. He said that the baseline scores in the 1999-2000 in comparison to the two year results we have now, shows growth. He said that the relative performance of students in mathematics to other content areas should be of some concern.
Commissioner Wilhoit said as you move to the middle and high school levels there is less growth. Sixty-two percent of the elementary schools met or exceed the goals, at the high school level less than 44 percent met or exceeded the goals. He said the high school level is not sustaining the percentile of meeting goals as compared to the elementary level. He said that the positive news is that 554 schools have met or exceeded their goals for this biennium. He said that assuming there is continued growth, they will meet the proficiency goal by 2014. He said that 90 schools are in one of three phases of assistance. That means that there is a concern for the 30 Level Three assistance schools. He said that according to law and regulations these schools will be audited and highly skilled educators will be placed in those schools. He said the middle level schools will be required to have reviews conducted by the department administered staff in conjunction with districts. He said that district will have responsibilities for conducting similar activities for the Level One assistance schools. He said that for the first time in the history of the accountability system there are six schools, five elementary and one middle school that have reached the proficiency goal. He said that the Level Three assistance schools are a concern. The schools have very detailed information not only from the state but also from the school performance from the district and the state-wide performance in each of the content areas. He said that all the information is broken down by subpopulations. He said it will provide them the type of guidance that is needed to make decisions concerning the performance. He said the most valuable work will be done at the school level and the department is providing as much assistance as possible.
Senator Kelly said that the first tangible evidence of the effort to intervene with low performing readers has had some measurable success. Since SB 186, there has been a fifty percent reduction in the number of novices. He said the score is up, but the target area is the number of students that were not performing at the proficient level. He said that the fifty percent decrease in that number is encouraging. He said that math scores are discouraging and some type of intervention has to be developed in that area. He said it is important to have the diagnostic testing, the intervention strategies, and quality control testing in some type of longitudinal study. He said that schools that are performing well are using these methods and are successful. Commissioner Wilhoit said there is agreement that the focus on literacy is producing positive results. He said that two things are happening in districts that were not happening before. He said districts are embracing the new programs that have been set in place for an overall school strategy for teaching reading to every student and they have identified ways to work with the students that are behind in intensive ways that have turned the reading level around. He said he looks forward to having an open dialog about what is being done and what needs to be done in the future. He asked if there needs to be some type of combined system where there is a robust type of work going on that is supported by some type of statewide assessment. He said there are values to both programs.
Senator Kelly said that the middle school and high school results in writing have declined. Commissioner Wilhoit said that in the last two years the scores have declined. He said that the overall trend is that the scores continue to improve, but there is not a dramatic improvement. He said that maybe there needs to be stronger evaluation in terms of curriculum offering. He said that as the system moves forward the things that fall behind should be an area of policy concern.
Representative Moberly said that the results in the last cycle were similar with respect to the problems in the high school. Commissioner Wilhoit said the department has already come up with some strategies that will be helpful. He said that some high schools are doing well and the idea is to get into these schools and see what can be learned from them. He said there are twelve other high schools that are looking at some innovative programs. He said this ranges from revamping the counseling programs and having stronger relationships with parents.
Representative Moberly said that there are schools with low socio-economic status that have done well because they have used best practices and are doing the innovative programs that fit each school.
Commissioner Wilhoit said that some of the schools will not receive their rewards this year even though they met their goals because they did not meet the drop-out reduction goal.
Senator Westwood said that there is success in the reading levels in the elementary grades, but he said there is concern about the aspect of accommodations. He said that even though a student may have comprehension, it does not mean the student knows how to read. He asked how many students in the lower grades were receiving accommodations with reading. Commissioner Wilhoit said he would get that information for him.
Representative Moberly introduced Dr. Ken Henry, Director, Office of Education Accountability (OEA). Dr. Henry said that there would be four issues to discuss with the committee. He said that KRS 7.410 requires OEA to submit the annual report each calendar year which does not have a due date. It also requires a report on the finance system which has an October 1 due date. He said in the past both reports were submitted on the same date. He said that the reports need to be separated so that each one can have full detail.
Senator Kelly moved that the two reports be separated. Representative Treesh seconded the motion. The motion passed by voice vote.
Dr. Henry said that the annual report’s format will be to identify issues that come from monitoring work through investigations or complaints that are received in the office. He said that future reports will provide background information, based on data and will present options to the committee for consideration.
Representative Moberly said that in the past the report has been brought before the committee with recommendations. He asked Dr. Henry if now, instead of a final report, would it be a draft with a discussion of the issue, the experience of the issue, and possible options for the committee to consider. He asked if the draft would be circulated to the various stakeholders so that the committee would have the benefit of hearing from them before approval would be given. Dr. Henry said that by separating the report it would give the time that is needed for the stakeholders to look at the issues and all the various options that are presented in the draft and give responses. Representative Moberly said that the committee will expect the information from the annual report in future committee meetings.
Dr. Henry said that the other issue is the implementation of SB 166 (2002). He said that legislation requires OEA to review personnel policies in districts that have at least an eight percent minority student population that does not have minority staff overall equal to the eight percent. He said that 45 districts fit this category and that the legislation also requires review in districts that have received complaints related to discriminatory practices in hiring or dismissal. He said that added six more districts to the number with a total of 51 districts with 500 or more schools. He said that a proposal has been sent to the members to implement this requirement. He said it requires review of documents from districts and interviews with staff at the local school level and then to report to the committee by November of 2003.
Dr. Henry said that the draft of a proposed operations manual of the OEA describes what is done in the office and the interactions with the subcommittee.
Representative Moberly said that the committee does not want to get into the business of approving an operations manual. He asked Dr. Henry to highlight a few areas of the manual for the committee. Dr. Henry said that one of the major issues is getting legislative requests to do studies. He said that anytime a member of the General Assembly has an issue that needs to be looked at by OEA, the member will need to come to the co-chairs of the committee with the requests and be considered by the full committee before OEA looks into the request. Representative Moberly said that the members have the document to review.
Representative Moberly asked about the assessment evaluation that was required by statutes. Dr. Henry said that he had spoken with the members of the National Technical Advisory Panel on Assessment and Accountability (NTAPAA) and he thought they were a good body to help with external validation. He said that the technical panel is interested in this issue and a suggestion would be to build a framework with the technical panel and then to make decisions on what would be the best way to go forward with this project. Representative Moberly asked the members of the committee to consider authorizing OEA to develop a plan for external validation and seeking direction from NTAPAA.
Senator Kelly asked if after OEA seeks the advice from NTAPAA, would Dr. Henry come back to the committee with a proposal. Representative Moberly said that he suggests that Dr. Henry bring back a proposal to the committee for consideration.
Representative Treesh moved for OEA to start conversations with NTAPAA and bring back a model to be considered. Representative Rasche seconded the motion. Senator Kelly said that with the pressure of the “No Child Left Behind” Act that has created all the demand for testing is a concern that we are not engaged in an exercise in looking at the test to say it is a statistically valid measure. He said that the study should determine if we are accomplishing the goals of the original legislation with an effective tool to measure progress and upon which accountably decisions can be made. We also need to know whether that is being done efficiently and effectively. Representative Moberly said that the statute provides flexibility in a manner in which the committee can design the goals and what is important to the validation study. The motion was approved with voice vote.
With no further business the meeting adjourned at 2:50 p.m.