The meeting of the Education Assessment and Accountability Review Subcommittee was held on Wednesday, March 21, 2007, at 1:00 PM, in Room 131 of the Capitol Annex. Senator Jack Westwood, Co-Chair, called the meeting to order.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Harry Moberly Jr., Co-Chair; Senator Dan Kelly and Representative Frank Rasche.
Guests: Clyde Caudill, Cindy Rausch, and Brent McKim.
LRC Staff: Sandy Deaton, Audrey Carr, Janet Stevens, and Lisa Moore.
Senator Westwood stated that a quorum was not present at the meeting so only testimony on the administrative regulation 703 KAR 5:020 would be heard and no action could be taken.
Senator Westwood introduced Mr. Kevin Noland, Interim Commissioner of Education, to explain administrative regulation 703 KAR 5:020. Mr. Noland explained the process that KDE used to solicit input from statutorily created groups such as the Office of Education Accountability, the School Curriculum Assessment and Accountability Council (SCAAC), the National Technical Advisory Panel for Assessment and Accountability (NTAPPA), and the Local Superintendents Advisory Council (LSAC). He said input was received from the groups at various meetings as well as written comments.
Mr. Noland introduced Dr. Pam Rogers, Associate Commissioner for the Office of Assessment and Accountability, and Ms. Rhonda Sims, Division Director, Office of Assessment and Accountability, Kentucky Department of Education, who explained a handout to members and responded to questions. Dr. Rogers said the accountability summary included the Kentucky Core Content Test (KCCT), which includes all the same content areas; remains the basis of the content area academic indexes reported in school accountability calculations; and includes annual testing in reading and mathematics in grades three through eight as required by the No Child Left Behind (NCLB) federal legislation.
Ms. Sims said the accountability summary also includes a norm-referenced test (NRT). In high schools, the ACT index will replace the norm-referenced index. The ACT index is included for five percent of the high school accountability calculation. The ACT Index is generated based on composite PLAN index and ACT index results that are averaged together.
Ms. Sims said the high school ACT index allows schools to earn credit based on the performance of each student along the score ranges for each assessment. She explained that in 2007, PLAN composite results will generate the ACT index. In 2008, following a statewide ACT administration, both PLAN and ACT results will be used in the calculation.
Ms. Sims said the middle schools would administer EXPLORE at grade eight and use information in the development of the student's individual learning plan (ILP). She said the elementary schools would administer a national NRT in reading and mathematics. She noted that both the middle and elementary schools would publicly report results and communicate individual student results with parents, and the results would not be included in the school accountability calculations.
Senator Kelly asked about the controversy surrounding the NRT being included in accountability at the elementary level. Dr. Rogers said a NRT has been used in the accountability system at the elementary level, and the reading and math portions were administered during the third grade. She said NCLB requires testing at the third grade level as well so discussions ensued that it was redundant to test third graders in reading and math twice during their third grade year, using the KCCT and the NRT.
Senator Kelly asked if all schools were required to take the Group Reading Assessment and Diagnostic Evaluation (GRADE) test. Dr. Rogers said it is required in all schools that have reading grants, about 77 percent of all elementary schools.
Representative Moberly asked what the requirement meant about a NRT being administered in elementary reading and math. Dr. Rogers said the elementary schools may choose which NRT they want to administer and determine in which grade level to administer it. Representative Moberly commented that it was removed from the accountability index so the flexibility for schools would remain. Mr. Noland said it was removed from accountability because NCLB requires testing in reading and math at the third, fourth, and fifth grades, which is included in accountability.
Representative Rasche discussed percentile scores and the month the test is given. He said schools could use the test as a diagnostic tool in the fall and the percentile score could still be reported. He said the test could serve two useful purposes in that manner.
Senator Westwood asked how Kentucky would maintain its promise to the parents that an accountability system would be utilized that included a NRT that compares local scores to students in other states. Mr. Noland said the individual student scores would be sent to the parents, and the scores and the aggregate for that school would be reported locally and on the annual report card.
Senator Westwood said if there were a number of different tests being utilized, how would it be weighed for the various differences. Mr. Noland said this is already the case, not every student in the country takes one particular test. Dr. Rogers said there is a loss of comparing one Kentucky school to another Kentucky school at a point in time, but parents can still compare the performance of their student, and their student's school, against a national group. Mr. Noland said the Commonwealth Accountability Testing System (CATS) tests could be used for comparisons against other Kentucky schools as well.
Representative Moberly discussed the statute requirement of the assessment program utilizing a NRT. Mr. Noland said KRS 158.6453 does state that the assessment program must include a NRT, but not in the accountability system (KRS 158.6455). Representative Moberly said the statute construction needs to be made more clear, and Mr. Noland agreed that some clarity could be provided there in the future.
Ms. Sims said the accountability weight increases reading and mathematics weights at elementary and middle school levels; uses 50 percent on-demand questions and 50 percent portfolio to calculate the content area for writing; and changes slightly the weights for some content areas and for nonacademic measures. She said the alternate assessment measures the same content areas as state-required assessments. She explained several pie charts for the accountability weights for elementary, middle, and high school. The pie charts are in the meeting folder located in the Legislative Research Commission library.
Dr. Rogers discussed on-going assessment work not reflected in regulation. She said the measure of growth is based on annual reading and mathematics assessments and will have to be discussed in the future.
Senator Kelly asked where PLAN and EXPLORER fit in to the middle schools. Dr. Rogers said PLAN is administered first and EXPLORER is administered in the eighth grade. Senator Kelly asked if the tests were included in middle school accountability weights, and Dr. Rogers said that they were not. She said the EXPLORER test currently given in Kentucky is not in a secure form so it is not included in accountability. Mr. Noland said it would be possible to contract with a vendor to get the test secured, but future appropriations would be needed because of the high cost of securing the test.
Senator Kelly said a goal of Senate Bill 130 was to test for diagnostics and use the information to prepare intervention strategies. He said the second goal was to reduce the amount of test taking wherever possible and incorporate the diagnostic testing into accountability. Dr. Rogers said there are many technical issues to be examined as the ACT is implemented in assessment in Kentucky schools.
In response to a question from Representative Moberly, Mr. Noland said in order to put the EXPLORER test into the middle school accountability, a vendor would have to make the test secure by designing a test specific to Kentucky, that not everyone else in the country can access, and this costs more money than an off-the-shelf test. Dr. Rogers said other states currently do not do this, but are open to discussions about it. Mr. Noland noted that appropriations would be needed if the General Assembly were interested in making the test secure. Representative Moberly asked if there were prior discussions about money for securing the test. Mr. Noland said he did not recall any previous discussions.
Senator Westwood asked about the weights in the high school ACT index. He asked how the weights compared to the four levels of attainment in the CATS, i.e., novice, apprentice, proficient, and distinguished. Dr. Rogers said a 13 is considered a medium novice; 26 is considered high novice; 40 is low apprentice; 60 is mid apprentice; 80 is high apprentice; 100 is proficient; and 140 is distinguished. She noted there is not a number or weighting for low novice.
Senator Westwood compared the scores in Colorado where students all take the same test to the scores in Kentucky's Core Content Test. He said it seems Kentucky's scores may be set too high and could be very inflationary. Dr. Rogers said data was analyzed from Colorado and other states who utilize statewide testing, as well as the benchmarks set by the Council on Postsecondary Education, with regard to the ACT. She asked if Senator Westwood was saying that the range from 12 to 14 is too challenging for Kentucky schools to meet as a low apprentice.
Senator Westwood said Colorado students, even with learning disabilities, suggest that an ACT composite score of 12.3 represents a very low level of performance. He said this is far more in-line with Kentucky novice scores rather than what would be considered a low apprentice. He wondered if Kentucky was setting its standards too low for what achievements are trying to be acquired, and asked if a 10 to 12 score should be considered high novice.
Mr. Noland suggested that since the goal for all students is proficiency, and proficiency starts at 21, anything less than that, is not considered good enough. Dr. Rogers commented that Colorado does not have a scoring system that is comparative because Colorado uses benchmarks set by their universities.
Senator Kelly asked about the technical aspects of using a composite score instead of disaggregated scores. Dr. Rogers said the composite scores were easy for schools to understand because it mirrored the scaling system of the NRT in the past. Mr. Noland said this is only used for the five percent, and will not be used when the ACT is integrated into the KCCT in accountability. Ms. Sims said this approach also keeps focus on the whole performance of the student.
Senator Westwood asked if there was anyone who wanted to speak for or against the regulation. Ms. Cindy Rausch, citizen, spoke for Mr. Martin Cothran, Family Foundation, who could not attend the meeting, and presented testimony that 703 KAR 5:020, the formula for determining school accountability, does not comply with statute for various reasons. One reason included the elimination of a single statewide assessment program as well as the elimination of the NRT at the elementary and middle school levels by eliminating it from the accountability index.
Ms. Rausch said the NRT was originally added to CATS because the public and parents wanted to know how their children compared to the nation and statewide. She said parents wanted to be sure that basic skills were emphasized in classrooms because whatever was being taught in the classroom is what was being assessed. She emphasized that one, statewide NRT was to be administered to all elementary students, and school districts should not be allowed to pick and choose their own assessment tests. She said this was contradictory to the information that parents requested for comparing their children to other students across the nation and within Kentucky.
Ms. Rausch said several state board members have requested that the KDE administer a statewide NRT at the elementary level in the accountability index, but KDE staff repeatedly indicated to board members that it was not necessary. She said the interpretation from Mr. Noland was also included in the statement of consideration.
Senator Westwood introduced Mr. Brent McKim, Jefferson County Teacher Association (JCTA), to speak on 703 KAR 5:020. Mr. McKim addressed some concerns that Senator Kelly noted about why Kentucky should not offset state core content questions with ACT questions.
Mr. McKim said the KDE appropriately identified content coverage and alignment as an issue, but JCTA is concerned about the nature of the questions. He said an accountability assessment should be used to determine the extent to which teaching and learning are occurring in the core content. He believes the ACT is not designed to determine whether or not teaching and learning are occurring with Kentucky core content. He said the ACT is designed to be a predictor of how students will perform in college, and it is the wrong tool to use to assess whether or not teaching and learning are occurring in the classroom.
Mr. McKim said the problem occurs when items on the core content test are replaced with norm-referenced questions, it ensures that Kentucky will never reach 100 percent proficiency. He said the ACT questions are designed and field tested to ensure about half the people taking the test will miss them in most cases. He also said that JCTA supports the aspect of Senate Bill 130 that provides access to the ACT for all students in high school, and some students taking the ACT will score higher than what they thought they could, and as a result may go to college. JCTA supports the state paying for all students to take the ACT, but feels an inappropriate line is crossed when criterion-referenced questions on the state assessment are replaced with norm-referenced questions that are designed for half of the people taking the test to miss them.
Mr. McKim summarized by saying that the affluence of schools can also make a difference. He said parents do not pay for tutoring services or personal computer software for the KCCT test, while many tutors and software programs are available that prepare students for the ACT and SAT tests. Therefore, schools that have a more affluent population would probably have more of the supplemental services and supplemental aides affecting the scores on the ACT assessment than on the state assessment. He urged subcommittee members to provide advice not to use the ACT in place of the KCCT for the reasons afore mentioned.
Representative Moberly asked Mr. McKim if he was speaking prospectively to the future when the ACT replaces some of the core content test. He said if the alignment study indicates that there is some alignment in some subject areas with ACT questions, is there still some question about whether the development of the questions still makes them inappropriate.
Mr. McKim said alignment would be necessary, but is not sufficient to replace core content questions with ACT questions because of the design. The purpose of the ACT assessment is different than the state assessment instrument.
Senator Westwood asked Mr. McKim if he had the same objections to using the Comprehensive Test of Basic Skills (CTBS), which is a NRT, for the same reasons. Mr. McKim said the CTBS is only five percent of the index so it is only a five percent problem, but he thinks it would be better to not use the CTBS test for the same reasons. He said that using the CTBS would not be as bad as the ACT because parents are not buying test preparation software and tutoring services, and it is less biased by affluence.
Representative Moberly asked if Kentucky should be moving toward preparing students for college readiness, and said the ACT does tend to move students in that direction.
Mr. McKim said Kentucky should be preparing students for college, and recommended utilizing performance-based assessments to help with college readiness. Performance-based tests are more engaging for students than a multiple-choice test. He said school districts should be empowered to have a component of the assessment that is performance-based, which they would develop and have approved through the state, ensuring their test is rigorous and appropriate. He recommended following the model of the Commissioner of Education in the state of Nebraska, Mr. Doug Christenson.
Senator Westwood said there is a difference of opinion in what the law says about what a statewide test really means. He feels it was legislative intent to have one statewide test administered to every student across the Commonwealth. He asked permission of other members to see if it would be advisable to ask the Attorney General to review the issue and determine what the language and law really means, and everyone agreed.
Senator Westwood said the subcommittee would meet again in the next week to take action on the administrative regulation since it was time sensitive. He said he would let members know during the last two days of session when the meeting would take place.
With no further business before the committee, the meeting adjourned at 2:35 p.m.