The fifth meeting of the Education Assessment and Accountability Review Subcommittee was held on Tuesday, December 5, 2006, at 1:00 PM, in Room 169 of the Capitol Annex. Representative Harry Moberly Jr., Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Harry Moberly Jr, Co-Chair; Senators Dan Kelly, and Ken Winters; Representatives Jon Draud, Mary Lou Marzian, and Frank Rasche.
Guests: Mr. Clyde Caudill, Kentucky Association of School Administrators and Jefferson County Public Schools; Mr. Wayne Young, Kentucky Association of School Administrators, Mr. Eddie Pawlawski, Cumberland University; Mr. Steve Newman, Northern Kentucky University; Ms. Debbie Hendricks, Kentucky Department of Education; Ms. Cindy Heine, Prichard Committee; and Mr. Marty Bell, Jefferson County Public Schools; Mr. Doug Hubbard, Ms. Janna Vice, and Ms. Judith Gibbons, State Board of Education members.
LRC Staff: Sandy Deaton, Jonathan Lowe, Janet Stevens, and Lisa Moore.
Representative Moberly asked for a motion for the approval of the minutes from the meeting of October 6, 2006. Senator Winters made the motion to approve the minutes, and Representative Rasche seconded the motion. The motion was approved by voice vote.
Representative Moberly introduced Dr. James Catterall, Chair, and Dr. David Miller, member, National Technical Advisory Panel for Assessment and Accountability (NTAPAA), to review the current status of Kentucky's testing and assessment system. Representative Moberly said that NTAPAA was created in 1998 by statute to provide the legislature with technical expertise on assessment and accountability issues. He said the Education Assessment and Accountability Review Subcommittee (EAARS) would like to work closer with the NTAPPA in the future. He also said he and Senator Westwood requested that NTAPAA provide the EAARS with a paper outlining the most current issues and projected changes in assessment, including a review of the steps that need to be taken to implement Senate Bill 130 and the inclusion of the ACT in the accountability system. He said a discussion of the issue of longitudinal accountability for students was also requested. The paper is in the meeting folder.
Dr. Catterall said every six or seven years, large-scale assessment systems need an overhaul. He said a packet has been provided to members that show the inclusion of the ACT, PLAN, and WorkKeys in the assessment system. He said Senate Bill 130 signifies some long standing and pointed concerns about assessment at the high school level. He said issues such as readiness of the students for college, acceptance into college, and performance during college are of primary concern.
Dr. Catterall said another concern is to have a way to identify students in the eighth and tenth grades who may need extra help and provide an intervention, as well as encourage students who are breezing through high school to take a program of more challenging standards.
Dr. Catterall said there are long standing issues regarding the motivation of high school students in Kentucky for taking the Kentucky Core Content Tests (KCCT). He said implementing the ACT at the high school level attempts to alleviate the motivational concerns for students.
Dr. Catterall said the National Technical Panel for Accountability and Assessment (NTAPAA) has seen no reason that the ACT cannot be used in the various ways that it has been proposed in Senate Bill 130. It is possible to augment the ACT with KCCT type items, and have it behave as the KCCT test for eleventh graders. He said the only problem remaining is that NTAPAA does not know how long it will take for the ACT test to work in the intended manner.
Dr. Catterall explained the basic design and the scoring system of the ACT test. He said the alignment of the ACT test with the core content in Kentucky is an outstanding issue, but NTAPAA supports that there is some alignment between the ACT and high school standards in Kentucky, but does not know how much and of what kind. He said it is greater than a 30 or 35 percent range, but less than 70 percent.
Dr. Catterall said it will be interesting to see how many more students end up attending college with an "ACT for all" policy. He said some students will realize they are ready to attend college when they may have previously assumed they were not college ready.
Senator Kelly said another point of the "ACT for all" policy is for students to learn at a very early age that they are not college material when they may believe they are. He said at that time, counseling and intervention strategies would be implemented to get the student enrolled into a more rigorous curriculum in order to overcome the deficiencies.
Dr. Catterall explained the basics of the Commonwealth Accountability Testing System (CATS). He discussed standards settings and said formal standards are set for each component test in CATS. He said CATS could also still be used in the high school to meet the requirement for the No Child Left Behind (NCLB) program, but alignment is again an issue. He noted that the augmented items in CATS would have to be open response questions as all the questions on the ACT are multiple-choice questions.
Representative Moberly asked if an alignment is performed before the development of the augmented questions, or is the entire process part of the alignment. Dr. Catterall said the process itself is part of the alignment. He said the big vision would include administering the ACT once without consequences within the accountability system, but disseminating the scores to all students, and then administering the ACT within the accountability system the following year.
Dr. Miller said the alignment issue becomes an on-going concern when forms are being created by outside vendors, rather than the current KCCT test that is used and created in Kentucky. Dr. Catterall said some of the process would be out of Kentucky's control because if the ACT makes changes to its test, then Kentucky would have to make similar changes if it was to continue to use it in order to keep the integrity of the test.
Representative Draud asked if alignment is realistic or not. Dr. Miller said studies can be conducted to determine how much is already aligned, but it almost certainly is going to have to include an augmented test that is given in conjunction with the ACT. Representative Draud expressed concern about giving another test in addition to the ACT, and Dr. Miller said the ACT would at least have to incorporate some augmented items in order to match the Kentucky core content. Dr. Catterall said NTAPAA supports the notion that the alignment can be done if the appropriate effort is devoted to the task.
Representative Moberly asked if NTAPAA's position was the same as the Kentucky Department of Education's (KDE) that the ACT could be used as a substitute for the current norm-referenced test (NRT) in the 2007-2008 school year, and the augmented test be used in place of the core contest Kentucky that is currently administered the following year. Dr. Catterall said that was correct.
Senator Kelly said he is very concerned about the delay in incorporating the ACT in the accountability system primarily because of adding layers of testing. He said there is significant resistance from some people, but he is a strong advocate of just implementing the ACT into the accountability system right away. He asked Dr. Catterall what the real concerns or negatives in student performance would be in implementing this in the 2007-2008 school year.
Dr. Catterall said the only thing that would harm student achievement would be testing students too much. Senator Kelly clarified the question by asking if the potential problems with the integrity of the test were potential threats to the achievement of the students. Dr. Catterall said it was not a direct threat to student achievement, however, results can be obtained from tests that are not necessarily matched to the items that the teachers taught during the school year, which can cause a problem within the system. On the technical side, a redesigned test, such as an augmented ACT, would have to receive approval from Washington D.C. to ensure standard alignment protocols.
Dr. Miller said it would not harm the student individually to implement the augmented ACT in 2007-2008, but it could hurt the NCLB requirements. Secondly, the validity of the information may be absent that year.
Senator Kelly said those are technical problems that will have to be dealt with whether it is implemented in 2007-2008 or 2008-2009. He said it will be a struggle to align the test to meet NCLB requirements, but it is a struggle the KDE faces now under the current system. He thinks the benefits of incorporating the ACT outweigh the technical problems.
Dr. Miller said there are some advantages for students to take the ACT right away, but requirements for NCLB and ensuring a fair accountability system for the school that year, are both technical issues that have to be addressed prior to implementation. He said the ACT could be implemented next year, but he would not drop the KCCT so quickly until the outstanding issues are resolved.
Senator Kelly said he wants the ACT incorporated as fully as possible next year to eliminate duplication in testing. He said Kentucky can work through the hurdles as they arise, as you would with any revamping of an accountability system.
Representative Moberly disagreed. He said the integrity of the accountability system depends upon the alignment. The ACT will still be a part of the accountability system by replacing the NRT, but the core content cannot be replaced because it ruins the integrity of the accountability system. He said the bill states that the augmented ACT will not replace the core content before alignment has occurred, which will take at least a year. Senator Kelly says he is ready for the change to occur now.
Representative Rasche asked if the discussion was about implementation of the augmented ACT in the 2007-2008 school year and the panel said yes. He asked if it would be possible to administer a smaller version of the ACT in 2007, to a smaller number of people on the same basis as it will be given universally in 2008, for the purpose of beginning the alignment process. Dr. Catterall said yes. Representative Rasche asked if this was a practical approach. Dr. Catterall said a group of 10,000 would be a good size to sample in 2007.
Senator Winters asked if a year and half was ample time to make these testing adjustments. Dr. Catterall said live operational data is missing from students because there is no previous data to see how students perform on the augmented ACT. He said a large scale experiment for one year would provide ample data before implementing the change statewide.
Senator Kelly said the experimental process had been started by administering the EXPLORER test to tenth graders, and asked if most students perform similarly on the ACT as they do on the EXPLORER. Dr. Catterall said that is the belief. Senator Kelly said the data is available now and there is no need to do a large scale experiment and wait another year.
Dr. Miller said there are many things that are just still unknown, such as how large the augmentation will have to be to the ACT. He said if many items already match, it becomes much more doable on a quicker schedule.
Senator Kelly said it is known that there is at least 35 percent alignment already, and possibly as much as 70 percent. He wants to move forward as quickly as possible to reduce the test taking time and duplication, while continuing to study and work towards the goal of alignment.
Dr. Catterall said there are cross purposes surfacing. If Kentucky moves forward in 2007-2008 by administering the augmented ACT, it threatens the integrity of the accountability system. He said if the rest of the NTAPAA panel were present at the meeting, the majority would feel it would be best to test it a year, and then go live the next year.
Dr. Catterall said House Bill 58 in 1998 said a longitudinal component should be added within the assessment system if it could be rendered valid and reliable. He said the CATS test did not test mathematics and reading every year in the elementary grades so it was very difficult to build a solid longitudinal component.
Dr. Catterall responded to the proposal of adding a longitudinal assessment when two years of data is available. NTAPAA has had multiple discussion of possible models and in-use models from other states and is gravitating toward a preferred approach to longitudinal scaling and reporting. NTAPAA has suggested constructing longitudinal scales from grade three to grade eight in reading and math. He said using "vertically moderated" standards to assign performance levels at all grade levels attends to the distributions of performance levels within grade levels; thus students are not simply placed on a continuous linear scale. Vendors, KDE, and NTAPAA are in close communication on this element of the new system. He said NTAPAA is reasonably confident that longitudinal assessment will be supported and implemented in the new testing system design.
Senator Kelly said he thought the longitudinal issue would be addressed by administering the PLAN in the eighth grade, the EXPLORER in tenth grade, and the ACT in the eleventh grade. He thought the tests were predictive of each other and could be tracked to see how students are doing over time. He asked if that was an erroneous assumption on his part or was it part of one of the moderated vertical scales.
Dr. Miller said it is not something that NTAPAA had specifically talked about. If it were to be done, it would have to in conjunction with the ACT, and he does not know if ACT would allow that. He said it would have to be discussed with the ACT group.
Senator Kelly asked if the longitudinal component was legislatively mandated. Representative Moberly said it was discussed, and Senator Kelly said there is no reason Kentucky cannot move ahead in that direction if it waits for one more year to get data. Dr. Miller said it is two different situations. Senator Kelly said he understands that the longitudinal technical challenges are huge, but he feels the alignment issues are small from a technical standpoint.
Dr. Miller said it took so long to create a vertical scale because NTAPAA did not know that it could technically be done when students were tested in subject areas every third or fourth year. He said consecutive years of testing did not occur until this year.
Representative Draud asked if Kentucky's core content is in-line with the ACT. Dr. Catterall said it is not known because the alignment study has not been conducted. Representative Draud said that could create a lot of anxiety among the teachers throughout the Commonwealth if Kentucky utilizes an accountability index that is not known to align with the core content. Dr. Catterall said NTAPAA is working to see if the accountability elements are aligned with the core content. Representative Draud asked when the information from the alignment of the core content to the ACT would be known. Dr. Catterall said it would certainly be known after an administration of the test. The administration of the test provides student level data of all items on the test for comparison purposes. Representative Draud asked if the KCCT and the ACT test had ever been compared.
Senator Winters said he has samples from various people that have compared the two tests with one comparison showing an 80 percent alignment. He said this does not include student performance, but just comparisons of the two tests. He feels that NTAPPA should look at the two tests for comparisons and give their advice with their specific testing expertise.
Senator Kelly responded to Representative Draud that data can be used from the EXPLORER test administered to tenth graders and by other methods of comparison. He said the current test is not 100 percent aligned. Dr. Miller said a sample of items are used from the core content so there is never 100 percent alignment. Dr. Catterall said every item on the test is aligned to the core content for assessment, but the entire content for assessment is not represented on the test.
Senator Kelly reiterated that it is better to start implementing the test now and continue to work on the alignment or it could drag out another 18 months. Dr. Miller said depending upon how much alignment is there, he is concerned about creating the test in 12 months time. He also said the supplemental materials will need to be field tested.
Senator Kelly said if the alignment is pretty good then it will take much less time to prepare. Dr. Miller agreed. Senator Kelly asked if the study of alignment was being delayed. Dr. Miller was not sure, and Senator Kelly said it was a question for the KDE. Representative Moberly asked Ms. Pam Rogers, Associate Commissioner of Education and Accountability, KDE, to come to the table.
Ms. Rogers said the KDE has submitted a Request for Proposal (RFP) to have an outside vendor look at the alignment of the ACT with the core content, and a vendor will be selected this month. Ms. Rogers said the responses to the RFP had been received and interviews would begin in a week with a member of NTAPAA participating as well.
Representative Rasche asked to what degree the ACT is a criterion-based test versus a norm-based test. Dr. Miller said all tests are never completely one or the other, and Representative Rasche said to what degree. Dr. Miller said the ACT is a norm-referenced test for the most part and that is why supplemental materials will be so important to cover the criterion-based needs.
Representative Rasche said he feels the ACT may discriminate toward the traditional college-bound students and the questions could be oriented to the expertise in that area. He asked how the test would distinguish between a high and medium novice, and some questions need to be geared toward the lower end students because Kentucky needs that assessment information.
Dr. Miller said he thinks the ACT covers a wide range of items, but it will be an issue that is looked at as the tests are studied. Representative Rasche said it was an intuition totally unfounded, but sometimes it is the level within the core content that needs to be assessed.
Representative Rasche asked about using a smaller sampling size for a pilot such as 3,000 or 5,000 students. Dr. Miller said he is not sure about the exact number, but the bigger issue is if there is an adequate sampling to get a broad range of Kentucky students.
Representative Moberly recognized the state board members in the audience. He communicated to them that they cannot get a clear sense of the implementation of Senate Bill 130 because the subcommittee is divided on the issue. He agrees with NTAPPA and the KDE on the implementation of the core content in the ACT, and said moving ahead aggressively is reckless and goes against the integrity of the accountability system.
Senator Kelly said if KDE had expressed an opinion to Representative Moberly he wished they had communicated it to him. He has not seen a response or communication from the KDE on their position of the implementation of Senate Bill 130. He said it was communicated in the meeting that the implementation could occur right away, although it may be preferred to have more data to preserve the integrity of the test and address other technical issues. He said advantages of implementing the ACT in 2007-2008 include encouraging student motivation and avoiding duplication of testing, while decreasing the time spent testing. He hopes it does not take eight years to get implemented, and Representative Moberly concurred.
Representative Moberly introduced Ms. Lu S. Young, Superintendent, Jessamine County Public School System and a member of the School Curriculum, Assessment, and Accountability Council (SCAAC) to discuss Senate Bill 130. She said she is supportive of all the juniors and seniors taking the ACT and believes that students will perform better on the ACT resulting in a higher college going rate in Kentucky.
Ms. Young said schools should be informed in a prudent way about changes in the accountability index to give them time to respond to new demands. She said it takes time for schools and teachers to adjust to major changes in the accountability system, and what seems like small technical changes can have large impacts when they are actually implemented in the schools. She said schools are well on the way to reaching proficiency by 2014, and changes in the accountability system could throw some schools off course.
Ms. Young said once it is determined how much the augmented ACT will weigh in the accountability system, schools will have to allocate resources, such as time and money, to prepare their students to do well on the ACT. She also said that 2008 is the second year of the Kentucky CATS biennium, and this is a vulnerable biennium as all these changes are occurring. She urged the committee to follow the bill language and administer the ACT to all students in 2008, and use this live data in the second year of the biennium to make some significant and important decisions.
Senator Kelly said he appreciated her optimism about the forthcoming changes. He understands why people in the field are also apprehensive about making these changes. He said there is a concern on the part of the legislators that over half of the Kentucky schools are not meeting their performance goals and are not on their way to proficiency by 2014. He also said the ACT test does not test unknown information, but helps to prepare students for college, which is one of the ultimate goals for students.
Representative Draud asked Ms. Young her opinion on students having too much testing. Ms. Young said she agreed with Representative Moberly that Kentucky can go ahead and include the ACT and PLAN as the NRT without making any dramatic changes. She also said 65 percent of public school students in Kentucky take the ACT now, so for the additional 35 percent of the students, the ACT will be an additional testing burden. She said the augmentation will work, but she believes open-response questions is a fundamental component of CATS that truly makes a difference.
Representative Draud asked if Ms. Young had any suggestions on tests that can be eliminated or consolidated. Ms. Young said the new KCCT package has done a good job of shortening areas not tested by open-response such as practical living, vocational studies, and arts and humanities. She does believe the augmentation of the ACT can result in a shortened level of testing, but she is not supportive of this in the 2007-2008 biennium.
Representative Rasche asked at what point the proposal became ACT "augmented" instead of ACT "supplemented". He believes that the KCCT should be the core and the ACT should be used to off-set some parts of it. Ms. Young said she appreciates that fundamental notion of KCCT supplemented as opposed to augmented. She cautioned not to produce a final product with just an open-response augmentation or supplementation because students who do not test well tend to do better starting out with multiple-choice questions.
Representative Moberly introduced Ms. Marcia Seiler, Director, Dr. JoAnn Ewalt, Research Division Manager, and Mr. W. Bryan Jones, Investigations Division Manager, Office of Education Accountability (OEA) to give their 2006 annual report, 2006 school finance report, and the proposed research agenda. Ms. Seiler said the General Assembly in 2006 modified KRS 7.410 that outlined the duties of OEA resulting in important changes. One is that OEA prepares a research agenda to be approved by the EAARS subcommittee. She said five research items have been developed for 2007 as OEA sought input from members of EAARS in developing the research topics.
Dr. Ewalt gave a Power Point presentation on the 2006 school finance report. She also told members that a complete 2006 school finance report was in their meeting folders.
Dr. Ewalt said OEA presented the 2005 school finance report in June, 2006. It contained unaudited, preliminary data for fiscal year 2005, and concluded that the equity gap has been improving in the past three years, and is narrower in all of the years since the implementation of the Kentucky Education Reform Act (KERA). From 1990 to 2005, the equity gap was reduced by 4 percent. She also said from 1990 to 2005, the inflation-adjusted equity gap was reduced by 36 percent.
Dr. Ewalt said the 2006 school finance report shows the equity gap is wider than initially reported. It was based on updated data for local, state, and federal school district revenue and contains audited, final data for fiscal year 2005. It also contains minor revisions for fiscal year 2003 and 2004.
Dr. Ewalt said wealth Quintiles are established by ranking districts from lowest to highest, divided into five groups, each with one-fifth of the state's students. She said Quintile 1 represents the lowest property wealth districts and Quintile 5 represents the highest property wealth districts. She explained the calculations for defining and analyzing the equity gap. A detailed handout is in the meeting folder located in the Legislative Research Commission.
Senator Kelly asked if the gap was referring to the gap between the total amount of funds received state and locally per pupil. Dr. Ewalt said yes, the Quintiles were established using wealthier and less wealthier districts in order to establish if they are improving over time.
Senator Kelly asked if the receipts included construction funds as well as operational expenses. Dr. Ewalt said the School Facility Construction Commission (SFCC) funds were not included in determining the equity gap, only operational expenses.
Dr. Ewalt said when comparing fiscal year 2005 audited to unaudited per-pupil revenue amounts, the districts in Quintile 5 (highest wealth) experienced greater increases in local and state revenues than the districts in the other quintiles. When comparing fiscal year 2005 audited per-pupil revenue amounts to fiscal year 2004 levels, the districts in Quintile 5 experienced greater increases in local revenues than districts in other quintiles. Quintiles 1-4 had more state revenue, but not enough to offset the gains Quintile 5 made in local revenue.
Representative Draud said the equity issue in Kentucky is good compared to the nation. Dr. Ewalt said Kentucky was in the forefront addressing equity issues, but was not sure if it is now. She will have to research to see how Kentucky compares to other states regarding equity issues currently.
Representative Draud asked if the revenue behind each pupil included the general fund and the building fund. Dr. Ewalt said it does not include the building fund, only the general fund.
Senator Kelly said he knows the figures were adjusted for inflation, but what happens if it were adjusted for cost of living. Dr. Ewalt said it would be possible to do a regional adjustment as well, but it has not been done thus far because there is no commonly accepted regional adjuster for Kentucky. Senator Kelly said he would like to see some figures of this nature, and Dr. Ewalt said she would get some figures for him. Ms. Seiler said the finance report for next year will include some different scenarios.
Senator Kelly said the districts that are growing, primarily located in the higher Quintiles, have money to hire teachers, but are struggling to keep up with the cost of building new facilities. Dr. Ewalt said growth districts were looked at in the SFCC report last year, but in general, the growth nickels and other things available to rapidly growing districts, have assisted these districts and they have lowered their unmet need faster than other districts. In some cases, districts have reported that once the building is open it is very hard to maintain, while other districts report that their problem is the initial construction costs.
Senator Westwood said the federal revenue impact on these issues was not discussed. He would like the impact of federal funding included in order to get a full picture. He also said the larger report indicates that the gap gets larger when federal funds are factored in for Quintiles 4 and 5. He asked why this because it would seem that Quintiles 1, 2, and 3 would receive the most federal dollars.
Dr. Ewalt said Fayette and Jefferson Counties really drive those figures because they receive so much in federal funds. In fact, Jefferson County had $3 million more in federal dollars in 2005 than in 2004. She said the smaller districts and lower quintile districts proportionally received more federal dollars, but in terms of actual dollars, they get less.
Dr. Ewalt discussed the 2007 research agenda for OEA. The topics include: 1) review of the flexible focus fund program; 2) review of the extended school services program; 3) understanding how tax provisions interact with the Support Educational Excellence in Kentucky (SEEK) formula; 4) 50-state education rankings compendium; and 5) review of the assessment and accountability of special needs students.
Senator Kelly made the motion to accept OEA's 2007 research agenda, and Representative Draud seconded the motion. The motion was approved by voice vote. Dr. Ewalt asked the members to approve the other studies that were presented in the past, such as the finance reports as well as the School District Efficiency and Effectiveness Report. Representative Moberly said the amendment to the motion was accepted, and no members had an objection.
Ms. Seiler briefly discussed the 2006 annual report. She said the OEA 2006 annual report is included in the members' folders located and a copy is located in the LRC library. Some highlights included: a background summary of investigations carried out by the OEA; summaries of research conducted in 2006, including a review of the SFCC, indicators of efficiency and effectiveness in primary and secondary education spending, and the 2005 school finance report.
Representative Moberly made a motion to accept the previous reports presented by the OEA. Senator Kelly made the motion to accept, and Representative Draud seconded the motion. The motion was approved by voice vote.
Senator Kelly would like OEA's annual report on the agenda for the next EAARS meeting. Representative Moberly agreed, and commended Ms. Seiler's leadership in getting the EAARS committee and the OEA in sync. He said OEA is following the direction that EAARS wanted and doing very good work.
With no further business before the committee, the meeting adjourned at 3:20 p.m.