The third meeting of the Education Assessment and Accountability Review Subcommittee was held on Thursday, August 31, 2006, at 11:00 AM, in Room 154 of the Capitol Annex. Representative Harry Moberly Jr., Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Harry Moberly Jr, Co-Chair; Senators Dan Kelly, and Ken Winters; Representatives Jon Draud, and Mary Lou Marzian.
Guests: Mr. Ken Draut, Jefferson County Public Schools; Mr. Jerry Whealen and Ms. Patty Wallace, Lawrence County; Ms. Madelynn Coldiron, Kentucky School Board Association; Ms. Mary Ann Blankenship, Kentucky Education Association; Mr. Steve Newman, Northern Kentucky University; Mr. Clyde Caudill, Jefferson County Public Schools, the Kentucky Association of School Administrators, and Data Resource Corporation; and Ms. Cynthia Heine, Prichard Committee.
LRC Staff: Sandy Deaton, Audrey Carr, Jonathan Lowe, Janet Stevens, and Lisa Moore.
Representative Marzian made a motion to approve the minutes of the June 13, 2006 meeting. Senator Winters seconded the motion, and the motion was approved by voice vote.
Representative Moberly introduced Ms. Marcia Seiler, Executive Director, Office of Education Accountability (OEA), and Dr. Jo Ann Ewalt, Research Manager, OEA, to present the final report of the Efficiency and Effectiveness of Education Resources study. Dr. Ewalt said the General Assembly directed the OEA to conduct an inventory and an assessment of indicators that can be used to evaluate a school district's efficiency and effectiveness. The approved study proposal also directed the OEA to study primary and secondary education spending in Kentucky.
Dr. Ewalt defined efficiency and effectiveness. She said school and school districts are considered effective when they achieve stated goals, however measuring effectiveness is surprisingly complex. When some outcomes are achieved, some students are well served, and some goals are met, but not others, it is very difficult to define effectiveness. She said policy goals must be explicitly identified in order to define what it means to be effective, otherwise policy makers would have no way of knowing which schools or districts are performing the best for their given level of spending, and whether the spending levels are appropriate.
Dr. Ewalt said that between 1990 and 2005, state appropriations have grown 109 percent in nominal terms and 39 percent in constant 1990 dollars. Inflation over the fifteen year period grew 51 percent, while growth and student enrollment remained relatively flat. She also said that averages mask a lot of variation within school districts as many districts in northern and central Kentucky have grown, while there were declines in many eastern and western districts. She said about half of Kentucky school districts have grown over the fifteen year period, with an average of about 13 percent, and about half have declined, with a change of about 17 percent.
Dr. Ewalt said when analysts examine allocation of education resources, they generally focus on three categories including: instruction; instructional support; and non-instruction. She said spending in these areas for Kentucky are in-line with national averages. The main five spending areas are: salaries; benefits; purchased services; supplies; and other spending, with 72 percent being spent on salaries.
Dr. Ewalt described the variations in current per-pupil spending by district characteristics which are: location; size; poverty; wealth; and student performance. She said analysis does not show cause-and-effect, and cannot show why the relationships exist at the district level, or how they relate to efficiency. Analysis shows patterns in spending that could be explored using more precise efficiency models.
Dr. Ewalt said variations in current spending per pupil varies by district characteristics based on enrollment, poverty, wealth, and performance. High poverty districts spend an average of $7,785 per students, and low poverty districts spend an average of $6,920 per students. She also said districts with the lowest performing Commonwealth Accountability Testing System (CATS) index scores spend about $700 more per pupil than those with the highest scores.
Dr. Ewalt discussed the variations in salary schedules for rank and years of experience for teachers by district characteristics. She said high poverty districts pay teachers almost $3,000 less than low poverty districts, high performance districts pay teachers over $1,700 more than low performance districts, and high wealth districts pay an average of $7,500 more for a Rank I teacher with twenty-five years experience.
Dr. Ewalt said teachers entering into the system with a baccalaureate degree generally have a Rank III, with teachers being required to acquire a Rank II within ten years. She said the highest ranking for a teacher is a Rank I, and advances in rank are aligned with post-baccalaureate education. High poverty districts tend to have fewer Rank I teachers, and there is also great variances within the Area Development Districts (ADD's).
Senator Kelly commented that the PowerPoint slide showed a high percentage of Rank I teachers in southeastern Kentucky, which is primarily a high poverty region, but this disputes Dr. Ewalt's assertion that high poverty areas have fewer Rank I teachers. Dr. Ewalt said there is a concentration of Rank I teachers in southeastern Kentucky, but the analysis shows there are fewer Rank I teachers in high poverty areas across the state. Senator Kelly said district by district numbers must represent something different than geographic areas, and Dr. Ewalt agreed. She also noted that half of all Kentucky teachers are at the Rank II level, with the other half of teachers being evenly distributed between Rank I and Rank III.
Dr. Ewalt said the underlying cause of district variation in spending and staffing is unknown, and the link to efficiency and effectiveness is unknown. She said that they demonstrate the need for more precise measurement methods, and the need for valid and reliable data.
Dr. Ewalt discussed the chart of accounts and its functions, programs, and projects. She said there are problems with data integrity because of: 1) federal coding instructions; 2) state-level expenditure coding, including expenditures coded in the wrong place; and 3) general expenditure reporting concerns.
Dr. Ewalt said that because there are no lower level function, object, and program codes, it is impossible to determine what districts are spending on professional development. She said based upon the OEA's analysis, it was determined that current spending was overstated between one and two percent between 2001 and 2005. She also said there some financial reporting issues that would say that current expenditures are understated, such as districts not reporting on-behalf payments for technology and student activity fees, which can impact the reliability of the data as well.
Dr. Ewalt discussed indicators for measuring efficiency and effectiveness, and said there are several methods used for measurement purposes. Good models recognize: student-level factors; nested data; changes over time; relationships among indicators; and regional differences. She concluded that it is possible to evaluate efficiency and effectiveness using available data and the methods and indicators reviewed in the report, however, accuracy and precision are reduced by data integrity problems.
Dr. Ewalt said the OEA has several recommendations for the Kentucky Department of Education (KDE). They are: 1) the KDE should review problems and concerns raised with data integrity; 2) the KDE should consider improvements to its data integrity efforts; 3) the chart of accounts should offer detailed descriptions and examples; 4) the KDE should implement training for district staff and improve coding; and 5) the KDE should report lower level function, object, and program codes that would allow the impact of spending to be evaluated.
Representative Draud asked if OEA found any other states who have already developed good models in relation to these efficiency and effectiveness issues. Dr. Ewalt said there are several models, but no ideal ones. All models have their own problems, and a lot of work needs to be done in this area. Research models are complex, and can create skepticism among people if they do not understand them completely.
Representative Draud asked if it was realistic to expect that Kentucky can develop a model that will be reliable and valid, and measure effectiveness and efficiency. Is it worth all the time and effort involved to get to the final product? Dr. Ewalt said she does not know if it is worth the effort because Kentucky has not done it before. She does know with more emphasis being placed on accountability that a look at how efficient districts are at producing outcomes deemed important to the state is, at least in theory, a worthwhile thing to do. She said it is realistic to say that a credible attempt can be made, but the data integrity issues need to be addressed before Kentucky tries to build models.
Representative Moberly asked if there are any current models that follow students into postsecondary education and relates it back to the secondary experience. Dr. Ewalt said there have been a number of smaller studies to obtain the large data sets needed to do that, but it is still difficult to track. The smaller studies look mainly at completion of college within four or five years, and future earnings as the two main categories to track.
Representative Moberly asked Dr. Ewalt to explain the problems of measuring the effectiveness of how well a district was teaching math. Dr. Ewalt said at this point, math core content expenditures are rolled up into regular instruction, which includes math and reading and others, and it is not possible to look at just spending for math programs. She said it is possible to do it, and there are codes in place, it is just a matter of instructing districts to do it, and having the software to break it out.
Representative Moberly asked if the "65 percent solution" was a model that some states are utilizing. Dr. Ewalt said a number of states are looking at this, and a few states have passed measures requiring the use of the "65 percent solution". She said the OEA did not study this solution in detail, but she did review research on this, which seems to suggest that there is no magic number, and rather than trying to achieve 65 percent, states should take a closer look at what actually is happening with the dollars being spent. Standard and Poor's has issued a report that says there is not a great correlation between student outcomes and spending 65 percent. She also mentioned that the Education Commission of the States released a new report that states that spending on instruction and instruction-related activities, was at 65.8 using new definitions. The report shows Kentucky spending at 65.7 percent in instruction.
Senator Westwood said the chart of accounts and inadequate training for staff has been a significant problem throughout the years since Kentucky implemented codings. He feels Kentucky is weak in this area, and needs better definitions to define spending in instruction and instructional support. He asked how easy it would be for the Kentucky Department of Education (KDE) to provide detailed descriptions and examples to the chart of accounts.
Dr. Ewalt said the KDE already has a template for the chart of accounts, and just needs to update descriptions. She said the KDE would have to answer the question of how long it will take.
Senator Westwood asked about consequences for districts reporting spending using different definitions. Dr. Ewalt said it poses consequences in a couple of areas, mainly because Kentucky cannot study how effective school districts are at spending money if it is not clear how districts are spending it. She also said if there are significant errors in a school district's coding, it can hurt them at the local level, while making it difficult to compare spending with other states.
Senator Westwood said the last recommendation about reporting lower level function, object, and program codes to allow the impact of spending to be evaluated seems to be extremely important. He asked if the codes should be broken down from a general expenditure into a more specific expenditure. Dr. Ewalt said the function describes if the expenditure is in instruction, instructional support, or non-instruction; the objects actually explain exactly what the expenditure is, and the program codes identify what subject area it was spent in.
Representative Moberly asked if anyone has a handle on what schools are spending for professional development. Dr. Ewalt said it is clear what schools have appropriated for professional development and services they have received, but there are many funding sources used for professional development so it makes it unclear to get a realistic idea of what schools are actually spending.
Senator Winters discussed program coding, and the magnitude of work involved in coding every academic involvement in the institution. He said it is realistic to work on the critical areas of math and reading, but to leave it limited to those areas at this point.
Dr. Ewalt said there are certainly cost benefit questions that the KDE will have to address. She also said that for most of the OEA's recommendations, the codes are already in existence, but the software issue needs to be addressed so the codes are not rolled up into something more general. She said the KDE needs to review the cost and benefits of the recommendations.
Representative Draud said he thought the KDE had a chart of accounts with examples and detailed descriptions. Dr. Ewalt said the current chart of accounts does not include a narrative to explain what expenditures are included and excluded in certain areas. She said they used to have a document like that, but it was never updated after they made changes to the chart of accounts.
Representative Draud said he does not understand why professional development funds cannot be tracked within school districts. Dr. Ewalt introduced Ms. Sabrina Olds, Financial Analyst, OEA, who said school districts are spending more than the grant money they have designated for professional development, such as Title I money and general fund dollars, therefore it would be necessary to have the lower level function codes to get a clear picture on what is actually spent on professional development. Representative Draud asked why school districts cannot list the expenditures in a professional development category. Ms. Olds said they do, and the district can analyze the information, but when the annual financial report is generated to send in to the state, the software program, MUNIS, rolls the lower level functions up into a higher level code, and thus makes the breakout for professional development impossible.
Representative Draud said 93 percent seems like an extremely high number for school districts to be spending on expenditures for instruction and support services. Ms. Olds said the reason for the 93 percentage is because districts are showing on-behalf payments such as retirement and health insurance.
Representative Moberly asked if an implication of the report is showing that Kentucky does not know what management practices or spending patterns would best drive student achievement. Dr. Ewalt said yes, it is fair to say that data integrity issues need to be addressed before analyzing data at school districts across the state. Representative Moberly said this hampers Kentucky's ability to provide management assistance.
Representative Moberly introduced Mr. Gene Wilhoit, Commissioner, KDE, to discuss proposed changes to the state's accountability system. Commissioner Wilhoit said there were several Commonwealth Accountability Testing System (CATS) academic and nonacademic measures implemented from 1998-2006. The three categories that determine a school's accountability are: 1) the academic index, this includes student performance in seven content areas of the Kentucky Core Content Test (KCCT); 2) the nonacademic index, this includes attendance, retention dropout, and transition to adult life; and 3) the norm-reference test (NRT) index, which includes student performance in reading and mathematics on a NRT, which is the Comprehensive Test of Basic Skills (CTBS).
Commissioner Wilhoit outlined several questions and recommendations for changing the CATS accountability index. Questions included whether the weight of reading should be increased; whether the accountability calculations would include a measure of growth using grade to grade assessments in reading and mathematics; should the focus on all content areas be maintained; and how the ACT should be utilized.
Recommendations included: 1) Increase weights for reading and mathematics to 21 percent for reading and 21 percent for mathematics at elementary, and 19 percent for reading and 19 percent for mathematics at middle; 2) Begin immediately with accountability taking effect when two years of data are available and include a growth measure as part of the reading and mathematics indices; 3) Retain equal weights for the content areas of reading, mathematics, science, social studies, and writing and the combination of art and humanities and practical living and vocation studies; 4) Beginning in 2008 after the first administration of ACT, utilize aligned items from the ACT and augment with the KCCT items to calculate indices for reading, mathematics, science and on-demand writing; 5) Do not include a norm-referenced index and redistribute the five percent to all content areas; 6) Assign the five percent weight of the NRT Index to a new National Comparison Index (NCI), a value calculated using normative data; 7) In 2007, after the first administration of PLAN in the fall of 2006, generate the NCI based solely on PLAN. After the first administration of ACT in 2007-2008, generate the NCI index by combining PLAN and ACT results; 8) Change the total weight and the distribution of weights within the nonacademic index; 9) Beginning in 2008, set benchmarks for individual student attendance; 10) Beginning in 2008, use the completion rate (number of years a student spends in each grade span) to capture both retention and dropout rates; 11) Beginning in 2008, use the graduation rate to capture both dropout and retention; and 12) Beginning in 2007-2008, utilize ACT, WorkKeys and Advanced Placement exams as measures of successful transition to adult life.
Senator Kelly questioned the growth concept. He also said the ACT should be utilized as often as possible to offset testing duplication, testing time, and provide student feedback.
Senator Winters expressed concerns about duplicative testing in the elementary grades. Commissioner Wilhoit said the KDE's recommendation is eliminating duplicate testing, and giving schools a choice whether they want to give the NRT test or not.
Representative Draud asked if the accountability model will require that the school district collects data on all students and parents. Commissioner Wilhoit said additional measures will ensure more conversation between the schools and the parents.
Representative Moberly expressed concern about the problems with data accumulation and feedback in the technology age. He expressed support and affirmation of the student assessment from year to year and the growth factor enhancement.
Senator Westwood asked about the proposed weights for dropouts and retention. Commissioner Wilhoit said the intent is to become more finite, and to line-up with federal requirements around graduation rates. He said the intent is not to reduce the emphasis on dropouts and retention.
Commissioner Wilhoit discussed high school accountability weights and the legal parameters that Senate Bill 130 requires. He said ACT items combine with Kentucky-developed KCCT items to produce a Kentucky scale for each subtest (reading, mathematics, and science). He said ACT is scheduled to come to Kentucky at the end of September to conduct an external, unbiased study to see how well the ACT matches up with the core content requirements. He said those items will be used as a part of the measure for the core content, and those items will be supplemented around three criteria: 1) what content in the core content is not measured by ACT; 2) what level of cognitive challenge is not measured; and 3) what modality deemed important in Kentucky's testing is not measured.
Senator Kelly asked if the ACT was starting to develop open-response questions. Commissioner Wilhoit said only in the area of writing.
Senator Kelly also asked if the ACT would still be counted as the norm-referenced component. Commissioner Wilhoit said there is a possibility that Kentucky will use PLAN as well as the ACT in the NRT calculation, and this has some advantages, or it could be only the ACT is used.
Representative Draud asked who would be responsible for performing all of the calculations in the proposed recommendations. He also asked if the items to be correlated with the ACT would be more rigorous in the accountability model than Kentucky's present testing system, and can these be measured against any national norms.
Commissioner Wilhoit said the anchor is not the items, but the core content and the program of studies. He said the KDE has spent the last year and a half aligning the core content against the postsecondary requirements and the measures that have been established by five national organizations through an organization called Achieve. The expectations were developed from higher education and the workforce. The new items will only be more rigorous to the degree that the core content was not sufficiently measuring the expectations of higher education or the workforce. Representative Draud asked when this would be implemented, and Commissioner Wilhoit said this year.
Commissioner Wilhoit discussed the security issues surrounding the EXPLORER and the PLAN assessments. The EXPLORER will be used as a diagnostic tool because the test is used over and over, with the results released, making it not secure. The PLAN items are changed annually, making it secure and alright to use as part of the assessment of students.
Senator Kelly asked if Kentucky's existing NRT was secure. Commissioner Wilhoit said that it was secure, because the items were not released. The only problem with the prior NRT was that is was administered over a long period of time, which posed a problem for leakage.
Senator Winters asked about the NRT remaining at the same weight of five percent with the impact of the implementation of the ACT. Commissioner Wilhoit said the impact of the ACT cannot be measured with the single measure of the NRT. He said when it is factored into the core content test results, it significantly increases the impact of the ACT.
Senator Kelly said it is possible that the ACT will adequately cover all the areas of the core content and will just need the addition of a modality to add some open-response questions. Commissioner Wilhoit said that it is possible. He thinks the percentage of coverage will differ in each core content area with greater alignment in reading than in science.
Senator Kelly said if the ACT will cover the core content areas it will positively impact the length of time it takes to administer the test, the length of time to get the results back, and value of the results at the school level because they will have some longitudinal data. Commissioner Wilhoit said it is complex because there is a loss by not releasing the items with a secured test, however the parents will receive a feedback report, but it will not be linked back to specific items.
Commissioner Wilhoit said he is not sure that the KDE is getting a true representation out of the current nonacademic measures. It appears some students have good attendance patterns while other students have atrocious attendance patterns. Attendance patterns have a direct correlation with dropout rates, and he would like to build into the accountability system an identifier to target those students with poor attendance. He said the KDE may set benchmarks for individual student attendance as opposed to overall averages. He explained the formula for attendance and graduation rate calculation information in detail and this information is located in the meeting folder in the Legislative Research Commission library.
Commissioner Wilhoit would like to see Kentucky go to a consistent graduation rate that is being reported across the country in the states that can do it. He said this is the movement nationally, but all states will have to put in place data systems that can collect the information.
Representative Draud asked how many states have agreed to be consistent with data. Commissioner Wilhoit said all the states, but one, agreed to this consistent reporting of graduation data rates. Representative Draud asked for a timetable for implementation. Commissioner Wilhoit said in order to reach the calculation, there must be a system that tracks progress of individual students over time. In the past, Kentucky has used proxies and relied heavily upon the accurate reporting of school districts with no way to verify the information.
Representative Draud asked about tracking students who move to other states. Commissioner Wilhoit said any student moving in Kentucky will be able to track their progress through the school years, and it may be possible to work out an agreement between surrounding states. He said another way to track those students is through the feedback records between the sending and receiving school district. If that does not occur, a false negative may occur and a school might be punished for a student who does graduate in another state.
Representative Moberly said it is disappointing to have all of these data collection issues in a society that is so dependent upon technology. Commissioner Wilhoit said the General Assembly gave the KDE some tools during the 2006 Regular Session to solve most of the data problems.
Senator Westwood said he is concerned about schools not being penalized if a student takes longer than four years to graduate. He would like for Kentucky to adopt a policy that allows students to get extra help after school, on weekends, or by repeating a grade, to ensure that they are completely ready to move on to the next level.
Senator Winters said he would like to see a study conducted about dropout, retention, and graduation rates, as well as behavior problems in the classroom, for students taking rigorous or advanced placement courses versus students who are not challenged within the classroom. Commissioner Wilhoit said not all dropouts are low performers, and disinterested students make the decision to drop out as well. He also said low performing students participating in a rigorous course, and receiving a lower grade than in a less rigorous course, have a higher performance level.
Senator Kelly received feedback from chief executive officers in factories in his area that would like to expand their companies, but cannot due to a inadequately trained workforce. He said Senate Bill 130 adds some prescriptives to the WorkKeys, and allows companies to visit schools and communicate to students who are not on a college path that if they obtain a certain score on the WorkKeys, they will be guaranteed a good job with their company. Commissioner Wilhoit mentioned that schools should receive credit for promoting employability certificates among students, as well as receive credit for students successfully transitioning to adult life.
Representative Moberly introduced Mr. Jerry Whealen, Lawrence County, who reported on the on-going situation within the Lawrence County School District. Representative Moberly said the members were aware of the problems in Lawrence County. He said the OEA report has been forwarded to the KDE and the subcommittee would be watching the Commissioner's actions on the matter.
With no further business before the committee, the meeting adjourned at 1:30 p.m.