The Education Assessment and Accountability Review Subcommittee met on Friday, May 14, 2010, at 1:00 PM, in Room 131 of the Capitol Annex. Senator Jack Westwood, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Kent Stevens, Co-Chair; Senators Vernie McGaha, and Ken Winters; Representatives Bill Farmer, Mary Lou Marzian, and Harry Moberly Jr.
Guests: Kevin Brown, General Counsel, Ken Draut, Associate Commissioner, Kentucky Department of Education; and Tanya Bromley, Kentucky Music Educators Association.
LRC Staff: Sandy Deaton, Audrey Carr, Janet Stevens, Ken Warlick, and Lisa Moore.
Representative Stevens made the motion to approve the minutes from the January 11, 2010, meeting and Senator Winters seconded the motion. The motion was approved by voice vote.
Consideration of Administrative Regulations
Mr. Kevin Brown, General Counsel, and Mr. Ken Draut, Associate Commissioner, Kentucky Department of Education (KDE), explained administrative regulation 703 KAR 5:060. The regulation establishes an interim assessment process as required in Senate Bill 1, enacted by the 2009 Kentucky General Assembly. The interim assessment period provides a transition time for the state assessment and accountability system. 703 KAR 5:060 is now revised to manage a significant change based on a statistical concordance methodology as a two-year bridge to link data in the state assessment and accountability system following a significant system change. The interim assessment process regulation outlines required assessments, data collection and reporting through the interim assessment period. Chairman Westwood asked for a motion to accept technical amendments made to 703 KAR 5:060. Representative Stevens made the motion to adopt the technical amendments and Senator Winters seconded the motion. The amendments were adopted by voice vote. Representative Farmer made the motion to approve administrative regulation 703 KAR 5:060, as amended, and Representative Stevens seconded the motion. The administrative regulation was approved by voice vote.
Mr. Brown discussed administrative regulation 703 KAR 5:180. Due to substantitive changes that need to occur in the regulation that would require action by the Kentucky Board of Education during its June 2010 meeting, voting was deferred on the administrative regulation until the next meeting. In response to a question from Senator Westwood, Mr. Brown said the deferral of the regulation would not delay the implementation of the corrective actions that will be required of the schools that are deemed to be under achieving or low-performing schools.
Consideration of Office of Education Accountability Reports
Chairman Westwood asked for approval of the two reports generated in the Office of Education Accountability (OEA) that were presented to the subcommittee in January, 2010. Representative Stevens asked how the data was collected for the survey relating to the report “Study of Leadership Training for Superintendents, Board Members, Principals and School-based Decision Making Councils.” Ms. Seiler said all the various groups were surveyed and asked for their opinions. In response to a question from Representative Stevens, Mr. Keith White, Analyst, OEA, gave a brief PowerPoint presentation explaining the validity of the survey and why some schools are performing at such low levels when leadership training is being reported as successful in the school. Mr. White reviewed the following conclusions of the report: statutory training requirements are being met; survey respondents credit training for positively impacting their duty-related preparedness; most survey respondents consider training requirements to be appropriate; collaborative training effects were present; and no linkages were found between levels of leader perceived preparation and academic performance. He noted that the OEA received very limited survey responses from the schools recently identified as being audited. He also said there is insufficient data to determine training effectiveness, and explained briefly a model that would be necessary to conduct such study.
In response to a question from Representative Farmer, Ms. Seiler said the survey questions were very straight forward and she believes the intended recipients of the survey completed the answers. Mr. White noted some of the comments were very personal and he did not see any reason to believe that the survey recipients had someone else to complete their answers.
In response to questions from Senator Westwood, Mr. White said more research is needed to determine if the leadership training is making a difference. He said the compliance model is in place. It is very expensive and time consuming to determine if an individual’s behavior actually changed due to the completion of the training. It is sufficient to say that the systems are not currently in place to immediately judge the effectiveness of the training.
Mr. White said there is not any merit in utilizing research from other states since Kentucky is the only state that requires school-based decision making councils. He also noted that the school-based decision making council members are only required to have limited training and pre-requisite knowledge, which makes it very difficult to measure their effectiveness. Senator Westwood said this is another example of why the model utilizing school-based decision making councils needs to be reevaluated. The concept was formed and implemented during the Kentucky Education Reform Act.
Representative Moberly commented that most principals are taking the training, but cannot link training to performance. He also noted that there is a high compliance of staff taking training, but a high number of schools still do not perform well. Mr. White said it is hard to measure the effectiveness of the training. He said it would be very costly in both time and money to track this information across the state. Ms. Seiler said the issue could be addressed to increase the number of training hours that school-based decision making council members are required to have, but this may eliminate a pool of qualified, working parents, who may not be able to commit to 12 or more training hours in a year.
Representative Stevens made the motion to accept the OEA report, “Study of Leadership Training for Superintendents, Board Members, Principals and School-based Decision Making Councils”, and Representative Farmer seconded the motion. The report was adopted by voice vote. Senator Winters made the motion to accept the OEA report, “Compendium of State Education Rankings 2009”, and Representative Marzian seconded the motion. The report was adopted by voice vote.
Status Report on the Development of the Assessment and Accountability System
Dr. Terry Holliday, Commissioner of Education, KDE, reviewed the major components of Senate Bill 1, enacted by the 2009 General Assembly. He said a KDE survey allowed various stakeholder groups an opportunity to provide input on the public version of the content standards that were available to date. KDE anticipates the final version of the English and language arts and mathematics standards by June 2, 2010. He noted that national organizations are working to create common standards in science and social studies; the timeline is unclear and these may not be available to meet the Senate Bill 1 deadline of December 2010.
Senator Winters said he has been frustrated at the level of commitment from the Council of Chief State School Officers and the National Governor’s Association. He has communicated the message to them that Kentucky is missing critical timelines due to their agencies missing deadlines.
In response to a question from Senator Westwood, Dr. Holliday said the problems with developing content standards in science and social studies is that the subjects are not as straight forward as mathematics and English and language arts. He mentioned that the Council of Chief State School Officers is also trying to accommodate 40 other states.
Dr. Holliday said the program reviews of the arts and humanities, writing and practical living and career studies are being piloted in 48 schools in 34 districts. He said school-based decision making councils are developing new writing policies. Content networks will be guiding district leadership teams through the process of unpacking the standards. He said district leadership teams will facilitate this process with local school leadership teams and professional learning communities. The focus is on change in practice at the classroom level. The primary goal is for the content and examples of work from these networks to be used to populate the continuous instructional software tool (CIITS). He said the standards have been given to the teachers and will be reviewed in 2010-2011. Teachers will be assessed on the new standards beginning in 2011-2012. This exhausts all the professional development money and there is no additional money left to train teachers on the content pedagogy.
In response to a question from Senator Westwood about highly skilled educators, Dr. Holliday said every school district is being assumed as low achieving and will need assistance incorporating these new, rigorous standards. He said KDE wants to provide all teachers with the capacity to understand and utilize the new content standards. He said highly skilled educators will facilitate this process by assisting teachers in every school district because there is $56 million available to assist low performing schools.
Senator Westwood applauded KDE for putting the new standards into classroom language that teachers can understand. Ms. Felicia Cumings Smith, Associate Commissioner, KDE, responded that standards are written based on broad goals, and in a global format. She said it is KDE’s job to break down the broad goals into specific instructional learning targets that teachers can utilize to promote student learning in the classroom. Senator Westwood would like to see the standards written in language easier to understand, but still targeted to students on a global level.
Senator Winters discussed the vertical alignment of the core content standards. Dr. Holliday said there is a core content team in place at the state level that will be guiding and monitoring this work to ensure alignment as students move through grade levels. He said higher education will be included in the group as well to include that transition is aligned as well. Ms. Smith said the standards are written in grade level progressions.
Representative Stevens is concerned about the selection process of the people who will be responsible for teaching the training in the districts. Dr. Holliday said the issue of selecting strong leaders that the teachers have faith in has been stressed to the school superintendents. He said KDE has provided consistency in training and there are over 1,000 people that have been trained in the model and spread throughout the Commonwealth.
Dr. Holliday said the KDE and the Council on Postsecondary Education (CPE) have drafted a unified strategy to reduce college remediation rates by at least 50 percent by 2014 in comparison to the 2010 rate and to increase the college completion rates of students enrolled in one or more remedial classes by three percent annually from 2009 to 2014. He is serious about these goals and there has been wonderful participation from higher education as well as K-12 education. There are many great programs in place and he said the objectives will be met.
Dr. Holliday concluded by discussing upcoming timelines for Senate Bill 1 implementation and Race to the Top (RTTT) activities, as well as potential budget implications. He supports the vision and goals of Senate Bill 1, but said the lack of resources could prove to be very frustrating to teachers. He said it is very important to try to get RTTT funding as it is perfectly aligned with the goals of Senate Bill 1. He feels if the General Assembly will not support charter school legislation, Kentucky will not be competitive in being awarded RTTT funding.
In response to questions from Senator Winters, Dr. Holliday said there are some Title II funds available for teacher and principal effectiveness training. He said these areas will be negatively affected if RTTT dollars are not obtained. He also said that the request to exempt certain school districts from charter school legislation was at the request of one particular school district. Senator Winters does not understand the logic of exempting school districts from a policy that would be optional.
Report from the National Technical Panel on Assessment and Accountability
Dr. Ronald K. Hambleton and Mr. Jeffrey Nellhaus gave a report on the National Technical Panel on Assessment and Accountability (NTAPAA). Dr. Hambleton said the panel has agreed to certain test design features for Kentucky. He said a short version of one of the norm-referenced tests (NRT) (e.g., Terra Nova, SAT10, ITBS) will be selected by the KDE and used in Kentucky. Alignment of the content of the standardized achievement test with the curriculum at each grade level will be critical in test selection. If the length is excessive, for example, exceeding one hour, it may be that only sections of the NRT in a content area should be selected for administration.
Dr. Hambleton said the criterion-referenced test (CRT) items should be constructed by the contractor to fill in the gaps in the curriculum not covered by the NRT at a grade level, and administered in a separate section of the state’s annual test. He said the NRT will be used to provide students and their parents with percentile scores (or one of the other normative scores). The NRT items that match up to the curriculum at a grade level should be used along with the CRT items to provide information in relation to the learning outcomes in the curriculum. This information will be reported at the student level, and these scores and averages of scores across classes, schools, and districts, have many other uses as well.
Dr. Hambleton said vertical scaling of tests across grade levels in reading and mathematics is not something that is popular with the panel because of the variations in test content from grade level to grade level. The panel believes that a vertical scaling is not needed to assess growth from year to year as other growth models are available to the state. At the same time, vertical alignment of curricula will be important, and this can be expected with the common core curriculum. He said consideration of the vertical scale that the NRT’s will provide will be considered at another time. The major limitation is that only a portion of the content of interest to the state is included in the NRT vertical scale.
Dr. Hambleton said the state test in a subject at a grade level might consist of two sections, separately timed. He said the NRT section should be precisely timed; the other section made up of the CRT items could be more loosely timed, with extra time allowed for most students and hopefully all students, to finish. In 2011, matrix blocks would be used to field test new items for the 2012 assessments. The field testing would be carried out in the spring of 2011. If possible, the field testing should be approached as if it were part of the 2011 operational testing. He said beginning in 2012, matrix blocks in the CRT portion of the assessment would be used to link forms of the CRT’s from year to year, and to calibrate new test items. The test contractor would be responsible for insuring that the NRT forms from year to year are equated correctly.
Dr. Hambleton said every effort should be made to have the contractor make available three or four parallel forms in the NRT section, so that the same items would not be used from year to year. They should not be sold in any form in the state during the length of the contract. He said the goal is to minimize score inflation due to item exposure.
Dr. Hambleton said three hours would be the maximum time available for testing in reading and mathematics; a maximum of one hour for the NRT section, and one hour or more for the CRT section, and then up to an extra hour for students to finish the CRT section. If a new CRT form cannot be produced each year because of cost or any other reason, the panel recommends that several CRT forms be constructed with items from the 2011 field test and these forms could be rotated in some appropriate design from year to year.
Dr. Hambleton said the state will want to evaluate the alignment of the test specifications and the tests with the curricula. The panel recommends that this alignment of tests, test specifications, and curricula be carried out by a third party. Most certainly, the test contractor should not be responsible for these analyses. He said a completely objective analysis is needed and unfortunately the test contractor has a major stake in the outcome of this study.
Mr. Nellhaus gave a PowerPoint presentation on academic growth in Kentucky. Kentucky’s tests currently show how each student is achieving relative to state standards. Growth measures change in an individual student’s performance over time. He said uses of growth data can be for: reconceptualizing performance; identifying strengths and weaknesses in student performance beyond traditional achievement data; targeting assistance; conducting program evaluations; and assisting in making accountability decisions, such as school and district, and teacher evaluations.
Mr. Nellhaus said one way to measure growth is through student growth percentiles. Each student’s rate of change is compared to other students with a similar test score history. He said the rate of change is expressed as a percentile, relative to a student’s academic peers. He also showed examples of median student growth percentiles and growth distribution charts which are located in the meeting folder in the Legislative Research Commission library.
Senator Westwood expressed his appreciation for all the hard work of the NTAPAA members. He announced the next meeting is tentatively scheduled for Tuesday, June 15, 2010.
With no further business before the committee, the meeting adjourned at 3:07 p.m.