Call to Order and Roll Call
The meeting of the Education Assessment and Accountability Review Subcommittee was held on Tuesday, October 11, 2011, at 1:00 PM, in Room 131 of the Capitol Annex. Senator Jack Westwood, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Ted Edmonds, Co-Chair; Senators Vernie McGaha, Gerald A. Neal, and Ken Winters; Representatives Bill Farmer, and Mary Lou Marzian.
Legislative Guest: Representative Derrick Graham.
Guests: Keith White, Office of Education Accountability; Wayne Young, Kentucky Association of School Administrators; Jim Thompson, Education and Workforce Development Cabinet; Mary S. Buckley, Department of Juvenile Justice; and Sue Cain, Council on Postsecondary Education.
LRC Staff: Sandy Deaton, Ben Boggs, Jo Carole Ellis, Ken Warlick, and Lisa Moore.
Approval of Minutes
Representative Edmonds moved for approval of the minutes from the June 13, 2011, and the September 13, 2011, meetings and Senator McGaha seconded the motion. The minutes were approved by voice vote.
Kentucky’s No Child Left Behind and Kentucky Core Content Tests data results for 2010-2011
Terry Holliday, Commissioner, Kentucky Department of Education (KDE), reported the final year of results for the Kentucky Core Content Test (KCCT). He noted that KDE has asked for a waiver from the federal No Child Left Behind (NCLB) reporting requirements. In a waiver to be submitted on November 14, 2011, KDE will ask for flexibility based on Kentucky’s new accountability model outlined in Senate Bill 1.
Mr. Holliday said the data show little improvement in grades three through eight. Significant improvement was made in Kentucky’s high schools and he attributes this to House Bill 176, enacted in the 2010 regular session that defined persistently low achieving high schools and challenged them to improve. Funding is a big concern going forward for assisting the low performing high schools and this will be reflected in KDE’s budget request. Funds designated for the highly skilled educators were previously utilized but this money was eliminated in 2012.
Ken Draut, Associate Commissioner, Office of Assessment and Accountability, KDE, reported on the percent of students designated as proficient and distinguished by grade span and year on the KCCT in the areas of reading, mathematics, science, and social studies. The detailed charts are located in the meeting materials in the Legislative Research Committee (LRC) library. Mr. Draut said the 2011 NCLB results reported Kentucky meeting 13 of 25 target goals, which is equivalent to 52 percent. Since schools must make 100 percent of their target goals to qualify as making adequate yearly progress (AYP), Kentucky did not make its overall AYP benchmark.
In response to a question from Representative Farmer, Dr. Holliday said it is a national trend that student performance levels are the highest in elementary school and the lowest at high school levels. Dr. Holliday attributed this to elementary students trying to please their teachers and this does not occur as much at middle and high school levels. In the upper grades, students often perform better when student accountability is associated with the assessment. He said there are typically 25 percent fewer students in high school and some of those may have been high performers. Principals and teachers have communicated that the new accountability model should help address this issue.
Mr. Draut said there were 245 Title I schools receiving consequences in 2011 compared to 134 schools in 2010. It is ironic that as student scores improve, more Kentucky schools receive consequences. He noted that 122 Title I school districts received consequences compared to 93 in 2010. There were 168 non-Title I schools eligible for state assistance in 2010 and 206 in 2011.
Rhonda Sims, Director, Division of Support and Research, Office of Assessment and Accountability, KDE, said Kentucky students improved their mathematics scores on the Iowa Tests of Basic Skills from 2010 to 2011, but reading scores varied and decreased in third and fifth grades. However, Kentucky students are performing above the national average, which is 50 percent.
Ms. Sims said Kentucky is beginning to make the transition to the new accountability system and this requires looking at gaps in comparison to goals. Kentucky’s goal is for all students to reach 100 percent proficiency in reading and math. She said most reporting groups showed a small closing of the gap, or slight improvement, between students testing at proficient and distinguished in 2010 to 2011. However, there was no change in the scores of Asian students, students with limited English proficiency, or students with disabilities.
Ms. Sims said all students are required to take the ACT in grade 11. The number of students tested in 2011 increased substantially from 2008 (42,922 to 44,053 students). Students tested in 2011 scored higher in English, mathematics, reading, and science. The overall student composite score for the ACT increased from 18.3 in 2008 to 18.8 in 2011.
Ms. Sims said the ACT will be used in the new accountability program to measure college and career readiness. Students must meet the benchmarks on the ACT test determined by Kentucky’s Council on Postsecondary Education (CPE) to be college ready. Students will also determine their college readiness by meeting placement test benchmarks. The career readiness benchmarks are divided into academic and technical career ready. Students can meet the career ready academic portion through the Armed Services Vocational Aptitude Battery (ASVAB) or the ACT Work Keys, which includes applied math, locating information, and reading for information. The career technical requirements can be met through the Kentucky Occupational Skill Standards Assessment (KOSSA) assessment or industry certificates. She said 15,746 graduates are college and/or career ready, or 38 percent, which increased from 34 percent in 2010.
In response to a question from Representative Farmer, Dr. Holliday said students score higher on the ACT because it affects them directly, unlike the KCCT scores. He noted that KDE strongly suggests schools and districts use the ACT end-of-course test scores to count for 20 percent of the student’s final grade.
Responding to questions from Representative Graham, Dr. Holliday said the EXPLORE test is administered to students in the eighth grade as a predictor on how the students will perform on the ACT. It also helps guide their high school coursework to align with career goals. The PLAN is administered in the tenth grade as another check to ensure students are on target to become college ready. All students are required to take the ACT in eleventh grade. Kentucky has filed for a waiver to bypass annual yearly progress benchmarks required in the NCLB Title I program. He expects a decision on the waiver to arrive by mid-January based on the results of the 2011-2012 testing.
Mr. Draut said the end-of-course test can be taken when the student completes the course. KDE has the test prepared, and some students have already begun taking the exam during the testing window of October 1 through November 15, 2011.
In response to a question from Senator McGaha, Mr. Draut said KDE is working with the ACT on a weekly basis to reach an agreement on how Kentucky can get files of Kentucky public school students on Work Keys back from the ACT. He said there are issues with the ACT protecting the confidentiality of its adult students in the database.
Dr. Holliday said KDE has been transparent about these numbers and issues and will be prepared to include them in the accountability model in 2012. He and Dave Adkisson, Executive Director, Kentucky Chamber of Commerce, will visit some of the larger chamber of commerces later this year to encourage businesses to support schools in administering the Work Keys assessment to seniors. The Workforce Investment Board helped fund some of the testing last year. Senator McGaha said several schools in his district tested in the top ten in the state on the Work Keys assessment and were very disappointed the scores were not counted.
Deferral of Administrative Regulation
Senator McGaha moved to defer 703 KAR 5:200, and Representative Farmer seconded the motion. The motion was approved by voice vote.
Office of Education Accountability’s Reports
Keith White, Research Analyst, Office of Education Accountability (OEA), said Kentucky is in good shape with test security as it moves into the new accountability model. There are no widespread problems. Senate Bill 1 relies heavily on assessments, and Kentucky needs a system that ensures the information is reliable and valid. He recognized the National Technical Advisory Panel on Accountability and Assessment (NTAPPA) as being helpful in recommendations on test security, and said that one panel member had written a book on the subject.
Mr. White said standards are developed by trained professionals before testing begins. KDE partners with high quality education testing vendors. The tests and assessments are designed to align with standards and items that are field tested to establish reliability and validity. Alignment and validity studies are ongoing as required by KRS 158.6453(17).
Mr. White said district and building assessment coordinators and proctors receive training on administering tests and assessments, and on other processes related to test administration, submission, and report. Test administrators must be trained on and sign the assessment code of ethics. Education professionals, students, parents, and stakeholders may report suspected testing code violations to KDE’s testing allegation coordinator.
Mr. White said draft score reports are reviewed by KDE and schools to correct scoring and submission errors after testing. Final reports are issued to schools and the public.
Mr. White discussed some specific recommendations in the report related to behavior, data, and testing events. Recommendations 3.1 to 3.3 are located in the meeting materials located in the Legislative Research Commission (LRC) library.
Mr. White said testing allegation categories range from completing exam sections out of order to blatant cheating or inappropriate assistance/intervention from 1996-2009. There were 2,407 reported allegations to KDE; 461 score reductions by KDE; and since 2000, the Education Professional Standards Board (EPSB) opened 91 cases and closed 67 of those. Thirty-five closed cases resulted in dismissal, while the remaining 32 cases were actionable.
Mr. White said outliers can include data points that are atypical or unlikely. Outliers are a good first step and can utilize various statistical methods. He discussed follow-up processes and typical test security audit components. The costs of contracting with a leading United States test security firm can be $100,000 to $450,000 for basic to expanded services. Proactivity increases likelihood of cost reduction.
In response to questions from Senator Westwood, Mr. White said most states build in checkpoints for atypical data into the vendor contracts. Other states rely on a third party government agency, but it is not typically performed by KDE staff. Kentucky has not named an entity to provide this service.
Ms. Seiler said Kentucky’s test complaints in the past have been submitted to OEA and to EPSB. The testing center performed the investigation. The noted complaint was not detectable by an outlier process, and ACT determined that is was a teacher or administrator error. Teacher and administrator complaints are handled by EPSB, which is good and unique to Kentucky. EPSB conducts administrative hearings to review the evidence, and the board makes the decision as to who is responsible and the appropriate punishment.
Responding to Senator Westwood, Ms. Seiler said consequences should be based on evidence of the particular situation. She said the evidence required to prove cheating is almost nonexistent. For example, the testing rooms are not filmed. There is no documentary trail to lead specific test scores with certain erasures to individual teachers. These checkpoints have to be built into the system, and this takes time. Linking a teacher number to the testing environment would be ideal. Mr. White said that KDE is looking to include this type of analysis in the new testing agreement with vendors.
Representative Graham said teachers experience low morale at testing time because administrators are nervous disseminating information. He noted the extremes teachers go through to ensure safety measures are in place to prevent cheating in the classroom.
Responding to Representative Graham, Mr. White said cheaters will find a way to cheat regardless of whether the assessments are completed using paper and pencil or online, but this is small number of people. Most educators want to collect the most valid information possible.
In response to a question from Representative Farmer, Mr. White said historical or trend mechanisms need to be incorporated into the new testing model. He noted the baseline for checkpoints is based on the last two years of assessment. The baseline has been established for each test, size, and grade level, and there is an expected mean and outlier threshold that can change each year based on the changed scores.
Responding to Senator McGaha, Ms. Seiler said OEA staff would be talking with EAARS members individually to discuss study topics for 2012. She said one possible topic is addressing the security of student data when testing online. This would include student personal data and test score information.
Brenda Landy, Research Analyst, OEA, said the Kentucky Educational Excellence Scholarship (KEES) study examined the KEES trends relating to goals, specifically college access and achievement, and keeping students in-state for college. She gave an overview of the background of the KEES program and discussed future recommendations. The information can be found in the meeting materials located in the LRC library. Since 2000, KEES program growth has been steady, and $100 million was disbursed to about 70,000 recipients in 2010.
Ms. Landy noted the KEES’ mission has been the subject of controversy and confusion at times, in part because ensuring access is the only purpose explicitly mentioned in statute. Some academics and policy analysts argue that performance-based programs are far less efficient than need-based programs. One recommendation was that the General Assembly should consider adding explicit statements of these goals to the statute.
Ms. Landy said grading scales vary across school districts. Some policymakers believe that higher standards can equal smaller KEES awards. Others say students earn more in the long run because they are more prepared for college. Grade information in the system lacks standardization and documentation and is unusable in analysis, and there is no ongoing systematic process for ensuring accuracy of the grade point average (GPA) reported for KEES purposes. Specific recommendations were included for these issues in the study and can be found in the meeting materials in the LRC library.
Ms. Landy said many states were concerned about brain drain in the 1990s. This problem was not emphasized in Kentucky, but there was room for improvement. In 1998, 86 percent of freshmen enrolled in Kentucky postsecondary institutions. This improved slightly to 89 percent in 2008. She also noted that students who earned KEES were twice as likely to enroll in college. This drops off after two years and that may be attributed to students earning a two year degree and deciding college was not for them.
Ms. Landy said there are concerns with the “on track to graduate” statutory language and implementation of the program is proving difficult. One issue concerns a mismatch between the definition of “full-time” and the definition of “graduating on time.” She said another issue is that institutions lack sufficient data on transfer students. The specific recommendations regarding the “on track to graduate” provision are located in the meeting materials in the LRC library.
In response to questions from Senator Winters, Ms. Landy said “on track to graduate” definitions differ across public institutions and definitions on back-load credit hours are imprecise for transfer students. OEA was unable to obtain definitions used by independent and proprietary institutions. Public institutions need detailed guidance on determining whether students are on track to graduate.
Ms. Landy said KEES does not have an unmet need problem like the College Access Program or the Kentucky Tuition Grant program. She said there is a comparison with KEES and scholarship programs in other states located in Appendix “C” of the study. Ms. Seiler suggested bringing a group together from the Kentucky Higher Education Assistance Authority (KHEAA), KDE, and school districts to look at GPA and grading scale issues.
Responding to questions from Senator Westwood, Ms. Landy said students have five years to enroll in a Kentucky postsecondary institution for KEES eligibility. For each year students are out-of-state enrolled in non-eligible institutions, they lose a year of KEES eligibility. Most Kentucky postsecondary institutions do not include grades from a prior institution in the GPA calculations for KEES, with Eastern Kentucky University being the exception. Private school students are eligible to earn KEES, but home school students and GED recipients are eligible for the ACT award only.
In response to a question from Representative Farmer, Ms. Rebecca Gilpatrick, Director of Student Aid Services, KHEAA, said that students receive a year for year extension on KEES if they are in active military service. Students can appeal to KHEAA and provide discharge papers.
Responding to Representative Graham, Ms. Landy said students do not get credit for high school courses completed in middle school. She also noted that students who graduate high school in three years lose a year of GPA awards. Hopefully, an admissions officer would recognize this and make up the difference. Representative Graham does not think Kentucky institutions are considering this. Ms. Landy said surrounding states are matching KEES dollars to recruit students.
Senator Winters said his bill on early graduation that will be introduced in the 2012 session will compute the average of the student’s three high school years to count as a fourth year. He said this ensures students are not penalized from receiving KEES if they graduate a year early.
Adoption of OEA reports
Senator Westwood said the committee will postpone adopting the OEA reports to the next meeting due to lack of a quorum.
Other Business
Senator Westwood told committee members to submit suggested OEA topics to Co-Chair Edmonds, himself, or Marcia Seiler.
Adjournment
With no further business before the committee, the meeting adjourned at 3:07 p.m.