Call to Order and Roll Call
The meeting of the Education Assessment and Accountability Review Subcommittee was held on Wednesday, December 10, 2014, at 10:00 AM, in Room 154 of the Capitol Annex. Representative Rita Smart, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Mike Wilson, Co-Chair; Representative Rita Smart, Co-Chair; Senator David P. Givens; Representatives Tim Couch, Joni L. Jenkins, and Mary Lou Marzian.
Legislative Guests: Representatives Derrick Graham and Arnold Ray Simpson.
Guests: David Wickersham, Kelly Foster, and Kevin Brown, Kentucky Department of Education; Sue Cain, Council on Postsecondary Education; and Clyde Caudill, Kentucky Association of School Administrators.
LRC Staff: Janet Stevens, Jo Carole Ellis, and Lisa Moore.
Approval of Administrative Regulations
Kevin Brown, Associate Commissioner and General Counsel, Kentucky Department of Education (KDE), said 703 KAR 5:260 is a new regulation and 703 KAR 5:122 will repeal 703 KAR 5:120 and 703 KAR 5:180.
Donna Little, Committee Staff Administrator, explained the suggested amendments to 703 KAR 5:260. The summary of the suggested amendments are located in the meeting folders in the Legislative Research Commission (LRC) library.
Representative Jenkins moved to accept the amendments to 703 KAR 5:260, and Representative Marzian seconded the motion. The motion carried by voice vote.
Kelly Foster, Associate Commissioner, Office of Next Generation Schools and Districts, KDE, said the purpose of combining the two regulations was to have all the information regarding priority schools in one location. The regulation will allow KDE to work with school leadership in low-performing schools to assist in the implementation of intervention options in priority schools and districts to help improve student achievement.
Dave Wickersham, Staff Attorney, Office of Guiding Support Services, said the Elementary and Secondary Act (ESEA) flexibility waiver submission requires Kentucky to develop a process to identify and serve priority schools and is outlined in 703 KAR 5:120, which KDE is seeking to repeal. KDE is also seeking to repeal 703 KAR 5:180, which outlines the intervention system. Both administrative regulations are several years old, and repealing them would eliminate some overlap that can confuse the affected school districts.
Responding to a question from Representative Marzian regarding the new administrative regulation affecting the agreement between the Jefferson County Teacher Association (JCTA) and the Jefferson County Public Schools (JCPS) priority schools, Mr. Wickersham said he does not think the changes in the regulation affect the agreement. There was a conflict with the collective bargaining agreement, and the parties involved must continue to negotiate pursuant to the terms of their agreement and agree to waive or modify any part of that agreement that conflicts with statute, or to select an intervention option that does not violate the agreement. Mr. Brown said JCTA and JCPS have negotiated, and the new regulation will not affect interactions.
Responding to a question from Senator Givens, Ms. Foster said the administrative regulation affects a priority school that has been in the bottom five percent for three years in a row and has not met its annual measurable objective (AMO) for three years. Mr. Wickersham explained that the two regulations being repealed are outdated. The ESEA flexibility waiver requires KDE to develop a priority school process outlined in KRS 703 KAR 5:120 and 703 KAR 5:180, and combine relevant parts into one new regulation.
Representative Marzian moved to accept 703 KAR 5:260, and Representative Jenkins seconded the motion. The motion carried by voice vote.
Senator Givens moved to accept 703 KAR 5:122, and Representative Jenkins seconded the motion. The motion carried by voice vote.
Office of Education Accountability Report: A Look Inside Kentucky’s College and Career Readiness Data
Karen Timmell-Hatzell, Acting Director, Office of Education Accountability (OEA), introduced Deborah Nelson, Research Analyst (OEA), and Gerald Hoppman, Director of Research. Dr. Nelson said beginning in 2012, Kentucky’s Unbridled Learning accountability system included a measure of graduates’ readiness for college or a career (CCR). This measure reflects policymakers’ concerns that students graduate from high school with the skills necessary to succeed in postsecondary education or the work force. Since the measure was introduced, CCR rates have climbed steeply, from 47 percent in 2012 to 62 percent in 2014.
Dr. Nelson said college or career readiness rates are generally reported as a single percentage but comprise three different components: college ready; college and career ready; and career ready. Within each component, KDE uses a variety of indicators to determine graduates’ readiness. The report analyzes the components and indicators that make up CCR, looking at how each has changed over time and varies among schools and different types of students.
Dr. Nelson said to be considered ready for college, students can meet college-ready benchmarks set by the Kentucky Council on Postsecondary Education (CPE) in English, math, and reading on one of three tests: the ACT college readiness test or two college placement tests, the ACT Compass (Compass), or Kentucky Online Testing System (KYOTE). The report looks at percentages of graduates who are college-ready by different measures and offers some preliminary data on the college outcomes for students who are college ready by different measures.
Dr. Nelson said the reports use high school graduation data from KDE and the Kentucky Center for Education and Workforce Statistics (KCEWS). KCEWS collects and links data from KDE, CPE, and the Education and Workforce and Development Cabinet, among other agencies.
Dr. Nelson said KCEWS, which has also been called the “P-20 data system,” compiles data that allows researchers to track outcomes from elementary and secondary students as they move into postsecondary education and the workforce. In large part because of the capacity of the data system, Kentucky ranks high among states by the Data Quality Campaign for its collection and use of education data.
Dr. Nelson said KCEWS data available for the report allows tracking of graduates into Kentucky colleges and universities but does not provide data on graduates who enrolled in out-of-state institutions, joined the workforce, or enlisted in the military.
Dr. Nelson said since the CCR measure was introduced in 2012, readiness rates have climbed steadily, from 47 percent of graduates in 2012 to 62 percent of graduates in 2014. Prior to 2012, the ACT was the primary measure used to determine graduates’ college readiness. Gains since 2012 have come largely from students meeting college-ready benchmarks on the Compass and KYOTE tests or students meeting career-ready indicators. The percentage of graduates meeting readiness benchmarks on ACT tests has also increased but not as steeply as graduates identified as CCR by other indicators.
Dr. Nelson said the percentage of graduates who enroll in Kentucky colleges and universities and require remedial coursework in English, math, or reading has dropped substantially. In 2011, 54 percent of enrolled prior-year graduates required remedial coursework in at least one subject compared to 38 percent in 2013. During the same time period, the percentage of graduates enrolled in fall or spring has remained at about 56 percent for each year.
Responding to a question from Representative Simpson about whether the 56 percent includes private and for-profit university students, Dr. Nelson said the percentage includes private universities and community colleges. She will research whether for-profit universities and employees who are trained in-house are included.
Responding to a question from Senator Wilson regarding lab work being included as a co-requisite for remediation, Dr. Nelson said the data includes the percentage of students based on their status when they graduated who would have been required to have taken a class that was considered remedial in some way. It does not count course data of students enrolled and dividing them in different groups.
Dr. Nelson said the percentage of total CCR rates comprised of different components and indicators varies among schools and student groups. For example, schools vary in the percentage of students who are college ready by meeting benchmarks exclusively on ACT tests versus students meeting benchmarks on a combination of ACT, Compass, or KYOTE tests. In some schools, most of the students who are deemed college-ready meet CPE benchmarks exclusively on ACT tests. In other schools, less than 50 percent of college-ready students do so. The proportion of college-ready students who meet benchmarks on ACT tests is much lower in higher-poverty schools than in lower-poverty schools.
Responding to questions from Representative Graham, Dr. Nelson said to track the various groups of students over time, the Kentucky Student Information System (KSIS) would be needed. CPE has all the enrollment data but does not know which students were college-ready based on ACT benchmarks, Compass scores, or KYOTE scores. KDE data would have to be combined with data from the CPE to differentiate between the groups.
Responding to Representative Graham regarding the differences in security of administering the ACT, Compass, and KYOTE tests, Dr. Nelson said the ACT test can only be administered at certain times and at predetermined locations to a specific group of students, while the Compass and KYOTE tests can be administered to a single student at any time. Given the many different times and locations in which Compass and KYOTE tests are given, KDE must rely primarily on reports submitted by individuals to identify instances of inappropriate test administration.
Dr. Nelson said instances of inappropriate test administration might also be identified based on identification of unusual patterns in CCR test data or on discrepancies between a student’s performance on the ACT tests and on other CCR tests. KDE has a contract with Caveon, a test security company, to review issues of security in Kentucky’s testing programs, procedures, materials, and policies.
Dr. Nelson said the percentage of students who are deemed career ready has more than doubled since the measure was introduced, increasing from 8 percent in 2012, to 18 percent in 2014. The percentage of graduates reported by KDE as career ready would be even higher (26 percent in 2014) if the department reported all of the students deemed college and career ready as career ready. Because of inconsistencies in the criteria associated with these measures, some college and career ready students are not currently counted as career ready.
Dr. Nelson said the percentage of graduates who were college ready increased greatly between 2010 and 2012 but the percentage of graduates who enrolled in Kentucky colleges or universities the following year remained flat.
Dr. Nelson said college-ready students enroll in college and earn higher grades than do students who are not college-ready, but enrollment rates and grades vary for students deemed college-ready by different measures. Enrollment and persistence rates and GPAs are higher for students who meet all benchmarks on the ACT than for students who meet benchmarks on a combination of ACT, Compass, and KYOTE tests or on Compass and KYOTE tests alone.
Responding to a question by Senator Givens regarding the enrollment rate percentages in the gender differences not equaling 100 percent, Dr. Nelson said it is the percentage of all graduates.
Responding to a question from Senator Wilson, Dr. Nelson said GPAs from high school were not used as an indicator of college readiness for this study. However, she has seen research suggesting that GPAs may be as indicative as testing scores to determine student college readiness.
Responding to a question from Representative Jenkins regarding college-ready benchmarks on the ACT versus Compass and KYOTE tests based on students’ eligibility for free or reduced-priced lunch, Dr. Nelson said the percentage of graduates who met CPE college readiness benchmarks was substantially lower for students who were eligible for the free or reduced-priced lunch program (41 percent) than for students who were not eligible (67 percent). Of college ready students, the proportion meeting benchmarks on all three ACT tests was lower for students eligible for the program (about one half) than it was for those not eligible (about three fourths). There is discussion about whether KDE should pay for students to retake the ACT after completing remediation classes. Currently, lower income students can take the Compass or KYOTE tests at no charge but can only take the ACT once.
Responding to questions from Senator Givens regarding the differences in GPAs of students who meet ACT benchmarks compared to students meeting college-ready benchmarks on the Compass and KYOTE, Dr. Nelson said students who meet Compass and KYOTE college-ready benchmarks have a 21 percent chance of earning a 3.0 GPA or higher versus college-ready ACT takers who have a 57 percent change of earning a 3.0 GPA or higher. The positive aspect is that Compass and KYOTE allow these students to be deemed college-ready and not have to complete remedial classes.
Representative Graham said it is encouraging that students are proving that they can perform if given the opportunity. Compass and KYOTE are saving money for families by not making students take remedial courses. It also saves universities money by not having to hire instructors to teach remedial classes.
Dr. Nelson said KYOTE and Compass allow students to take credit-bearing courses without remedial classes, which is a significant cost savings to the student. The validity of Compass and KYOTE is not questioned. In addition to the cost savings, students are making good grades and graduating earlier than if they had to take remedial courses.
Responding to a question from Representative Marzian, Dr. Nelson said the study did not compare how high school GPA would equate to the ACT, Compass, and KYOTE tests in terms of predicting college outcomes. The KSIS data system would allow this comparison, and the committee can request the study for the future. Representative Marzian commented that some universities are no longer requiring ACT or SAT scores for college admission.
Responding to a question from Senator Givens, Dr. Nelson said the percentage of graduates considered college-ready was not reported at the state level prior to 2012 by KDE. Prior to 2012, students were taking Compass and KYOTE but were not included in the accountability system. Some students are college-ready and others are permitted to take a credit-bearing course without remediation. For this reason, students should be separated for comparison purposes and not all grouped together.
Dr. Nelson said the cause of differences among schools in college-ready tests is unknown. In some schools the percentage of students who are college-ready as graduates is two or three times as great as the percentage of students who were college-ready when they took the ACT in the 11th grade. These jumps might reflect intensive effort by high schools to address students’ academic deficiencies and ensure that they acquire the skills needed to succeed in college.
Dr. Nelson said there is evidence that the validity of some students’ scores on ACT and Compass math tests may have been influenced by the use of “Zoom Math” calculator software that can solve algebraic equations. In 2013, postsecondary math educators in the Commonwealth raised concerns that, by using this software, it was possible for students to meet Kentucky benchmarks on ACT and Compass tests without understanding the content tested. The “Zoom Math” software was not allowed on KYOTE exams.
Dr. Nelson said KDE investigated and found evidence to support the concern that the software could lead to artificial inflation of math scores. KDE and ACT are removing the use of the calculator software beginning in 2015.
Dr. Nelson reported the OEA’s recommendations to the study. A complete listing of the recommendations is located in the meeting folder in the LRC library.
Responding to a question from Representative Graham, Dr. Nelson said large differences in graduates’ readiness rates on ACT tests and their readiness rates on Compass and KYOTE tests may reflect schools’ intensive efforts to address students’ academic deficiencies. It is possible that the discrepancy between ACT scores and Compass or KYOTE scores indicates inappropriate test administration practices in some schools.
Dr. Nelson said for students to be college and career ready they must meet the technical requirements of career readiness and pass the college readiness test. In Kentucky’s accountability system, districts and schools are awarded extra points for students who are deemed college and career ready; these students meet the technical criteria for being considered career ready and also pass college readiness exams. College and career ready students are worth one and one-half points when CCR percentages are calculated in the accountability system. In 2014, the difference in the percentage of students CCR with and without the bonus calculation was less than ten percentage points in most high schools. Some educators have raised concerns over the bonus points.
Dr. Nelson said there is a discrepancy in criteria and some career ready students are not being reported. This is due to an inconsistency between KDE’s definition of college and career readiness and its definition of career readiness results in underreporting of the total number of students who are career ready. This is an issue that affects reporting only and there are no negative consequences for schools or students. Schools receive additional bonus points for college and career ready graduates regardless of whether those graduates are reported as career-ready.
Dr. Nelson said KDE should not use the CCR measures as the sole or primary indicator when reporting progress of student outcomes over time or evaluating the impact of particular programs or policies. College or career readiness rates should not be used in isolation to compare student outcomes among districts and schools. Given the change over time in measures used to determine CCR, it is important to interpret changes in CCR rates with caution. Caution should also be used in drawing conclusions based on the CCR measure alone about the effectiveness of programs implemented during this time period or about changes in student learning.
Representative Marzian requested a study comparing the high school GPA and the ACT, Compass, and KYOTE test scores to determine college readiness. She also would like data on how many colleges and universities are deleting the requirement for a standardized college readiness testing score for admission. Dr. Nelson said the data would be interesting.
Responding to a question from Senator Wilson regarding KDE’s response to the OEA study, Tracy Goff Herman, KDE legislative liaison, said KDE will prepare a written response to the study and distribute it to members.
Representative Graham thanked OEA for a useful report and said the information should help address the ongoing commitment to closing the achievement gaps.
Senator Givens would like to continue the dialogue between KDE and Dr. Nelson at the next EAARS meeting.
Senator Wilson moved to approve Kentucky’s college and career readiness data report, and Representative Jenkins seconded the motion. The motion carried by voice vote.
Office of Education Accountability’s Proposed 2015 Study Agenda
Ms. Timmel-Hatzell explained the proposed 2015 studies, which includes reports on student safety, the annual district data profiles, recess and physical education in the K-5 schools, biennial compendium on state rankings, and a primer for Kentucky independent school districts.
Senator Givens moved to approve the study agenda, and Senator Wilson seconded the motion. The motion carried by voice vote.
Other Business
At the request of Representative Smart and Senator Wilson, staff will work on getting the EAARS subcommittee a regular meeting date.
Adjournment
With no further business, the meeting adjourned at 11:45 AM.