Call to Order and Roll Call
The meeting of the Education Assessment and Accountability Review Subcommittee (EAARS) was held on Tuesday, September 13, 2011, at 11:30 AM, in Room 131 of the Capitol Annex. Representative Ted Edmonds, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Representative Ted Edmonds, Co-Chair; Senators Vernie McGaha and Ken Winters; and Representative Linda Belcher.
Legislative Guest: Representative Keith Hall.
Guests: Jonathan Lowe, Jefferson County Public Schools; Alicia Vinson, Fayette County Public Schools; Robert Duncan and Jacque VanHouten, Kentucky Department of Education (KDE); Phil Shepherd, KCAE; Jim Thompson, Education and Workforce Development Cabinet; Cindy Heine, Prichard Committee; Sue Cain, Council on Postsecondary Education; and Chris Lilly, Legislative Research Commission.
LRC Staff: Sandy Deaton, Ken Warlick, Ben Boggs, Jo Carole Ellis, and Lisa Moore.
Adopt Minutes of June 13, 2011 Meeting
Minutes were not approved due to lack of a quorum.
Kentucky’s Assessment and Accountability System
Dr. Pat Roschewski, Member, National Technical Panel on Assessment and Accountability (NTAPPA), reported that NTAPPA met twice in the past six months in the months of March and May. The two topics discussed in March included program reviews and security.
Dr. Roschewski noted the program reviews included a series of self reports, and documents had not been developed at the time of the March NTAPPA meeting. School districts were asked to provide feedback on programs that were written into Senate Bill (SB) 1. It was recommended that the Kentucky Department of Education (KDE) develop a specific rubric to give guidance to school districts and serve as a scoring guide to ensure specific program criteria were met. Discussions included the importance in training the school districts on how to use the rubric. KDE took NTAPPA’s suggestions seriously and completed the rubric to meet the guidelines of specificity. A pilot of the program reviews was conducted and will be field tested before full implementation.
Dr. Roschewski said NTAPPA asked KDE to provide an overall plan for security. NTAPPA reviewed the performance outlier report in March, but this report should be only one piece in an overall security system for the state. She said other things to include would be reviewing reports and conducting interviews.
Dr. Roschewski said the Webinar in May focused on three topics. Mr. Ken Draut, Associate Commissioner, Office of Assessment and Accountability, KDE, gave an update on the new contract. She said it is important for NTAPPA to be informed on follow-up information from preceding meetings to ensure that its advice is being followed or to be advised that a new direction is being pursued.
Dr. Roschewski said the ACT and end-of-course (EOC) assessment issue was discussed because KDE is trying to determine how to correlate the ACT end-of-course scores to a cut score and grade. School districts are aware that student motivation is needed in an accountability system for optimal results. The meeting concluded with the belief that a traditional standard setting process should occur but be enhanced by a few additional steps that will translate into grades. Grades are one of the most arbitrary things that can happen inside school districts. It was recommended that the standard setting panelists have a clear understanding of the definitions of proficient or advanced to ensure the terms mean the same thing to everyone. Panelists should be well informed on what other states are doing as well.
Dr. Roschewski said the standard setting process for the whole accountability system should incorporate all the components to determine a final cut score. The entire system should be easily understood by the public to determine a school or district’s effectiveness. NTAPPA recommended incorporating a broader set of panelists to represent community and higher education interests, and recommended making proficiency level descriptors very clear and expectations defined. She said it should be easy to identify a school that is labeled “distinguished.” NTAPPA strongly encouraged KDE to provide models of the schools to the panelists.
Dr. Roschewski said training and clarity of the rubric are the keys to having a solid accountability system. Teachers, parents, and administrators all need to understand the rubric and ask panelists if they are implementing suggestions received in training.
In response to questions from Senator Winters, Dr. Roschewski said she is the only member of NTAPPA who has seen the rubric, and she has not seen any of the results. She said there is less opportunity for upward bias if the rubric is clear and the training is extensive. There is great value in auditing, and Nebraska audited every district.
Dr. Roschewski said NTAPAA did not discuss the inclusion of the teacher and principal effectiveness in the overall system. This is another piece that will take time to develop.
Responding to questions from Senator McGaha, Dr. Roschewski said she reviewed administrative regulations 703 KAR 5:200, 703 KAR 5:220, and 703 KAR 5:230, but other members of NTAPAA have not reviewed the regulations. She has not provided feedback to KDE on the administrative regulations.
Dr. Roschewski said policy weights are an important discussion and opinions vary. There is no “right” system for determining the percentages, but balances need to be kept in check. The state board in Nebraska approved a system with a good balance of points that schools believe to be fair and then evaluate each year to ensure it is proportioned appropriately. The most difficult part is distributing the points among the low-performing schools that show great improvement and trying not to punish the excellent schools because they are at the top of the scale. Trying to find an equitable way to distribute the points is a huge and complex task. It was noted that one NTAPPA member mailed in written suggestions that were reviewed by all members of NTAPPA about specific ways to weight points. She said there was no verbal discussion among the members, but their thoughts were distributed to LRC. Senator McGaha said he is concerned that the high-performing schools will not be able to accumulate any points for growth.
Responding to questions from Senator Winters, Dr. Roschewski said adding the EOC score to the student’s transcript is a clerical issue and is a reasonable medium because it is a state cut score. She suggested reviewing the numbers in the first year before adding a requirement to place the score on the transcript.
Mr. Kevin Brown, Associate Commissioner and General Counsel, KDE, said Dr. Holliday is out of the country but will be at the Education Assessment and Accountability meeting in October. He said the administrative regulation 703 KAR 5:200 is deferred until October, and two other administrative regulations will follow.
Mr. Draut said Kentucky, like many other states, is developing a model to grade schools on a variety of items, and not just a single test score. Next-Generation Learners, Next-Generation Instructional Programs and Support, and Next-Generation Professionals are the three areas that will make up the overall accountability score. The Next-Generation Professionals, which includes the percent of effective teachers and leaders, will take some time to develop.
Mr. Draut said the Next-Generation Learners goes into effect this year and the test scores will be included in the accountability system. The five areas included in this category are achievement, gap, growth, college and career readiness, and graduation rate. The performance measures vary by category, and their respective weights differ in the overall accountability system between elementary, middle, and high school. There is an example of the overall score reporting for Next-Generation Learners for a Kentucky high school provided in the meeting materials located in the LRC library.
In response to a question from Representative Belcher, Mr. Draut said scores will be released this year in late August or early September because the standard setting process will take some time to complete. Scores will be released on an annual summer basis schedule beginning in two years.
Responding to questions from Senator McGaha, Mr. Draut said the Kentucky Board of Education (KBE) agreed to give schools a one-half bonus point for any student who is distinguished. However, it is offset by deducting a point for novice students. The rationale behind this was trying to influence behavior and give schools an incentive to push proficient students into the distinguished category.
Mr. Brown said the KDE had received some suggested changes from LRC staff, and they are included in a lengthy staff amendment. KDE is willing to define more terms in the regulation using the amendment process before the hearing in October.
Mr. Draut said KDE will incorporate suggestions from NTAPPA as it begins the standard setting process with the KBE in December. Senator McGaha is concerned that EAARS members are being asked to approve an administrative regulation based on a standard setting system that has not yet been developed.
In response to a question from Senator Winters, Mr. Draut explained that the chart for the weights for Next-Generation Learners in the meeting handout has been included in the administrative regulation to help explain the weights as suggested by LRC staff. Mr. Brown clarified that students are counted in the formula reported by schools and districts if they have been enrolled in the school for 100 or more days.
In response to questions from Senator McGaha, Mr. Draut said he thought it was intended for NTAPAA to review the finished administrative regulation after completion, but KDE does not communicate directly with NTAPPA members. KDE has set up an internal process that utilizes LRC staff to correspond with NTAPPA.
Mr. Brown responded that SB 1 did not include program reviews for foreign languages or the primary program. However, SB 1 did not have limiting language in it that would prevent the KBE to fulfill its other statutory roles over the common schools and the assessment system by adding additional requirements to the assessment and accountability system. Senator McGaha said SB 1 did not give permission for the KBE to add program reviews that were not listed in the bill.
In response to a question from Senator Winters, Mr. Brown said SB 1 amended several assessment and accountability statutes. He noted it did not amend the board’s authority of its broad oversight to strengthen the accountability system without being in contradiction to SB 1.
Ms. Felicia Smith, Associate Commissioner, Office of Next-Generation Learners, KDE, discussed implementation of the program reviews defined in SB 1. An accountability model should influence behavior at the local level. All the program reviews were based on three areas identified in SB 1, including arts and humanities, practical living and career studies, and writing. NTAPPA assisted KDE to address concerns regarding subjectivity and the following actions are occurring: establishment of a rubric; one year validation of content; field review; audit process; and establishing common evidences. KDE will partner with districts and schools, the School Curriculum Assessment and Accountability Council (SCAAC), and NTAPPA to address the primary issues related to subjectivity.
Ms. Smith said the addition of the program reviews for the K-3 primary program and world languages has generated much discussion. KBE felt the three program reviews mentioned in SB 1 were identified because it was assumed that paper and pencil was not the best way to assess the content areas of arts and humanities, practical living and career studies, and writing. KDE also believes it captures a number of teachers in the non-assessed areas that are mentioned in the Next-Generation Learner component. KDE and KBE feel it is important to capture how well implementation at the local level is occurring in regards to access, opportunities, and instructional approach; however, they feel it is just as important to make sure students in the K-3 program are college and career ready, which is consistent with the intent of SB 1.
Ms. Smith said KBE made the recommendation to add the program review for the K-3 primary program based on rationale that is believed to measure teacher effectiveness for a larger group of teachers. This ensures all schools are held accountable for student performance. KDE received recommendations from the Early Childhood Development and Education Task Force that improves K-3 quality level and believes adding the program review will help to guide schools in implementing their recommendations. It could assist Kentucky in being awarded a Race to the Top Early Learning Challenge grant that has criteria for K-3 programs adding value to a college and career readiness agenda.
Ms. Smith said the rationale for the addition of a world language program review is to ensure that Kentucky is competitive globally and places an emphasis on access and opportunity for children to have world language experiences. Enhanced language proficiency increases academic achievement by increasing cognitive development and abilities and reading abilities and skills. The study of world language is required for college admissions and keeps Kentucky in alignment with a pre-college curriculum in K-12. World language skills increase job potential and ensure Kentucky’s competitive edge in the global market.
Responding to questions from Representative Belcher, Ms. Smith said a K-2 diagnostic measure as outlined in SB 1 will be used to measure K-3 programs. It will be administered at the local level but will model the standards and demonstrators of other program reviews. The intent is to make the program reviews as consistent as possible to relieve the burden at the local level.
Ms. Smith said adding the program reviews for K-3 and world languages may add a burden to elementary teachers, but it is the right thing to do for the children. Representative Belcher said she is supportive of assessing children as they leave the primary program, but is not in support of adding programs that are not funded. She said it is valuable for children to learn foreign languages early in life, but schools need support and money when new programs are mandated. Ms. Smith said the SB 1 and KDE budget request will include the financial aspects of funding the new program reviews.
In response to questions from Representative Belcher, Mr. Brown said SB 1 was not funded either, and KDE has done the best it can with existing appropriations and will do the same implementing the new program reviews. Representative Belcher said this concerns her because this is a more specific mandate.
Ms. Smith said the program reviews will allow schools flexibility in exposing children to world languages outside of the traditional teacher and student ratio instruction format. The program reviews will be manageable and offer students innovative opportunities to learn world languages such as after school programs. She noted schools would receive consequences or sanctions if they do not meet requirements in the accountability system. Senator McGaha said some research shows foreign language is learned at a faster rate by students who have completed puberty instead of in the elementary years.
Dr. Larry Stinson, Associate Commissioner, Office of Next-Generation Schools and Districts, KDE, discussed school and district accountability recognition, support, and consequences. One overall accountability score will be comprised from a mixture of 70 percent Next-Generation Learners, 20 percent Next-Generation Instructional Programs and Support, and 10 percent Next-Generation Professionals. The original recommendation was a 60/20/20 split, but shareholders such as the Local Superintendents Advisory Board, SCAAC, and various cooperative meetings provided feedback that emphasis should be placed on the learner. It was decided among the stakeholder groups that the 70/20/10 split was optimal and KBE agreed with the field. This is for recognition purposes and does not change the classifications that schools receive for the three subparts.
Dr. Stinson said the standard setting process will establish the goals and cut scores or point totals that determine school and district placement in one of four classifications (Distinguished, Proficient, Needs Improvement or Persistently Low Achieving). The standard setting process will occur after data are available from the first administration of the new state required assessments outlined in Senate Bill 1. The top five percent of the distinguished classification is recognized as high distinction. The bottom five percent or five, whichever is greater, is persistently low-achieving and support is outlined by KRS 160.346.
Responding to a question from Senator Winters and McGaha, Mr. Draut said SB 1 requires students to be tested in science once in elementary and high school and does not accumulate the points that reading and mathematics accumulate because students are tested each year in those subjects. Senator McGaha responded that SB 1 did not ask for the program review in world language, and it is equivalent to the same points as science. Mr. Draut said science is valued more in elementary and middle school and is only equivalent with the program review in high school.
In response to a question from Senator McGaha, Dr. Stinson said the Next-Generation Professionals will be a part of the overall accountability score beginning in the 2013-2014 school year. He said the points are redistributed between the other two categories during the interim.
Senator McGaha asked Ms. Donna Little, Administrative Regulations Committee, if she had reviewed administrative regulation 703 KAR 5:200. Ms. Little said she skimmed the regulation after receiving a phone call to be at the EAARS meeting, but she has not read SB 1 to know what is authorized by statute. She noted that KRS Chapter 13A requires that administrative regulations be specific and establish all the requirements that are necessary unless the requirements are already established in statute. Senator McGaha said that, if the administrative regulations are not drafted properly, it could open up the possibility of legal challenges. He recommended that the EAARS administrative regulations be reviewed by the Administrative Regulations Subcommittee routinely in the future.
Mr. Brown explained that a standard setting process cannot be set up without a first year of results and a first year of results cannot occur without having the administrative regulation and assessment system in place. Mr. Draut said the first year test scores need to be reviewed before determining the final cut scores to serve as a baseline.
Responding to a question from Senator McGaha, Mr. Brown said it is a legislative branch decision to decide if the EAARS administrative regulations should be reviewed by the Administrative Regulations Subcommittee. Ms. Little said KRS 13A.290 removed the review of EAARS regulations from the Administrative Regulations Subcommittee and would require statutory change; however, she said that she would review the regulations informally if EAARS staff had no objection. Chairman Edmonds agreed to let Ms. Little review the administrative regulation informally. Mr. Brown said the EAARS staff made several suggestions to KDE that improved the regulation.
Senator Winters asked KDE to strengthen the math and science components in the next version of the administrative regulation. Chairman Edmonds said no action would be taken on 703 KAR 5:200. He deferred the Office of Education Accountability’s report until the next EAARS meeting.
With no further business before the committee, the meeting adjourned at 1:35 p.m.