The third meeting of the Education Assessment and Accountability Review Subcommittee was held on Monday, June 6, 2005, at 10:00 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly Jr, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Jack Westwood, Co-Chair; Representative Harry Moberly Jr, Co-Chair; Senators Dan Kelly, and Ken Winters; Representatives Jon Draud, Mary Lou Marzian, and Frank Rasche.
Guests: Helen Mountjoy and Hilma Prather, Kentucky Board of Education; Bob Tarvin, School Facilities Construction Commission; Gene Kirchner, Walton-Verona Schools; Molly Williamson; State Journal; Billie Walker, Riverside Publishing; Clyde Caudill, Jefferson County Public Schools; James Cibulka, University of Kentucky College of Education; Wayne Young, Kentucky Association of School Administrators; and Fred Bost, Education Testing System.
LRC Staff: Sandy Deaton, Audrey Carr, Jonathan Lowe, and Lisa Moore.
Representative Rasche made the motion to approve the minutes of the May 20, 2005 meeting, and Senator Winters seconded the motion. The motion was approved by voice vote.
Representative Moberly said the subcommittee would be hearing about the cost of the Commonwealth Accountability Testing System (CATS). He introduced Ms. Marcia Seiler, Director, Office of Education Accountability (OEA) who introduced Ms. Sabrina Olds, OEA Finance Analyst, and Mr. John Perry, Staff Economist, Legislative Research Commission (LRC). Ms. Seiler said Senate Joint Resolution directed that the study include an examination of the actual costs associated with CATS, on a per pupil basis. She said this portion of the study presents information on the costs incurred by state government agencies and local school systems for the development, administration, scoring, and dissemination of CATS accountability tests and results.
Ms. Seiler said the costs were broken down into three areas: 1) advisory and research; 2) implementation and administration; and 3) accountability. She said the estimate is for fiscal year 2004.
Ms. Olds presented the state cost section. She said the cost estimate of CATS incurred by the state was obtained in a direct manner by adding together expenditures of the agencies charged with various activities relating to the accountability system. She worked closely with the Kentucky Department of Education (KDE) finance staff in obtaining the data, and reviewed specific bills for contracted assessment services. The expenditures are broken out by state and federal dollars. Federal funds represent slightly less than eight percent of the total expenses of CATS, which amounts to approximately $1.6 million.
Ms. Olds said the advisory and research section of the report is further broken down into the different entities that offer advice and conduct research regarding CATS. She discussed the National Technical Advisory Panel of Assessment and Accountability (NTAPAA), who are a panel of national experts on testing appointed by the LRC to advise the Kentucky Board of Education (KBE), KDE, and LRC on issues relating to CATS. The total state expenditures for NTAPAA in fiscal year 2004 was $107,000, which is paid for by the LRC.
Ms. Olds said KDE contracts with HumRRO, a Louisville consulting firm to perform on-going technical studies related to CATS. She said payments to HumRRO were $239,000 in fiscal year 2004.
Ms. Olds said the School Curriculum Assessment and Accountability Council (SCAAC) is charged with advising KBE and LRC on issues related to the creation and implementation of CATS. She said state expenditures for SCAAC were $4,000.
Ms. Olds said that half of the advisory and research expenses were spent on the KDE Division of Validation and Research. These expenses include payroll, travel, and office supplies in the amount of $346,000. She said the total expenses for entire category was $696,000.
Ms. Olds said slightly more than half of the state level expenditures related to CATS are for implementation and administration of the accountability test. She said CTB McGraw-Hill is the test contractor hired by the KBE. Some of the contractor's responsibilities include developing the assessment, providing assistance in administering the assessment, and scoring and reporting the results of the test. The amount is set out in contract and varies from year-to-year. She said approximately $8.2 million was paid from state funds, and $1.6 million in federal funds for fiscal year 2004 totaling $9.7 million.
Ms. Olds said KDE contracts with the University of Kentucky (UK) to provide support to districts and teachers in administering an alternate portfolio to students who cannot take the regular CATS test. The contract was $249,000 for fiscal year 2004.
Ms. Olds said the KDE Office of Assessment and Accountability is responsible for overseeing the implementation of CATS. These expenses in the amount of $848,000 include personnel expenditures for the associate commissioner's office, and the Division of Assessment Implementation. The total expenses for this category were $10.8 million.
Ms. Olds said the accountability expenses include expenditures for reporting the CATS results to the schools and public, and any rewards for performing well, or assistance for the schools who do not perform well. The KDE Division of School Improvement oversees the highly skilled educators, the Commonwealth School Improvement Fund program, and scholastic audits and reviews. She said schools that are designated as a Level III school, or need assistance as a result of their CATS scores, are required to received assistance from the highly skilled educator that is trained by the KDE. There were 47 highly skilled educators in Kentucky schools for fiscal year 2004.
Ms. Olds said the Commonwealth School Improvement Funds are to be used to reduce achievement gaps for schools being classified as "needs assistance." She said of the $2 million spent, approximately $1.8 million went to the local school districts, and the remainder was spent on the KDE personnel and operating expenses.
Ms. Olds said Kentucky schools that fail to meet specific CATS goals are subject to a scholastic audit, or review. A team of educators reviews the personnel and academic operations of the school and makes a recommendation regarding how these funds can improve and enhance student academic performance. She said in fiscal year 2004, $375,000 was expended for this program.
Ms. Olds said the state spent $294,000 in printing and distributing report cards to the public that includes CATS information. She said the total state expenditures were $2.7 million, which equates to a per assessed student expenditure of approximately $43.00.
Mr. Perry presented local level CATS expenditures. He said estimating the local level expenditures was not as direct as the state level because there is no single source of local level expenditures labeled as CATS.
Mr. Perry said two surveys were used to estimate local level expenditures. They were sent out electronically to the school districts' central office personnel, and another to school personnel. He said 132 central offices responded representing approximately 79 percent of assessed students, and 796 schools responded representing approximately 62 percent of assessed students.
Mr. Perry said the implementation and administration category was broken down into five subcategories. They were: 1) district assessment coordinator with an estimated expenditure of $2.3 million; 2) writing consultants or cluster leaders with an estimated expenditure of $1.8 million; 3) supplemental personnel with an estimated expenditure of $3.1 million; 4) equipment, supplies, and software with an estimated expenditure of $2.5 million; and 5) prep tests with an estimated expenditure of $1.3 million. The total expenditures in this category was $11.1 million.
Mr. Perry said the accountability category was broken into three subcategories. They are: 1) professional development with an estimated expenditure of $2.0 million; 2) test incentives and rewards with an estimated expenditure of $1.9 million; and 3) other costs with an estimated expenditure of $1.3 million. The total expenditures in this category was $5.3 million.
Mr. Perry said there are no advisory and research expenditures for the local level. The implementation and administration of the per assessed student figure was $23.18 representing 68 percent of the total cost, and accountability was $11.10 per assessed student, and 32 percent of the total cost. The total estimated local expenditures were $16.4 million, and $34.28 per assessed student expenditure.
Mr. Perry said there was a large level of variation that existed in the local level survey results. This could be due to: 1) schools and central offices are spending different amounts for CATS; 2) similar expenditures could be classified differently; and 3) perceptions of what CATS expenditures are differ.
Ms. Seiler concluded that the state and local CATS related expenditures totaled $37.2 million for fiscal year 2004. She said this means on average almost $78 was spent per pupil as a result of CATS.
Representative Draud asked how the cost compared to last year's cost. Ms. Seiler said it was $20 million more than the survey conducted by the Program Review and Investigations Committee. She said Program Review did not do a local level analysis.
Representative Draud asked about the data at the local level and said that it seemed as if people did not take the survey seriously. Ms. Seiler said it depends on how people perceive the costs, and if they think the costs are just part of regular instruction costs.
Senator Winters asked about the huge discrepancy in the $77 per pupil amount for fiscal year 2004 and the $44 cost in fiscal year 2003. Is this because there was no data in the earlier survey relating to local cost? Ms. Seiler said she did not think so.
Senator Kelly asked if there was any information about these costs compared to other states. Ms. Seiler said they did not compare the costs with other states.
Senator Kelly asked if there was any attempt to try to identify the local responses that seemed to be more valid. Mr. Perry said they contacted some local school districts for clarification. He said other than that, it is difficult to know if a reported expenditure is exactly what was spent. The cost surveys were designed so that the people who would answer them would be the people most closely tied to where these expenditures would be made. Senator Kelly said there is some attempt in statistical analysis to take apparitions out in order to get a better picture, and wondered if there was an attempt to do this. Mr. Perry said statistically speaking, there are techniques to look at different variations, but there was no formal process here to exclude outliers because it is difficult to define what an outlier is for this type of data.
Senator Kelly asked what the $1.3 million was spent on for the prep tests. Was this for materials or teacher time? Ms. Olds said it was all materials.
Representative Marzian discussed the costs associated with distributing the reports back to the schools and the parents. She asked if these were paper reports or if it was electronic reporting. Ms. Olds said these costs were to mail the reports because the statute requires the reports to be mailed to every parent. Representative Marzian said the cost of postage and printing is probably significant. Ms. Olds said the results are also posted on KDE's website. Representative Marzian said maybe the statute could be amended in order to save money since more people have access to computers.
Representative Marzian said the $77 per student for testing is less than one percent of all expenditures, and seems like a fairly reasonable amount.
Senator Westwood said he is trying to understand the local costs. He said the draft report referenced two reports, the LRC Program Review and Investigations report, and a Hoxby (2002) report, which both quoted much smaller figures per pupil tested. He asked if these differences were strictly because of the way local districts do their reporting, or is Kentucky seeing an increase in the amount of dollars over those two or three years.
Ms. Seiler said Carolyn Hoxby is from Harvard, and her report was a general overview, and did not consider local costs. She said neither of those other two studies considered accountability costs, which could account for the difference.
Senator Westwood asked what the cost would have been if accountability costs would have been included. Ms. Seiler said she did not know if rewards were included in the year of 2002 when the Hoxby study was conducted.
Senator Westwood said he is not getting any relevant information from this study because he does not know how to interpret the data. Ms. Seiler said the state costs would be a better comparison to the other studies because the other two studies did not include local costs. She also said Ms. Hoxby's study was a study across the country and not just Kentucky.
Senator Westwood discussed rewards and sanctions. He asked what the average payout was to a school for rewards before 2004. Commissioner Wilhoit said a little over $22 million. Senator Westwood asked about the cost of sanctions. Ms. Seiler thought about one-half million.
Representative Draud said some parameters needed to be defined for the local districts in order to report local costs. He also said follow-up needs to occur with the districts to ensure they respond and take seriously that the data they report is important for making policy decisions.
Mr. Perry said the surveys were mailed with the idea of trying to obtain costs where there is no formal accounting system. If one school reports one number, and another school reports another number, it is hard to determine why clear differences are there. Representative Draud said however, if categories were defined, it would be more likely to receive helpful data. Ms. Seiler said the categories were defined in the survey.
Representative Moberly said it is irritating to hear that certain data cannot be obtained because of the way things are coded in the system. It seems like technology is a hindrance to getting information. He would like for someone to take a look at how this problem can be remedied.
Senator Kelly said there is plenty of expertise on local finance that could look at the survey results and identify where there is good data. It could be a representative population, but OEA should go back and ask schools for more information where a real good desegregation of the cost is identified.
Senator Kelly said the cost of the assessment window also seems to be missing from the results. He said two weeks preparing for the exam would seem to be a pretty significant cost if one criticism is the amount of time spent on the test. Ms. Seiler said that, other than those people whose job was strictly assessment related, the costs of teachers and principals was left out. Senator Kelly would like a number for what two and a half weeks of test taking is costing. He said this is opportunity cost as well as a direct cost.
Senator Winters asked about the 47 highly skilled educators that worked in the Kentucky schools last year. He asked if the $6,126,000 related to a special salary for the teachers that are there performing that function, or a replacement for that person in the school system. Ms. Olds said it was only for the highly skilled educator. She said the KDE signed a Memorandum Of Agreement (MOA) with a district to take the highly skilled educator. The district still continues to pay the salary, and the state reimburses the district for its cost. She also said someone is hired for the teacher's job, while the highly skilled educator is gone for two years.
Senator Winters asked what is included in the highly skilled educator budget. Ms. Olds said it is pays for their travel and meals, but it is mostly salaries. Senator Winters said that equals $130,346 per highly skilled educator. Ms. Olds agreed and said there is a cap as to much how much they can make.
Senator Westwood asked if there was any information available about how the highly skilled educators are doing in turning around struggling schools. Ms. Seiler said the OEA has not conducted any studies on that issue. Senator Westwood said they seem to be pretty effective in teaching schools to perform better on the CATS, and with the writing portfolios.
Senator Westwood asked if the cost was broken down for students that took the CATS and those that took a norm referenced test (NRT), such as the CTBS. Is there an average of all students who are testing in Kentucky? Ms. Seiler said the survey request was for per pupil so all costs were combined and were divided by the total number of students assessed. Senator Westwood asked if this skewed the results. Ms. Seiler said she did not know if they could break the numbers down by year, but prior studies were all conducted in the manner of this one by per pupil cost. Senator Westwood asked if the cost is underestimated because of that factor. Ms. Seiler said she did not know. He asked her to find out.
Senator Kelly discussed opportunity and direct costs. He said if 20 percent of a teacher's time is spent on test-taking techniques, this is huge amount of time being consumed for that purpose. He asked Ms. Seiler to get the cost for 20 percent of the teacher's time.
Mr. Perry said opportunity costs were not directly considered. Senator Kelly said he is not asking staff to calculate opportunity costs, but the direct cost. Ms. Seiler asked if he was referring to calculating the cost for the time teachers spend teaching the CATS test based on survey responses. Senator Kelly said the time it takes to administer the test is extremely relevant, and he would also like the prep time cost.
Representative Moberly asked how many surveys were sent out to teachers, and how many teachers we have teaching in the state. Ms. Seiler said that there were 338 respondents of the 1143 surveys mailed, a 30 percent response rate. Representative Moberly said he was skeptical of the numbers based on a 30 percent response rate.
Representative Rasche said validity is another issue that needs to be addressed. He said Kentucky needs to ensure it is testing for what it wants. He said certain parts of test prep are legitimate because it makes the test more valid. He said if a student does not know how to take a test well, then subsequently it is testing test-taking ability as well as the content of the test, and causing mixed results. He believes most of the issues about what is CATS and what is not really revolve around the question of whether what Kentucky is doing is valid for achieving our goals of assessing what the children know, and assessing what is being taught or learned.
Senator Kelly said he does not agree with the idea that if this is a valid test of student achievement then it is appropriate to spend however much time on test preparation. He said he thinks everyone agrees that some test preparation is necessary to have a valid test, and he doubts the report's figure of 20 percent of teacher time spent on test preparation. He said over 50 percent of teachers said they spent 20 percent of their time doing something other than working on the writing portfolio or teaching core content. He said he would be very concerned if he found that the 20 or 30 percent figure was really true because he has been hearing from teachers anecdotally that they do not have time to teach children what they should be testing because they are spending time on test preparation.
Representative Rasche said there are some abuses in the system. He said children do not think at higher levels until they practice it. He said open response questions are set up this way. He also said he agrees that the amount of time spent on writing portfolios would be much better spent on another part of learning, but it does not mean that the writing portfolio is necessarily bad training. He said if teachers are using the test prep as a diagnostic tool to find the holes in a child's learning, it is extremely valuable; if not, practice tests and other test prep measures are a waste of time.
Senator Kelly is highly skeptical about teachers spending 20 or 30 percent of their time in noneffective test preparation, but feels it needs further investigation. He asked about stimulating higher thinking from Kentucky students and attention by Kentucky's teachers by having open response questions. He also said it is costing Kentucky a tremendous amount of money, and the tests are less valid because it asks fewer questions, and the grading is subjective because of the nature of open response. He said the teachers are spending a lot of time in test preparation and test administration when some open response questions do not test higher thinking skills, just give the student a writing opportunity. He said multiple choice questions do test higher thinking skills.
Representative Draud said the General Assembly created a high stakes testing program with a lot of accountability. He said reactions from educators was to teach to the test. He said people assume this is a bad thing to do because they are uninformed about the issue. He said the assumption is that if what is being tested is worth learning then it is not wrong to teach to the test. He said in school districts where teachers and administrators are well informed they are teaching the concepts that are on the test, not teaching the exact questions. He said the key to open-ended response questions is teaching the teachers how to ask good open-ended response questions that require higher order thinking skills. He does not think a test that uses all multiple choice encourages higher order thinking skills. He said how to have a high degree of reliability and validity with open-ended response is difficult, and a lofty goal. At the same time, a lot of school districts are learning a lot about higher order thinking skills by responding and practicing open-response questions. He hopes that teachers showing students how to beat the system is not a rampant issue. He said historically, teachers have always complained about the amount of time they had to use for testing. This is not a new phenomena. He said teachers do not like a lot of tests, and it is alright for teachers to teach the concepts of the test.
Representative Moberly said it is appropriate for students to practice open-ended questions because it does teach higher order thinking skills. He believes the issues can be corrected with better professional development and implementation so schools understand better how to teach these items.
Senator Westwood asked if $77 is a combination of all students? Ms. Seiler said yes, all students that were assessed. Senator Westwood asked what the $6 to $8 for the CTBS pays for and how it fits into the $77.00 cost. Ms. Seiler also said she is not sure what all the cost includes. If it is just the paper test, reporting, or other accountability factors such as advisory and research, and implementation. She said those things are not included in the $6 to $8 dollar figure quoted for the NRT test.
Mr. Perry said there is a lot included in this $77.00 figure. He said he does not know what is included in the $6 to $8 figure for the NRT. Senator Westwood asked if he could get those numbers. A spokesperson for CTB McGraw-Hill spoke up and says it cost typically $6 to $8 for the test booklet, answer document, and the basic scoring services. It does not include the additional cost for shipping the materials to the districts. It can vary from that if someone needs a wide variety of reports. Mr. Perry said Kentucky is spending on average $78.00 per student no matter what test he or she takes in any given year.
Senator Kelly said the reason for studying the cost of CATS is because it is a custom designed test that is very expensive. He said that if one-third of students who take a standardized test are combined with the other students, it diminishes the cost of the more customized and expensive test.
Representative Moberly said the assessment system is part of a total system. He said state level expenditures on advisory activities and research would also apply to an NRT test. He said the highly skilled educators are part of an assessment and accountability system.
Senator Kelly said the focus is how to make the customized test work rather than use a NRT with all the problems that go with just using a NRT. He said averaging the per capita for all the students is diminishing the total cost of the customized test.
Representative Moberly said OEA and other LRC staff will produce any data that Senator Kelly needs, but he will not accept the data as valid unless it takes into account that CATS is one testing system that includes a NRT.
Representative Moberly introduced Mr. Gene Wilhoit, Commissioner, KDE, who gave his response to the study report. Commissioner Wilhoit introduced David Rhodes, Helen Mountjoy, and Hilma Prather, Kentucky Board of Education (KBE) members.
Commissioner Wilhoit said he estimated direct costs for CATS at $9.1 million, and if other costs are added in, it is between $12 and $13 million. He said this is not too different from previous survey costs.
Commissioner Wilhoit said he has more questions than answers about the local level costs. He said he is not sure that Kentucky can obtain the results it needs with another survey, but instead OEA should pick a few places and study them in more detail. He said categories need to be identified to use as qualifiers of parts of the system. He said another survey would probably result in the same disparity of this survey.
Commissioner Wilhoit said that successful school districts integrate their local practices into the context of the state process. He said they are giving formative assessments that are not only aligned in terms of content but also in terms of methodology. He said it is wrong for any school system to pull students aside and give them practice on a test item that looks like a question on a test. He said this type of practice should not exist in the Commonwealth. If a school district is giving a student an open-response question in context to the lesson that was just prepared, that is good practice. He said both points of view are turning up on the survey, and what people are finding is that some people feel they are spending an enormous amount of time on the CATS, which could be a positive. On the same survey, another school district could respond that they are spending a vast amount of time on the CATS, and that could be a negative response.
Commissioner Wilhoit said any system that is put into place, many of the costs associated with the study and a part of our system are really practices that are going to be going on whether it is this test or another test in place. He said Kentucky will always find ways to help local districts improve, look for ways to put resources and assistance behind those places that are not doing well, attempt to find better ways to practice education, and to work with children. He said he would provide additional written comments to the report as he further reads it, but these were his initial reactions to the first reading.
Representative Moberly asked Commissioner Wilhoit what was being done to improve the open response questions so that they teach critical thinking skills to students. Commissioner Wilhoit said if a school understands curriculum instruction and assessment and its proper role, then CATS will take an appropriate function in that school. He also said however, that if a school does not have an identified curriculum in place, or not using assessments in appropriate ways, and then adds a high stakes accountability system, instead of building up traditional practices in education, curriculum instruction, and support for children, those schools tend to try to figure out ways to do better on the assessment.
Commissioner Wilhoit said Kentucky has to have stronger criteria around professional development, and this has been done. It means Kentucky needs to promote to its school districts the very serious work around curriculum, assessment, and accountability, and their appropriate roles. He said most districts now are much better prepared to implement these changes than they were a few years ago. He said the state was going to provide teachers with tools that have not been provided in the past, and provide sample lessons, and appropriate assessment items that could be used for formative purposes at the local level.
Representative Moberly asked about the difference in cost for the augmented NRT versus the customized test. He said Kentucky would still have the expense of research expenditures, school improvement fund, scholastic audit and review, and other associated costs with whatever test Kentucky chose. Commissioner Wilhoit said the state needs those comprehensive approaches to have a systematic improvement, which he feels is absolutely essential.
Representative Draud discussed the open-ended response question issue. He said at his former school, teachers were required to provide sample open-ended response questions each week, and principals were critiquing these questions to ensure quality, and encouraging higher order thinking skills. He said this practice is the key along with professional development in ensuring the quality of the open response questions.
Commissioner Wilhoit said the principals provide a pivotal role in talking with teachers within the classrooms. He said Kentucky must support its principals in their own development so in turn they can provide support to our teachers.
Representative Moberly said there was a bill in 2004 that provide individualized student plans for children who were not meeting their goals. He said this intervention is needed to get some students back on track. He said schools that have succeeded have seen tremendous leadership from the principals and administrators within the school. Schools that are succeeding show demonstrated teamwork between the teachers and principals.
Senator Winters said it is beneficial for schools to have programs where higher education faculty work closely with the K-12 school system.
Commissioner Wilhoit agreed it is absolutely essential for postsecondary institutions to work closely with K-12. He is going to talk to the KBE to establish some centers of conversation sponsored by universities and partnerships with K-12 teachers to discuss these issues. He said the federal No Child Left Behind resources can be used to build a stronger correlation.
Senator Winters asked about upcoming KBE initiatives. He asked how much money it would cost to add the new dimensions of the KBE's assessment directions particularly helping student in choosing career fields. Commissioner Wilhoit said it is KDE's sense that Kentucky would be better off if it had some means of providing schools and higher education institutions some general progress reports. He said KDE's concerns about the results of college entrance admission tests being given to students in the 11th and 12th grades is that it is too late, and he would rather see an 8th and 10th grade predictor that would be directly tied to the college admission standards. He said this could be done for a fairly limited resource cost, but it would have to be put out for bid.
Senator Kelly said when the Kentucky Education Reform Act (KERA) was enacted it was to create high standards and a high stakes accountability system based on performance based testing that would test critical knowledge. He said it was realized well into the effort that there were problems with the testing system. The standards had to be re-written due to critics saying there were not high standards. He said significant adjustments have been made to the performance-based efforts, and there was a major overhaul which produced the CATS test. He said Kentucky still has significant issues with the level of reliance on open-response questions, the role of the portfolio versus on-demand writing, and the reliance on a customized test rather than some sort of augmented NRT. He wants to know what is going to be done about it and when. The RFP has to go out in August and consensus needs to be found before creating the RFP. He does not want to commit to another two to four years of the same assessment system.
Commissioner Wilhoit said the board agrees that changes need to be made based on concerns that have been identified. He said the new test design will rely more on multiple choice than on open-response questions. He said the board prefers a customized KCCT test rather than a customized NRT. He said the board has made some significant changes with the writing portfolio, but is still open for more suggestions in that area.
Senator Westwood asked why in 2002 Kentucky was spending $6,532 per pupil by local districts, and the figure is $8,029.84 spent per pupil in 2004. What caused the increase? Commissioner Wilhoit said the 2004 figure may have combined federal and state resources, but he would have to examine it closer. He also said the last two sessions have given significant increases to teacher salaries, and health insurance increases. He will provide Senator Westwood with a complete breakdown.
Senator Winters discussed the writing portfolio program. He said it is incumbent to gather consensus among members and roll out a new and enhanced program rather than allow individual components to be erased. Commissioner Wilhoit agreed and said the KDE and KBE stands ready to work with members as full and cooperative partners.
Representative Rasche said he has some extremely tough mathematics questions written in the format of short answer, which is a cross between multiple choice and open-response questions. He said it still has to be manually graded, but at the same time, the correct answer is not suggested as one of four or five choices. It is also faster to grade than the open-response test. He said there may be some things that Kentucky is trying to test that would be suited for short answer and reduce some of the expenses involved with grading the open-response format. Commissioner Wilhoit said he agrees with him and is exploring and having conversations with consultants about the advantages of those.
Representative Moberly thanked Commissioner Wilhoit for his input. He asked for a motion to accept the report and forward it to the LRC. He said this is not approving the report, but just accepting it. Senator Kelly said he would like staff to address the issues raised and review it at the next meeting. Representative Moberly postponed accepting and forwarding the report to the LRC until some members receive additional information and the report is supplemented.
Senator Westwood asked for the raw data from the study. He would like to see how the questions were worded, and read specific responses. Ms. Seiler said the survey responses were in the back of the members' handouts. Senator Westwood said he would still like access to the raw data. Representative Moberly told Ms. Seiler to communicate with Senator Kelly and Senator Westwood to determine specifically what additional information is needed.
Representative Moberly introduced the members from the National Technical Advisory Panel on Assessment and Accountability (NTAPAA) who discussed the proposed changes to Kentucky's accountability and assessment system. Dr. John Poggio, Co-director, Center for Educational Testing and Evaluation, University of Kansas, summarized NTAPAA's role and gave a brief background of its history.
Dr. Poggio said NTAPAA believes that Kentucky needs to seriously study the focus and the intent of its core content. He asked members to ask questions of the panel.
Senator Kelly discussed concerns about Kentucky's assessment system. His concerns are: 1) reliance on a single score to judge if a school is improving or not; 2) having a customized test prevents Kentucky with having comparison to what is happening in other states; 3) emphasis placed on open-response questions, and all the problems it creates with how long it takes to administer the CATS test, how reliable it is, and how quick information can get back to the teachers; and 4) not holding on to a test design out of loyalty to the concept.
Dr. Suzanne Lane, Professor, Department in Psychology in Education, University of Pittsburgh, said Kentucky's assessment program has a favorable attribute in that it is based on a variety of content areas, and not one single index. She said individuals look at Kentucky as being advanced in that area. She also said there is evidence that using multiple choice and open-ended questions on an assessment will create positive things in instruction. She urged the members to evaluate the benefits of the open-response questions and realize the cost that is lost in instruction if these questions are omitted.
Senator Kelly said the Commissioner of Education just said Kentucky students have a problem with mathematics. He said Kentucky is 14 or 15 years into this testing system, and he does not see the direct benefit of open-response questions in math instruction in Kentucky. Students are not showing any measurable improvement in this category.
Dr. Robert Linn, Distinguished Professor, and Co-Director, National Center for Research on Evaluation, University of Colorado at Boulder, discussed the customized test versus an NRT. He said if having comparisons to other states is important to Kentucky, then it would need to get away from the customized test. He agrees with Dr. Lane that an assessment should contain some open-response questions because it measures things that are impossible to capture with multiple choice questions. He would like for NTAPAA to specifically look into the issue of the math instruction in Kentucky.
Senator Kelly said he has seen Kentucky schools that have done extremely well, and they all contained the same factors that caused them to excel. He said the schools contained strong academic leadership that understood teaching, the ability to communicate to the teachers and build morale, develop strategies such as assessing early and looking at the results, develop intervention strategies, and then re-test. He said another use of assessment is for accountability purposes. He said Kentucky relies on the latter and this has been very problematic and very costly. He said the schools that excelled did not contribute their success to the CATS test.
Representative Draud asked if diminishing the open-response questions would have an impact on instruction. He said it is important to maintain the right balance to keep teachers focused on instruction and higher ordered thinking. Dr. Linn said Kentucky's open-ended response questions count two-thirds and multiple choice counts as one-third, while other states have less of a weight on open-ended responses and there has still been an impact on their instruction. Dr. Linn said there is evidence across the country that open-ended questions have an impact on what is happening in instruction.
Dr. Andy Porter, Director, Learning Sciences Institute, Vanderbilt University, said Kentucky has a history with CATS and with the reform, and a good study would be needed to answer alot of these questions. He said the open-ended response questions and the use of the writing portfolio has given Kentucky alot of attention both positive and negative. He said if Kentucky backs away from it, it will be a symbolic statement to the rest of the nation.
Representative Draud asked if Kentucky had the proper balance of multiple choice and open-ended questions. Dr. Porter said the only way to find out is to do a study to see what the effect is. He said the anecdotal evidence survey is one of the worst studies to do. He said the people that normally respond are usually not representative of the majority. Representative Draud asked if he was implying that Kentucky needs to do another study to find out this information. Dr. Porter said that was correct.
The NTAPAA members further discussed the weights of open-response questions versus multiple choice. Dr. Poggio said student scores have improved since shifting more weight to performance. He said it is dangerous territory to look for a right mixture or a number for balancing these two different types of questions. He said evidence has shown in the last 10 to 12 years for states to use the item type that supports the outcome interested in evaluating, and some of these outcomes as defined by the core content are best served by open-response questions. He said there is no empirical definition to find the right balance and Kentucky needs to understand what it is trying to measure, and what is the purpose. He said if Kentucky wants to influence instruction in the classroom, then the sound decision would be to have some performance assessment.
Dr. David Miller, Professor and Chair, Educational Psychology Department, University of Florida, said the number of open-response questions could be reduced, but he would not suggest removing them altogether.
Senator Westwood asked how students in other states would perform if taking the CATS test. He also wondered if the CATS test had ever been given to students in another state. Dr. Porter said he did not think that had ever happened, but one excellent mechanism Kentucky has for comparing to other states is the National Assessment of Education Progress (NAEP) state results, which Kentucky participates in. He said Kentucky does about average on the NAEP scores.
Dr. Linn said there has been some improvement on NAEP scores, but they have not been dramatically better. He said most other states are using some open-response questions although they may be in the format of short answer or extended response questions. He said very few states use entirely multiple choice questions.
Dr. Poggio said when states begin to compare information it typically fails. He said NAEP offers no flexibility as a NRT would.
Dr. Porter said Kentucky would probably do better than other states in writing because of the focus on the writing portfolio. He said the NAEP would miss this.
Senator Winters said 90 percent of complaints he receives about Kentucky's assessment system are related to the writing portfolio. He said he has not heard any conversations about whether Kentucky has the right balance of open-ended response questions and multiple choice. He said the sentiment of a huge number of people in the state is against the writing portfolio.
Dr. Linn said one way to deal with the issue would be to decrease the weight placed on the writing portfolios, and increasing the weight on the on-demand writing.
Senator Kelly discussed purpose and opportunity cost. He said we have cross purposes in our accounting system which include affecting what is happening in the classroom, and measuring what is happening in the classroom. He said open response questions hinder measuring student progress because fewer questions are asked, and they are not child specific, and must be done over a period of years. He said the opportunity cost is huge that Kentucky is losing. He also said the anecdotal evidence is overwhelmingly negative concerning the current assessment system.
Ms. Seiler discussed the study proposal for the review of the School Facility Construction Commission. Representative Moberly asked for a motion to approve the workplan for the study. Senator Kelly made the motion, and Senator Winters seconded the motion. The motion was approved by voice vote.
With no further business before the committee, the meeting adjourned at 1:05 p.m.