The 3rd meeting of the Education Assessment and Accountability Review Subcommittee was held on Wednesday, September 1, 2004, at 10:15 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Co-Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, and Dan Kelly, and; Representatives Jon Draud, Mary Lou Marzian, and Frank Rasche.
Guests: Hilma Prather, Kentucky Board of Education; Clyde Caudill; Cindy Rausch; and Wayne Young, KASA.
LRC Staff: Erin McNees, Audrey Carr, Janet Stevens, Jonathan, Lowe, Sandy Deaton, and Jo Ann Paulin.
Representative Rasche moved for approval of the minutes of the June 2, 2004, meeting and Representative Marzian seconded the motion. The motion passed by voice vote.
Representative Moberly introduced Mr. Gene Wilhoit, the Commissioner of Education, who gave the Department of Education's current review of the Commonwealth's accountability testing system. Mr. Wilhoit introduced Hilma Prather who is a member of the State Board of Education. She is the chair of the Assessment and Accountability Committee of the state board.
Mr. Wilhoit said that in the committee folders there were three documents that he will reference. Mr. Wilhoit said that the work that the Department of Education (DOE) has done has been in response to earlier conversations the board has had with this committee and also interactions with people in the field. The comments are directly related to how an accountability system can be valid and reliable, but also be one that supports positive teaching and learning that results in supports for students, teachers in the classroom, and the schools. They feel like they have made some advancements and are moving ahead but there are some areas that need to be addressed. They have been in this process for ten years and they have ten more years to go. Some schools have reached proficiency goals but many who have not. More troubling is that some schools are not on track.
Mr. Wilhoit said his comments would cover three broad areas: 1) teachers' understanding of core content; 2) improvements in the actual assessment process; and 3) issues of accountability in the current system. All the changes or suggested improvements they are attempting to make are in context of meeting two criteria: 1) do they continue to maintain a valid and liable assessment and accountability system; and 2) is there a system that is pushing instruction and teaching in the right direction.
Mr. Wilhoit discussed the document entitled "Seven Steps Forward - Taking Kentucky's Successful Education Reform to the Next Level" (which is a part of this record). He said on page seven there are some issues that need improvements.
The first issue is the need for a longitudinal component in the system. It is one that they are moving aggressively to address. Teachers continue to say they need some way to measure one cohort against another. It would be important for them to find some way to measure progress with individual students, or groups of students over time.
Secondly, they have heard a continuing theme that in order for the Commonwealth Accountability Testing System (CATS) or any assessment to be highly effective the result must come back to the teachers as soon as possible. If they don't have the results in a timely manner then three processes are not taken advantage of: 1) they don't have an immediate opportunity to counsel students about areas for improvement; 2) they don't have a way to adjust the instructional program; and 3) they don't have an opportunity to make an impact on professional development. The challenge is to provide a system that gives results much quicker.
Thirdly, they continue to hear that the teachers are accustomed to what they are being asked to do. It would be good if they could begin to engage them in the scoring process rather than relying on scores from outside the state.
Finally, there continues to be a major concern around student accountability. The department continues to hear that there needs to be a way to hold individual students accountable for the work they perform on this and other assessments.
This is an opportune time to be thinking about this because in September 2006 the current testing contract will be complete after the results are issued back from the 2005 - 2006 school current testing year. A new Request for Proposals (RFP) will be issued which will give DOE an opportunity to make some changes and improvements.
The DOE has identified seven improvement strategies that Mr. Wilhoit hopes will respond to the four areas of concern. He said that no system could remain static and he directed the committee members to page nine of the same document titled "Content and Student Performance Standards." Mr. Wilhoit said it is important that in any system you continue to review those standards to make sure that they are up to date and that they meet certain criteria. He said there are two criteria that they need to make sure they are aligned with. There are emerging national conversations and documents that need to be considered when making judgments. The most recent that they participated in was the American Diploma Project. It is an attempt to assess state standards against national criteria set by the business community and higher education. Kentucky is one of five states that went through the initial review and is attempting to align its standards. The expectations of the business community and of higher education are coming closer and closer together. All students need algebra, geometry, and a course in probability statistics. They need an ability to work through algebraic concepts. They are reviewing the National Assessment of Education Progress (NAEP) and other types of documents. They need to make sure that Kentucky's expectations are still world-class and are aligned with other state and national expectations.
Mr. Wilhoit said that he is concerned about the continuity of the standard statements across the disciplines and about the consistency and number of those expectations. When the assessments were developed a decision was made that an assessment would be made at incremental grade levels. They attempted to capture the learning that accumulated up to those points. This puts a huge burden on the four, fifth, seventh, and eighth graders and the teachers who are trying to deliver the instruction. They are now attempting to break those down into annual statements of responsibility which they think will help in terms of spreading the responsibility. This work is under way and they expect some results. The two areas of greatest work are mathematics and reading but they are moving into writing, social studies, and science. This is critical to have before they issue the new testing and assessment contract.
Representative Moberly asked Mr. Wilhoit to elaborate on what he meant by breaking the expectations down on an annual basis. Mr. Wilhoit said that when the expectations were developed they did so around those grades assessed. For example, in mathematics, there are statements of the knowledge and skills needed to be acquired at the fifth grade level. There were no statements made to the fourth, third, or early primary grade teachers about what pre-requisite skills are needed to be taught or mastered in order to move to the next level. The curriculum is not taught at the incremental levels and a great load of responsibility is placed on the accountability grades. They are talking about reviewing the expectations and breaking them down so as to carry the concepts forward at each of the grade levels. They will present these to the teachers in a way that they can understand that it creates a better document for organizing the testing procedures and also helps with the work done for classes three through eight in terms of meeting the annual assessment responsibilities under the "No Child Left Behind" Act.
Representative Moberly said that he was surprised because he always thought this had been done. He asked how else would the teachers in the other grades do continuous assessment that was thought to be a part of education reform. Mr. Wilhoit said the early thought on this was that this would be done, but it is not consistent behavior across the state. Representative Moberly said now he could see why some teachers say they don't have to worry about it because it is for the accountability level. He said he had never heard it expressed as Mr. Wilhoit expressed it today and he was surprised and a little disturbed. Mr. Wilhoit said he was also and they are going to correct this thinking.
Mr. Wilhoit said they will be much more specific in terms of guidance to the teachers. He said they have heard a lot of comments about the writing process. Those comments have been basically about the overload of writing responsibilities at certain grade levels, the quality of the training for the writing process, the grade level content that is expected in the writing process, and the scoring process that is being used by different districts - whether it should be uniform or not. He said he saw too much dropping of the curriculum at those assessed levels with almost total attention by certain teachers to the development of the portfolio. He said he saw too much emphasis on the "end" product, the correction, and the final phase and he saw too much writing out of context. They have organized a writing task force to help work through some of those issues. The goal is to not do away with writing in the curriculum. They feel that every student should be a powerful writer. They should be able to respond to a writing challenge and have that turn into an articulate response. They should also be able to do a more thoughtful analysis.
The writing task force has had two meetings. They will be reporting back in time for the October State Board of Education meeting. They hope that by December they will have some direction that they want to provide in terms of adjustment.
Representative Draud said the writing issue is very important to him because he thinks great progress has been made. The writing aspect of the reform program has been one of the most successful parts. He said he knew that Mr. Wilhoit is getting a lot of pressure from different groups to make changes. He isn't opposed to modifications but he doesn't want writing to become a lessor priority. Some groups are saying that we are doing too much writing but a part of being successful in education is writing. It forces you to think and the way you learn to write is by writing. He said that he does not want the system to de-emphasize writing in anyway in the state.
Mr. Wilhoit said that he hoped he made the point that this is not an attempt to de-emphasize the writing process, but it is to improve the practice of writing in the schools. He said they need to look internally at what they have done in terms of setting up a system to see if in fact the guidelines and the rules and the procedures are causing some of the inappropriate practices. He said his sense is that some are. For example, there is a senior writing piece that needs to be submitted. There is no reason why teachers couldn't provide greater guidance to the schools about the nature of that writing piece. He said there is no reason why they can't talk to the higher education community about using that writing piece as a part of the schools admission policies. That means there needs to be some greater guidance about the nature of that piece. The senior writing piece could be an accumulation of knowledge in a particular interest area of a student that could be taken to the next level of the education process. That writing process could be around an area of interest with research in that area being a particular component. The piece could be adjudicated at the high school level and then the student could carry that forward as an example of the quality of work that student can do as they enter the college system. That still holds to the concept of powerful writing, but it adds a context of relevance that has not been there and it adds a sense of meaning to the student that is engaged in the writing.
Senator Kelly asked how the timing of conclusions of the task force on writing and the review that is being conducted by the Office of Education Accountability (OEA) are going to align with the timing of the RFP. Mr. Wilhoit said he wasn't sure about the ultimate timeframe for the OEA report, but he thought it might be complete by December. They have a similar timeframe for the writing task force's work. He thinks both of those will be completed in plenty of time to be used in the new RFP.
Senator Kelly asked what the RFP schedule was. Mr. Wilhoit said in the papers distributed titled "Staff Note" there is a section under RFP. Ms. Prather said basically as a board they will have to make all their decisions regarding the design for improvements on CATS by the end of this calendar year, January at the very latest, in order for those to be in the design for the RFP. In February, the RFP will go out to potential contractors. The task force should finish its work by the end of October and they will be bringing back several suggestions for the board to consider prior to December. She said that Mrs. Siler at OEA has attended the task force meetings which has been very helpful for the task force. For once it seems everything is working together as far as time lines are concerned. Ms. Prather said the first change that will be seen with the assessment as far as it relates to accountability will be spring 2007. That is the normal sequence of things when you develop an RFP and establish contracts. Senator Kelly said that if you want the spring 2007 assessments to be different the conclusions on what will be done will have to made by December or February at the latest.
Representative Moberly said some of the members visited the Oldham County School system at the last meeting of the Subcommittee on Elementary and Secondary Education. There they learned how writing should be done and integrated into the classroom. He said he was very impressed. Mr. Wilhoit said that while they are looking at writing they are also looking at the arts. The current way of prompting students in the arts seems grossly inadequate because students can't react to artistic work on a paper and pencil test. They are exploring areas to enhance this area by using some prompts and technology. The state is not in line with the national developments that have occurred since the development of our system. At the national level the system is generally an exposure or an expression of the arts for kindergarten through eighth grade with some specialization at the high school level.
Mr. Wilhoit said that the longitudinal assessment is moving ahead. He said he thought the appropriate time to do this was in line with the next accountability cycle. The next conversation should be whether they take it into the next level beyond English and mathematics and do the same in science and social studies.
Mr. Wilhoit said the next issues represented are critical. The topic of on-line testing and teacher scoring are related to the drivers that he mentioned earlier. Teachers need the results faster if the results are to have an impact on the process. Mr. Wilhoit said in order to make these kinds of improvements there needs to be a way of assessing paper and pencil testing. There is a desire to move from the paper and pencil testing concept to one that is technology based. DOE is talking about a system where at a point in learning a teacher would determine that the student is ready for an assessment of a certain content area. That student would go to a secure screen, would open the book electronically, and then go through a two part assessment. It will have to be a divided assessment. They want to continue to have both multiple choice questions and open responses. Both are valuable and to give up either one would not be a good design. As soon as the students are finished with the assessment they close the book electronically. Currently the test is boxed, accounted for, sent to a scoring site, then they wait for everyone to convene to score the tests. The results are tabulated and then they are sent back. The DOE envisions multi choice responses that could be immediately scored and provided to the teacher, school, district, and the state. They also envision a process as they refine the core content statements and structure for those statements. This would be a process whereby they can have more released items that can be held at the local school level for teachers. This gives the school results for decision making and a mechanism for student accountability.
Mr. Wilhoit said this will take some work and he proposes they pilot it this year in some schools at the high school level but in controlled situations so they can begin to test the capacity of local schools to administer the test and of reporting back to the state and the local districts. Technology can and is doing this in other sectors.
Representative Moberly asked if there would still be a testing window. Mr. Wilhoit said that this opens up a tremendous potential and he would assume in the short term that there would still be a testing window. It does provide for an opportunity for a wider window than is present right now because they need the time for the linear process on the other end. It does allow for some greater latitude within security parameters for a broader window of assessment than is present right now. One can reconfigure the testing environment and, down the road, it does open others. As the move is toward performance-based accountability for students they could assess a student at a point in learning as opposed to a particular time in a secure environment that could still be used for accountability purposes. What that does is open up tremendous opportunities in terms of high school curriculum. For example: A student that is accelerated may not have to wait until the end of junior year to be assessed if their academic learning program is taken through that content at an earlier stage. Mr. Wilhoit said that it is exciting but he didn't want that excitement and promise to over shadow that there are some technical issues that will have to be worked through and monitored to make sure the system operates very effectively.
Representative Moberly asked how that individual student accountability at the option of the local district relates to the question of whether it is reliable at the individual level. Mr. Wilhoit said that is one of the technical issues that they are thinking about. He said one dilemma is how to cover core content. Kentucky is the only state that can say they have 100 percent coverage of core content on assessment. In order to be diligent in this aspect there are six forms and a combination of 24 plus six tests with open ended response questions in each content area. If Kentucky does a good job of identifying the core content rather than looking at a cumulative system or looking at the end result, in those grade levels, it really does provide a more stream-lined guide for our blueprint development. That gives the possibility of fewer forms so there would be the ultimate test package with development items and matrix items that wouldn't be released, but some released items would be common items that all students could take. That is a different design, but it is possible and one that can be defended.
Representative Draud asked Mr. Wilhoit how many pilot projects he was talking about. Mr. Wilhoit said that it would be a small control group in multiple sites at different high schools.
Representative Draud asked if they were thinking about the technology part for formative and summative evaluation. Mr. Wilhoit said it was both and he wanted to talk about support services. He said they didn't want it to be a guessing game for teachers or students so they are talking about a development of a snap-shot assessments that could be used for diagnostic purposes at the local level. It would mirror the format and substance of the assessments that would be in the secured test, but they would be open for teachers and released on a regular basis. A student could use those formative assessments at the beginning of the year to make sure they are at a point of learning. It could be used for developing an instructional program. They could use it mid-year for a check point and it would be developmentally designed so it would fit the sequence of learning or the expectations. They are finding that teachers are asking for this type of diagnostic assessment and it is going to be a very positive tool.
Representative Draud asked if they would also do a summative assessment. Mr. Wilhoit said the summative assessment on their part would be the secured assessment. Ms. Prather said the snap-shot assessments they are proposing would not be part of the accountability model. These would be in response to what teachers have said they need to see how students perform on "Cats-like" assessments as teachers are proceeding with instruction. It is a two-pronged approach. They would be using technology in the accountability model as well as in response to what teachers have said they need at the formative stage. Representative Draud said he thought this was exciting and commendable that there are going to be pilot projects. Mr. Wilhoit said those tools are going to be divisible depending on teachers security and skill. There will be an item bank - multiple choice items that can be pulled down independently by the teachers. There would also be the self-contained ones for the teachers that don't have the time or feel uncomfortable with developing those kind of assessments. There would also be open ended and multiple choice questions.
Mr. Wilhoit said the end of course exams are included. Over the last few years there has been a conversation at the Southern Regional Education Board about a number of states exploring the concept of student accountability. They are doing it in two ways. He said he has expressed this to the committee in the past, but personally he has been opposed to high school exit exams for several reasons. He said high school exit exams; 1) are too late in the process to really make judgments about student progress and they don't give you the opportunity for incremental development; and 2) in other states they have been called high school exit exams and have been geared to the eighth and ninth grade levels in competency and they force the curriculum to be designed to a lower level. He said they did not want that. This is incremental accountability against curriculum. It allows these states to re-test in those areas and students can then go back relearn and retest and upgrade their assessment results. Mr. Wilhoit said he was not proposing that these become a part of Kentucky's accountability system, but that they are tools that the state could develop and provide. The Department would only gather information about student results, but local schools could use it for whatever accountability purposes they would want. This would provide consistency in results around course taking patterns. He said they had done all they can do at the state level in terms of rigor. They put the new standards in place in 2002 around the curriculum expectations. They still have to deal with this in areas of mathematics. Generally, they are telling schools that they have to teach a rigorous curriculum. They have to show on a students transcript that they have participated and been successful in a rigorous course of study. What they are getting is very different results. They surmise that what one describes as Algebra I in one school is different in another school. If the description is the same they have two teachers teaching two different levels of knowledge. End of course exams would provide a way to judge students against the performance of other students across the state. It allows the district the capacity to begin to correct some things. He said he would recommend that it be a tool to make some analysis of what is happening in the curriculum.
Representative Marzian said she thinks it sounds fascinating and she asked if it could save some money. Mr. Wilhoit said it saves time and has tremendous turn-around advantages. He said he has talked with other states and has found that it is about one-third the cost of regular processing. If you consider the whole package there would be some additional cost. One area where it would be more would be teacher scoring. Mr. Wilhoit said they want to move the scoring process to Kentucky. Right now the process is to package the tests and then ship to Indianapolis for scoring. He said they want a regionalized scoring process using Kentucky teachers after the close of school. It would take about 1,500 teachers two or three weeks to do this processing. What they would have to pay teachers would be more than the testing contractor is paying its scorers right now. While there would be a savings on the technology there would be some off-setting cost in terms of pay to the teachers for scoring. They are shooting for a total package of no additional cost.
Representative Marzian said they would have immediate results on the multiple choice questions. Mr. Wilhoit said the wonderful part of the system is that the teacher would know that day how the student scored.
Representative Moberly asked if CTBS would be doing the technology and the pilots. Mr. Wilhoit said they want to separate the exercise of trying to figure out the capacity from the actual redesign of the system. They have a subcontract to do scoring and development of tests with a group called E-College for those students with disabilities that need some type of assisted program work. The platform work is there, but they have not applied it to a broader scale. They would have to take the test items they now have and transport them to a technology format and then have those applied and scored to the technology based learning. There would be some additional cost that will hopefully be covered with federal resources for the "No Child Left Behind" application.
Representative Moberly asked if it would be picked up on the assessment if a teacher was teaching Algebra I not quite up to the appropriate standards. Mr. Wilhoit said it should be caught. Representative Moberly said a lot of what Mr. Wilhoit is saying is that after looking over the system, things are not being done the way they were originally intended. Teachers in the field who are sincere and hard working are asking us to do some things differently.
Mr. Wilhoit said this was basically the issue. What he thinks they have captured is a responsiveness to what they perceive as being some of the issues that need to be addressed in order to move forward. He said he thinks by doing so they keep the standards high and they are still operating within the intent of legislation. Mr. Wilhoit said that if we don't do some critical work in the next few months then we will be facing another RFP and will not have done the kind of deliberate work that needs to be done to move ahead.
Senator Kelly said he was starting to understand now why making and implementing policy at the government level is so difficult. He said it is exciting to talk about some of the things that can be done with technology, but by trying to look at all the issues addressed and keep them within the same concept of how the CATS test is administered now, it is just creating all kinds of problems. Talking about student reliability and the fact that students are being tested in a way that allows system-wide testing but not individual. That makes it very difficult to have this longitudinal type study where there is earlier assessment of the students to see if they have improved. We have been struggling with this problem for over a decade. Senator Kelly said one of the things that jumped out at him was the arts program. He said he hates the thought of spending a lot of resources and energy and effort to see how assessment can be used to promote the arts. We know what good art programs are and there have been good models all over the state. Trying to design a test that is going to measure the arts which is hugely difficult and secondly then to hope that the measure will hopefully promote good practices in the schools rather than just promote teachers running around pulling their hair out trying to figure out how they can make the students do well on the arts part of the test. Instead they should be thinking about how to have a great arts program that really excites students and exposes them to a real full and rich program. Senator Kelly said we think we have to test everything for accountability. He said the students need to be tested diagnostically when they come in and then assessed as Chairman Moberly suggested and then teachers need quick turn-around of the results. He said there is a lot of effort and expense to figure out how to get the information back to teachers earlier when we know on a individual level it is not valid data. It is not really helpful to the teachers to get that information back early if is not designed to give an individual assessment of student performance. He said we have a few months to make some critical decisions about how we are going forward on the assessment. There seems to be a huge impediment in that we seem to be locked into the model that we have been struggling with for so long. The teachers are just starting to get use to this system of assessment and they don't want us to throw out everything that has been done. Certainly some major adjustments and changes seem to be in order if they are well thought out and could build on what has been learned. We should not be afraid to make some major changes.
Mr. Wilhoit said he would agree and he hoped they were seeing this as an attempt to move from sole reliability and dependence on a single test. That test can give us a check every year about a goal of making sure that everyone of the schools is making progress. It can provide another measure, but a lot of the proposals here are adding some elements that are educationally sound and don't call on that system to continue to carry as much weight as it has in the past. For example, when the student opens that book they may not see anything about the arts because we have asked everyone of those schools to make some judgments about how their students are doing in the arts and simply report that to the board. There may be a better way to get students engaged in a robust set of arts activities. That same thing might apply to practical living and vocational skills.
Senator Kelly said that was exactly the point that he wanted to make. He said we know what best practices are and assessment and accountability programs could include just accessing practices and if they appear to be implementing the practices that we know are good they don't necessarily have to perform on a test. Mr. Wilhoit said there are places that have attempted to do this. He said what they also see occurring is that practices don't many times lead to the positive results that they are expecting. He said you do need some way to look at all the activity has on learning.
Senator Kelly said what has been seen at the schools that are successful is that they are assessing the students early and then coming up with strategies as to how to deal with kids that are not performing. They are then assessed again to see how or if they have improved. As long as the students are being assessed with inexpensive, easy to apply assessments, and students can show they are doing better on the end of year assessments than at the beginning of school assessments, that is all that is needed. Mr. Wilhoit said the only issue he would want to make sure is that the standard they are using to measure that success is a standard comparable across all the schools. Senator Kelly said that is an area that the department can assist by providing some agreement on what the assessment tools ought to be. What seems to be of the most concern is that it is a "Cats-like" assessment. It can't be an assessment that is easy to administer, inexpensive, and gives quick results fast because it has to be "Cats-like. " Mr. Wilhoit said when he uses "Cats-like" he was not talking about so much the structure or environment of that test. It is the expectation levels of the students. Mr. Wilhoit said he would agree with Senator Kelly totally that if a school is waiting on CATS results as a sole barometer of whether they are successful or not that school is going to be a failure. Every school ought to have a series of diagnostic assessments in place. In fact, one of the components of this system is to make sure that as a department we are helping the teachers develop these and providing resources for teachers to do so. The biggest issue Mr. Wilhoit said he would have about each school going in its individual direction and making those individual judgments would be that you move away from a standard that has been established or an ability to measure accomplishment against that standard.
Senator Kelly said we have not reached the point where we think we have a tool that has really given us a reliable measure. There are so many other measures out there. We are sinking a lot of resources, effort, and energy into trying to develop a single model assessment given at one time of the year that is going to tell whether or not there is success going on in the classrooms. This is interfering with the ability to adapt some of these other things are working. Senator Kelly said the goal should be to adopt what they know is working based on experience over the decades. One would be less reliant on some universal test given at a specific time that is going to be a measure of what is going on in the school.
Mr. Wilhoit said it is a complex task. He said that he would still submit that it is a worthy goal to have some sort of benchmark against which schools should achieve. Senator Kelly said he didn't disagree that there should be benchmarks he said he just questioned how it should be measured. He said in conclusion his idea is that you have a school that assesses their students and they show that out of this population, they began here in the beginning of the school year and have produced these results. How they are doing on NAPE and ACT and various scores should also be considered. Schools should have in place the appropriate practices of screening students early for diagnostic purposes, developing sound intervention strategies, and then have a reliable measure to see that they are being successful in their intervention strategies.
Representative Moberly said he had all that in HB 193, but it got lost in the Senate somewhere.
Senator Casebier said that he wished Senator Kelly could have been with the members last week in Oldham County. The Subcommittee on Elementary and Secondary Education went to Buckner School in Oldham County. He said they heard from two teachers and the instructional superintendent. The two teachers were writing teachers and they talked about how they implemented the program. He said he wished that this could be replicated throughout the state.
Senator Casebier said the instructional superintendent talked about the math portfolio that is no longer in the state assessment, but has been continued in Oldham County because they feel like it is a valid instructional tool as is the writing portfolio. ACT and SAT are both going to include a writing component on their next assessment at a time when the state is talking about eliminating the writing portfolio. The instructional superintendent in Oldham County said that if the state did that they will continue with it because of the results they have seen. It has been such a struggle through the years to get everyone on board at the classroom and school level. In those instances where everyone has gotten on board we have seen it produce what it was intended to do. He said that the reason we keep holding on to "Cats-like" assessment is not so much that we are wedded to the test but because of what it has done to drive instruction. The music assessment has been a struggle since the visual and performing arts graduation requirement came into place. He said he and Mr. Wilhoit have had countless discussions and have talked with the Kentucky music educators who have helped design and a pilot this assessment system. Music and other educators have said that what is tested is what is taught and since it was finally placed in prominence in the state assessment it became important. Administrators throughout the state started to pay attention to the visual and performing arts classes because it is part of educating the total child. It had some unintended consequences because of the implementation in the schools. Some of the principals were saying it was either or and it was not intended to replace the performing organizations in those schools.
Senator Casebier said that Mr. Wilhoit has been very willing to listen and work with members of the arts community. He said he thinks it is important to have that in the state assessment yet not necessarily as part of the CATS test. When it is not significant it gets pushed to the back burner. Senator Casebier said that when the Kentucky Education Reform Act (KERA) passed, music was the original performance based assessment. As performance assessment has spread from band to orchestra to choir and art assessments and evaluations of that nature it has elevated those content areas just as it has with science and practical living and the others skills. Those subjects are not more important than reading and math, but to some students they are just as important. Those are the subjects that get some students in school and keeps them.
Representative Moberly said that the Oldham County teachers had managed to grasp the concept that we are looking for. They have a powerful writing program and have avoided the unintended consequences. He said that the original intent was to have continuous assessment, powerful intervention strategies and this won't work unless you have a benchmark state standard assessment. It drives those kind of instructional strategies. You have to have that or you won't have an effective set of continuous assessment and intervention strategies. This has to evolve and a problem that occurred during the crisis in 1998 was that we had refused up to that point to acknowledge that there had to be an adjustment and evolution, as there was a need for change. Representative Moberly said he was excited about Mr. Wilhoit's presentation. It reinforces that we have the right concept but that some of the things that should have been done to help with implementation are not being done. Representative Moberly said that at least Mr. Wilhoit is coming in and telling the committee that some teachers are not teaching adequately enough and there needs to be end of course assessments and other things implemented to attain the original concept. He said he thought it was healthy that we are admitting we need to change and do better in certain areas. He said he was firmly committed and believes in the original concept that there needs to be a statewide assessment.
Representative Rasche said that in Oldham County they also consider what is on the ACT and SAT exams as part of expectations of students when they were aligning their curriculum. That is a performance assessment that these students have to participate in. Their math portfolio is not like the one that the state had. Instead it is "Cats-like" and covers a variety of areas and is done at all grades. He said he had read two articles lately in which the teacher discussed the type of open response questions that are on the CATS as opposed to what teachers normally assess with. In both instances the teachers were surprised to discover traditionally students are tested at a higher level with reading problems. They said that is different than the open response. The open response does go a level more. The teachers had children who were excelling in their normal problems and reading problems who would bomb on the open response. It does ask for a different skill. Having these questions available to the teachers all through the year for their on-going assessments will be one of the biggest contributions. Representative Rasche said that Mr. Wilhoit pointed out that we are the only state that covers the core content and most tests only do 20 to 30 percent so we would end up emphasizing only a portion of what the curriculum really amounts to. We have to have something that we develop until off the shelf test cover a lot more area than they have so far.
Representative Moberly thanked Mr. Wilhoit for an outstanding, thoughtful, and interesting presentation. He said he appreciated the processes they are going through and that he thinks they are doing and outstanding job.
Representative Moberly said the other item on the agenda was the status report of the committee's study of the CATS system by Ms. Marcia Seiler, Director of the Office of Education Accountability.
Ms. Seiler said she was glad Mr. Wilhoit went first so the committee could see the steps that DOE is taking in reviewing the assessment and know the study her staff is doing is looking at the current assessment as it is today.
On June 2, 2004, the EAARS committee was presented with a study plan that was prepared to address the issue set out in SJR 156. This was reviewed and approved by EAARS. Ms. Siler said they took that same plan and pulled out of it the two elements that needed RFP's and they created and sent them out. They pulled this information and presented it to LRC. LRC approved the issuance of the RFP's and delegated the authority to sign contracts to the co-chairs.
She said today she would give an update of where they were in the process in both the study and the RFP's. Ms. Seiler said that at the last EAARS meeting it was suggested that Commissioner Wilhoit and the department review their study plan and make any suggestions or recommendations. Mr. Wilhoit did send a letter to the co-chairs. Ms. Siler said she wanted to point out one observation that Mr. Wilhoit made today. He said the primary focus in this study was on student results whereas the CATS assessment is a school level assessment. She said he made other important observations in his letter and she will take those into consideration as they proceed. She said she will continue her communication with Mr. Wilhoit and others in the department to make sure they are receiving the most updated information from the department and also they know what the department is doing as far as their review of the assessment.
Ms. Siler said that after approval was granted LRC staff then prepared two RFP's and mailed over 50 copies to individuals, universities, and organizations across the United States. In addition they posted the RFP's on the LRC web site. After 30 days they received four proposals for each of the two RFP's. The first one was for an extensive review of literature regarding large scale assessment and specifically the component used in CATS. Based on that assessment the contractor is to report whether various methods of assessment including open response, multiple choice, on demand, the writing portfolio, and the alternate portfolio have been shown to yield valid indicators of students knowledge of core content. That was all set out in the RFP. The second RFP was for focus groups of educators, parents, students, school board members, and administrators. They were requesting in that RFP that the successful contractor go out across the state, convene focus groups, and discuss topics that are set out in the study. The results of the focus groups are to be used to conduct a survey of the same groups statewide. In addition, they will use the survey and focus groups response to respond to other questions set out in the survey.
Ms. Siler said that in late July a group of OEA and LRC staff convened and they went through all the proposals. It was agreed that two organizations submitted the best proposals and were most responsive to what they were asking. She said they asked those representatives to come to Frankfort and present a study plan and answer questions and discuss their proposals. In mid August LRC staff again met on two separate days and heard the presentations of both groups. The staff met again and they unanimously agreed that there was one company that presented the best plan for both RFP's. At this time with agreement with the Commission they are in the process of recommending to the co-chairs that contracts be issued to this company under these RFP's so they can start their work. There is a tight timeframe. Their hope is that they can get these contracts signed and get the work underway.
Ms. Siler said that it is anticipated that once the contracts are approved that the literature review will be conducted within 75 days. That will put it in early November for that document to be completed. The focus and survey work is going to take more time. They are hoping to do between 24 and 48 focus groups before the survey and then send the survey out. They will have to give adequate time for response. One date that puts a hitch into the plan is CATS results come out in early October. The contractor says they need to make sure the survey doesn't straddle that date so the survey will have to be in mid October. They hope to get the focus groups conducted in September and early October. In addition to work done by outside contractors, OEA staff has been collecting data concerning the cost analysis. However, much information is going to be gained through the survey and focus groups. They will be discussing the cost analysis with superintendents and administrators in the districts. In addition they have begun work on the CATS and NCLB alignment issues and Ms. Siler said she has been sitting in on the writing task force discussions. She said this has been most informative to her because she is getting insight into what is going on out in the state as far as the writing programs and what the concerns are.
Senator Kelly asked if internally OEA would be pulling together a lot of this information that will part of this study. Ms. Siler said yes, but the main focus has been to get the contracts going and gathering information that they can provide to the contractors so they can do their work. Staff has been pulling information on the portfolio audit results and trying to analyze that information with the help of LRC staff.
Senator Kelly asked how Ms. Siler feels about the amount of staff she has available for the work. Ms. Siler said they hope to handle the work load within the timeframe. She said they have graciously been offered the assistance of other LRC staff, the economists and the Program Review Committee staff. Finance staff within OEA is working on the cost analysis. It is going to be difficult, but she said she believes at this point with the contractors doing their job and with the help of LRC staff they should be able to do it. She said she would update the committee if the timeframe is not being met.
Representative Moberly said the committee appreciated the update and asked Ms. Siler to let the committee know if there is a problem of support and time.
There being no further business before the committee, the meeting adjourned at 11:40 a.m.