The 2nd meeting of the Education Assessment and Accountability Review Subcommittee was held on Wednesday, June 2, 2004, at 10:00 AM, in Room 131 of the Capitol Annex. Representative Harry Moberly, Chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator David L. Williams, Co-Chair; Representative Harry Moberly, Co-Chair; Senators Lindy Casebier, Dan Kelly, and Ed Worley; Representatives Jon Draud, Mary Lou Marzian, and Frank Rasche.
Guests: Clyde Caudill, Kentucky Association of School Superintendents and Jefferson County Public Schools, and Paul Blanchard, Eastern Kentucky University.
LRC Staff: Sandy Deaton, Audrey Carr, Janet Stevens, Jeff Adamson, and Lisa Moore.
Representative Rasche moved for approval of the minutes of the March 18, 2004, meeting and Representative Marzian seconded the motion. The motion passed by voice vote.
Representative Moberly introduced Ms. Marcia Siler, Director, Office of Education Accountability (OEA), who explained the study proposal for Senate Joint Resolution 156: Study of the Commonwealth Accountability and Testing System. Ms. Seiler explained and discussed six objectives of the study proposal. The six objectives are: 1) to determine the appropriateness of CATS components in measuring achievement levels of all students related to the core content; 2) to assess the validity, reliability and adequacy of the CATS component scores as indicators of achievement for each individual student and to identify appropriate uses of the individual scores; and to assess the quality of the information provided to parents; 3) to assess how well the components of the Commonwealth Accountability Testing System (CATS) test align with the assessment requirements of the federal No Child Left Behind Act (NCLB); 4) to review the validity and appropriateness of the writing portfolio; 5) to identify the specific cost factors that should be included in an estimate of the actual dollar costs of CATS, and using available data regarding those factors, to estimate a per student cost of the current assessment system; and 6) to determine the opinions of state educators regarding the positive and negative effects of CATS on the development of curriculum and on instructional practices. A copy of the study proposal is available in the Legislative Research Commission (LRC) library.
Senator Kelly said an issue he would like to see addressed would be the open-response questions, which is an extremely important part of a student's assessment, but also a troublesome part. He said the insistence upon using open-response questions has made it difficult to have individual student responsibility because fewer questions are asked, grading takes longer, and there is more subjectivity interjected. He said the policy standpoint for using open-response questions was to improve critical thinking skills among students. His question to Ms. Seiler was to determine if the open-response questions improve critical thinking skills in students, and promote more productive teaching strategies in light of the fact that they produce so many problems with the amount of time, validity, and subjectivity in their use. Ms. Seiler agreed that this could be an area for the experts to evaluate.
Representative Draud said the issue was appropriate to examine, but he thought it would be difficult for researchers to ascertain the information on any kind of objective criteria. He said receiving input from teachers on their opinions of open-response questions and improving critical thinking skills among students could be valuable. Representative Draud said most teachers feel that these questions do encourage critical thinking.
Ms. Seiler said there is a plan to contract with a group or an individual to conduct focus groups throughout the state, and input could be received from teachers, administrators, parents, and students during those meetings.
Senator Kelly said there was a comment on performance events that brought this question to his mind. He said performance events were incorporated into the assessment for the same idea as the open-response questions, but researchers said the performance events were in fact skewing the results of the test. He wants to ensure that researchers believe the open-response questions are in fact promoting critical thinking skills among students.
Ms. Seiler said getting the literature review contract approved is the first priority because the results can impact various other parts of the study.
Senator Williams asked if anyone was present in the meeting from the Department of Education (DOE). Representative Moberly invited Ms. Bonnie Brinly to join Ms. Seiler at the table. Senator Williams asked Ms. Brinly if staff in the DOE examine the appropriateness of CATS on a daily basis. He does not understand the reasoning behind the hiring of experts to study the issue when there could be staff located within the DOE already working on this subject.
Ms. Brinly said DOE staff has been taking a series of policy issues before the Kentucky Board of Education. They are looking for ways to further improve the system, and for ways to strengthen the system to the assessment in order to improve instruction. Senator Williams asked Ms. Brinly if staff was looking at issues at the level of the study proposal. Ms. Brinly said yes, the DOE has been meeting with experts, and consulting with the National Technical Advisory Panel on Assessment and Accountability (NTAPAA), and working with other states by looking at policy reviews.
Senator Williams asked Ms. Seiler if discussions with DOE staff had occurred in regard to the topics of the study proposal. Ms. Seiler said that the DOE was not involved in creating the study proposal, however the OEA does attend the NTAPAA meetings who review issues such as are in the study. Senator Williams said the OEA as it exists should be doing only this type of policy work. He said there is a problem that there is not alot of discourse between DOE, OEA, and the legislature.
Senator Williams asked Ms. Brinly if DOE believes it can only discuss policy issues with the state board, or only bring problems to the Education Assessment and Accountability Review Subcommittee (EAARS). Ms. Brinly responded that it is DOE's intention to talk about issues in both places. She said Commissioner Wilhoit identified some changes that he felt would be necessary for improvements in the assessment system. She said DOE is facilitating broad consensus before changes are made. Ms. Brinly also commented this is a good time to make changes in the assessment system because the system is currently being revamped in order to address the requirements of the NCLB Act.
Senator Williams asked Ms. Brinly if DOE had reviewed the study proposal's objectives, research questions, and research methods. Ms. Brinly said she had read the proposal this morning. Senator Williams said that DOE should have an opportunity to provide input before the EAARS committee makes any serious decisions on the direction of the study.
Representative Moberly said the DOE and NTAPAA are working on all of the issues identified in the study proposal. He said there is regular discourse with the department and with NTAPAA and the EAARS committee. He understood the idea behind Senate Joint Resolution 156 was to conduct a study independent of the DOE and NTAPAA. Senator Williams commented that an independent review does not necessarily mean an isolated review. He said it is not fair to ask DOE staff to make constructive suggestions to the workplan if they have not had an opportunity to review it. He assumes after DOE staff receive a copy today, that DOE will have time to look over the study, and provide input on it to the EAARS committee. Representative Moberly said that would be fine, and Ms. Brinly agreed.
Senator Kelly mentioned a meeting that he had with a group of teachers from Bowling Green during the 2004 session of the General Assembly. He said they mentioned that the preparation to help students prepare for on-demand writing was different from the preparation in assisting students with writing portfolios, and instructional techniques were actually in conflict. Senator Kelly said he would like to see this issue addressed in the study.
Representative Moberly told Ms. Seiler the issue Senator Kelly was discussing fits into the practices and procedures required by KDE for integrating the portfolio into the day-to-day learning environment, which is part of a much larger problem about the inordinate amount of time that is taken on the writing portfolio. He said this plays into the larger question of whether the continuous assessment has been integrated into the day-to-day learning environment, or whether in some schools they still use extra time to prepare for the test. He said the implementation of the concepts of how we prepare for the test should be analyzed in the study.
Ms. Seiler said OEA will survey and conduct focus groups to determine the opinions of educators regarding the positive and negative effects of CATS, and the development of curriculum and instructional practices. She feels that Senator Kelly's and Representative Moberly's concerns will be addressed in these focus group meetings.
Senator Kelly said opportunity cost is another cost that needs to be weighed. This could include teacher time resources as another factor. Is the writing portfolio truly being integrated in the day-to-day curriculum, or is there an opportunity cost being paid that was unintended. Ms. Seiler said the focus group opinions and survey results will be merged into the cost review to get the type of information that Senator Kelly is requesting.
Representative Draud asked Ms. Seiler if there was an estimate of the cost of the study. Ms. Seiler said she did not have an estimate right now. She is compiling literature reviews, Request for Proposals (RFP's), and focus group information to try to get an estimate.
Representative Moberly asked the members to review the memo from Ms. Seiler that outlines the information necessary for hiring consultants. Ms. Seiler discussed the fact that Senate Joint Resolution 156 sets an aggressive deadline for the completion date of the study by September 15, 2004. Senator Kelly discussed the timeline and said it is his hope that if OEA feels the timeline is not conducive to give this issue a meaningful review, that EAARS could provide enough flexibility to the timeline to make sure a thorough and beneficial study is completed. He said the study would need to be completed in sufficient time before the 2005 session to provide adequate time for developing legislative proposals if the conclusion is made that the General Assembly needs to make some changes in the current assessment system.
Mr. Bobby Sherman, Executive Director, LRC, said it is a typical situation that if a committee determines that it needs an extension in order to make a study meaningful, it is given more time. Senator Kelly asked if a vote was needed to extend the timeline on the study, or if it could be approved just by voice vote of the committee. Representative Moberly said the September 15, 2004 deadline may be too quick for a thorough job to be given to this study. He asked Mr. Sherman if action needed to be taken at the meeting today. Mr. Sherman said he thought no action was needed today.
Representative Moberly asked Senator Williams if he wanted Ms. Seiler to hold off on any further work on the study until comments are obtained from staff at KDE. Senator Williams said Ms. Seiler should go ahead with her work, but if DOE staff identifies anything in the study that is glaring, or something that would be helpful in the review, staff should notify OEA and members of the EAARS committee. He said informal changes could probably be made since it is a workplan that does not necessarily have to be adopted by the entire committee.
Representative Moberly told Ms. Seiler to move forward according to the workplan that she has presented to the EAARS committee. He told Ms. Brinly that the committee wants the DOE staff to provide comments.
Representative Moberly referred back to Ms. Seiler's memo about the hiring of the consultants. She said she would like to issue two RFP's for personal service contracts for two consultants. The first one will do the extensive literature review, and the second contract will be for a consultant or consulting firm to conduct the regional focus groups.
Representative Moberly said a motion needs to be made by EAARS to the LRC to request the two RFP's. Representative Moberly said this request to the LRC for hiring of consultants is to help the OEA complete the study. Senator Kelly made the motion, and it was seconded by Senator Worley. The motion passed on a roll call vote of 8 years, 0 nays, and 0 passes.
With no further business before the committee, the meeting adjourned at 10:50 a.m.