The August 22, 2003 meeting of the Program Review and Investigations Committee was held at 10:00 AM in Room 131 of the Capitol Annex. Senator Katie Stine, co-chair, called the meeting to order, and the secretary called the roll.
Present were:
Members: Senator Katie Stine, Co-chair; Representative Charlie Hoffman, Co-chair; Senators Brett Guthrie, Paul Herron, Jr., David K. Karem, Vernie McGaha, and Dan Seum; Representatives Adrian Arnold, Sheldon Baugh, Dwight Butler, Jack Coleman, Ruth Ann Palumbo, Tanya Pullin, and Dottie Sims.
Guests: Commissioner Gene Wilhoit, Kentucky Department of Education; Tim Jackson, Deputy Secretary, Cabinet for Families and Children; Joel Griffith, Service Region Administrator for Northern Kentucky, Cabinet for Families and Children; Rose Pennington, Director of Professional Development Training, Cabinet for Families and Children; Pam Cherry, Adult Protective Services Supervision, Louisville, Kentucky; Richard G. Innes, Villa Hills, Kentucky; and Senator Westwood.
LRC Staff: Greg Hager, Committee Staff Administrator, Lowell Atchley, Lynn Aubrey, Kara Daniel, Tom Hewlett, Joseph Hood, Margaret Hurst, Erin McNees, Stacie Otto, Cindy Upton, and Susan Spoonamore, Committee Assistant.
Minutes of the July 10, 2003 meeting were approved without objection by voice vote upon motion made by Rep. Baugh and seconded by Rep. Arnold.
Sen. Stine stated that because of a recent incident in northern Kentucky resulting in the death of an elderly man, she asked Tim Jackson, Deputy Secretary, Cabinet for Families and Children to come before the Committee to discuss protections in place, coordination between the Cabinet’s services and state and local law enforcement, and training for the care and protection of vulnerable adults received by local law enforcement officials.
Dr. Jackson introduced Pam Cherry, Adult Protective Services Supervisor in Louisville, Kentucky; Joel Griffith, Service Region Administrator for Northern Kentucky; and Rose Pennington, Director of Professional Development and Training, Cabinet for Families and Children.
He explained that the Cabinet had worked hard over the last couple of years to raise the level of awareness about the growing problem of vulnerable adults, but there are better legal protections, resources, and awareness regarding the mistreatment of children. He explained that the Adult Protective Service network helps to provide services for elders, but that it is not an infrastructure. He stated that the Cabinet employs approximately 2000 permanent workers. Only 110 employees are dedicated APS social workers who work exclusively with adults including vulnerable elders, domestic violence victims, and any vulnerable adult over the age of 18 who has a handicapping condition. He said that last year the Cabinet received about 6,000 referrals and it was questionable whether the Cabinet had the personnel and resources to respond as it should. He also stated that nationally it is estimated that as high as 85 percent of incidents of elder abuse are never reported. He explained that the Cabinet had worked hard over the last year to help local communities establish local coordinating councils. He stated that the development of an infrastructure to protect elders was important to ensure everyone knew his job and to provide some means of accountability. He stated that the Cabinet’s role was to identify whether or not abuse, neglect or exploitation had occurred and to put in place a protective service plan. He also pointed out that law enforcement plays a critical role as well because when an older person is hurt, it is a crime. The perpetrator of that crime should be investigated and prosecuted if convicted. He explained that the Office of Inspector General has an important role to play because that office licenses, regulates, and monitors long-term and community care facilities. He also stated that the Long-term Care Ombudsman investigates resident complaints in nursing facilities and long-term care facilities. He stated that in Kentucky, over 90 percent of people over the age of 65 are not in nursing homes. He stated that elder abuse is under-reported and that the public is not as aware of the issue as it should be. He stated that it would take a commitment of more resources to ensure that the vulnerable are safe.
Pam Cherry, an Adult Protective Services Supervisor in Jefferson County, stated that for years they sent their copies of referrals to the police, but they did not know what happened to them. The police filed them because they did not know what to do with them. She stated that two years ago the police obtained a grant that was used to develop the Crimes Against Seniors Unit. After several informational meetings with police and other agencies, the County Attorney’s Office and the Commonwealth Attorney’s Office began to prosecute the crimes. She stated that because of the good working relationship between the agencies, the police would make home visits with them if necessary. She said that a bank asked her to train their personnel on the signs of exploitation. She said that after the training, she received a call from the bank’s security department because an elderly gentleman had requested an ATM card and had withdrawn a large sum of money. Ms. Cherry was able to get a police detective to accompany them to the bank, and was able to get the elderly gentleman’s family involved by putting safety measures in place. She said that working together had helped educate the community that elder abuse was a crime regardless of whether or not that person had family.
She stated that an Elder Abuse Board had been formed consisting of approximately 40 different agencies. The Board applied for a small grant through United Way which was used to develop a placement resource, known as the Elder Shelter Network. She stated that it had been a wonderful experience sharing resources and working together. She also stated that Jefferson County had done a great job of identifying the needs of vulnerable adults.
Joel Griffith with Adult Protective Services in Northern Kentucky, stated that protecting vulnerable adults involved commitment and collaboration of agencies in the community. He said that northern Kentucky could emulate the Jefferson County model program. He stated that northern Kentucky was more than willing and ready to increase awareness of elder maltreatment by recognizing the importance of collaboration between agencies and people in the local community. He stated that northern Kentucky has an Ethics Committee that focuses on adult protection issues, representing about 30 agencies, law enforcement, hospitals, home health, county attorney and all the agencies involved in the adult mental health system. They meet regularly to talk about issues affecting adults. He stated that the Ethics Committee had agreed to become the local coordinating council for issues concerning vulnerable adults. He stated that the Cabinet was providing the leadership for adult protection services in northern Kentucky’s three urban counties. He said that the Adult Protective Services staff are all specialized tenured staff who did nothing but adult protection in those three counties, but it is different for the rural counties.
Rose Pennington, Director of Professional Development and Training, Cabinet for Families and Children summarized the training system in place for employees of Adult Protective Services. She stated that the educational and training policy specified that each supervisor was required to ensure that employees had the correct amount of training to do their jobs. (A copy of Ms. Pennington’s informational handout regarding education and training policies can be found in the LRC Library file.)
Sen. Stine asked if training was conducted for staff of the Cabinet for Families and Children. Ms. Pennington stated that was correct.
Sen. Stine asked if law enforcement officials were offered any training so that they could recognize potential problems. Dr. Jackson stated that he did not know what kind of training existed for law enforcement agencies. He stated that the Cabinet had widely distributed a video to law enforcement agencies containing information on recognizing signs of abuse, neglect, and exploitation. He explained that the issue of adult abuse is not being as aggressively addressed in some rural areas as it is in the more urban areas, probably because of resource issues. He stated that a problem was that people perceive that police officers know more than they do and that police officers are as uninformed about elder abuse as the public.
Sen. Stine stated that in reading the statute on elder abuse, it actually says that any person who has reasonable knowledge of elder abuse shall report it.
Dr. Jackson stated that mandatory reporting is a part of state law.
Sen. Stine observed that mandatory reporting applies to ordinary citizens too. Dr. Jackson said that the Cabinet has an abuse hotline that is not just for children, but that it is to report abuse. The number is 1-800-752-6200.
Sen. Herron asked why something had not been done to prevent the abuse and murder of the gentleman in northern Kentucky, especially since the police had been called to the residence several times in the past. Dr. Jackson stated that he could not comment on any specific case.
Rep. Baugh asked Ms. Cherry if coordination in the Jefferson County/Louisville area included providing information to churches. Ms. Cherry stated that they take every opportunity to educate the public. They contact churches if any of the referred individuals are affiliated with a church.
Rep. Sims asked what type of funds were being used to pay for training of the Cabinet for Families and Children’s staff. She also stated that she was concerned with adult abuse in the rural areas not being recognized or reported. Dr. Jackson and Ms. Pennington stated that they thought that Title IV-E and Title IV-B money funded training.
Dr. Jackson stated that the Cabinet offers more training about child protection than it does about adult protective services. He stated that building an infrastructure not only requires reliable funding, stronger laws, and public awareness, but also a statewide network of people who were committed to adult issues.
Ms. Pennington stated that training is held in all regions and that each region receives training specific for that region.
Rep. Sims asked if training was provided in rural areas. Ms. Pennington stated that training was provided in rural areas and urban areas. She stated that each of the 16 regions had a regional training coordinator. She stated that the information she handed out to the committee included a detailed list of all training seminars.
Rep. Butler asked what recourse a bank had if an employee suspected that an elder was being deceived by a caretaker, a family friend, or a family member. Ms. Cherry stated that most banks were hesitant to turn over banking records and information because the bank was not aware that Kentucky is a mandatory reporting state. She said it was important to educate banks about their responsibility to report suspected abuse
Rep. Butler asked at what point charges could be filed. Ms. Cherry stated that law enforcement officials gather evidence and press charges.
Rep. Butler stated that he felt that seniors who had been taken advantage of were ashamed to report it and therefore it was being under-reported.
Sen. Stine agreed.
Sen. Karem asked Dr. Jackson why he would not comment on the case in northern Kentucky. Dr. Jackson stated that he could not comment because of confidentiality and because the tragedy had not been reported to the Cabinet.
Sen. Karem asked if the Cabinet were to find out that case workers or law enforcement officials had not properly performed their jobs, would the Cabinet be hesitant to tell the committee. He again asked why the Cabinet would not comment on the case in northern Kentucky. Dr. Jackson stated that if the Cabinet were to find out that kind of information he would not hesitate to say something to the committee, but if the committee asked specific questions regarding the case in northern Kentucky then he could not comment. He assured the Committee that the death of the gentleman would be reviewed.
Sen. Karem asked when the death occurred.
Sen. Stine stated that it had happened around the end of June.
Sen. Karem asked if the Cabinet had reviewed the case yet. Mr. Griffith stated that the Cabinet’s Adult Protective Services had not reviewed the case yet.
Sen. Karem asked why the Cabinet had not been looking at the case as a model case. Dr. Jackson stated that across the state there were many cases from which to choose, and those cases would be reviewed as part of an adult fatality review.
Sen. Karem asked the Cabinet to look into the case and point fingers if necessary.
He stated that in his opinion, the Cabinet was giving the impression that they were not going to point fingers or comment on the case, when they should be saying that the case was a tragedy, that they would be looking at the case tomorrow and if there were fingers to be pointed, whether it be at the Cabinet or other people, then the Cabinet would do what was right. Dr. Jackson stated that Sen. Karem was correct.
Sen. Seum asked what authority did the Cabinet use when asking a bank for various records. Ms. Cherry stated that under the adult protection law, the Cabinet has the authority to go to a bank, a hospital, or anywhere to obtain records needed to conduct an investigation.
Sen. Seum asked if anyone carried a badge for identity. Ms. Cherry stated that they did not have badges for identification. She said that once they explained the statute regarding elder abuse, then there usually was no problem. She stated that if a bank was not willing to give them the information, they would advise the bank to talk to their legal staff or the Cabinet’s legal staff.
Sen. Seum stated that a business owner might be confused as to whom he is talking. He asked if Adult Protective Services had subpoena powers. Ms. Cherry stated that they did not.
Sen. Seum asked if Adult Protective Services was responsible for investigating facilities that were not clean. Ms. Cherry stated that the issue of cleanliness would be referred to the Office of Inspector General because that would be a regulatory issue.
Sen. Seum asked if all staff were employees of the state. Ms. Cherry stated that they were.
Sen. Seum asked if Adult Protective Services covered state-run facilities. Ms. Cherry stated that they did.
Sen. Seum asked if a mentally retarded 40-year-old adult who resided in a state facility would be considered an elder. Dr. Jackson stated that he or she would not be considered an elder.
Ms. Cherry stated that anyone over the age of 18 who has a physical or mental dysfunction that prohibits them from protecting themselves would fall under Adult Protective Services.
Rep. Hoffman asked for a brief synopsis of the case in northern Kentucky.
Mr. Griffith stated that from reading about it in the newspaper, the elderly gentleman had a habit of letting people into his home. He said that there had been a couple of drug-related arrests or calls to the house involving people living in his home. He stated that the Cabinet had not been aware of the situation prior to the death.
Rep. Hoffman asked if an investigation was being conducted. Mr. Griffith stated that there was an ongoing criminal investigation into the homicide.
Sen. Stine stated that according to what she had read, other people had been living in the gentleman’s home and inviting visitors. Neighbors had called the police and the police had made arrests there. She said that one neighbor was quoted as having seen the man being pushed around. Eventually, the man was beaten to death. She stated that the neighbors had noticed things, but did not know what to do.
Mr. Griffith stated it was a tragedy that the neighbors and law enforcement officials never raised a red flag. He said he thought that it would be important to get the results of the criminal investigation.
Sen. Stine stated that the lack of public knowledge and law enforcement training was a real problem that needed to be addressed.
Sen. Herron asked if the son who took his mother out of the nursing home in Louisville, against medical advice could have contributed to the problem of his mother being neglected by the private caretaker. Ms. Cherry stated that she did not know the final outcome of the police investigation, but the son did return his mother to the nursing home.
Sen. Herron stated that the Cabinet for Families and Children needed to do more to protect elderly adults.
Rep. Arnold asked if the Committee would be exploring the issues of adult abuse at future meetings.
Sen. Stine stated that the issues of elder abuse should be added to the list of potential study topics.
Rep. Arnold stated that he was concerned about the privacy rights and privilege of an individual wanting to bring a person into his or her home. Dr. Jackson stated that Adult Protective Services were voluntary.
Sen. Stine stated that there is a need to balance the privacy of an individual but also protect the people who were being taken advantage of.
Sen. Stine introduced Erin McNees and Lynn Aubrey of Program Review staff for the presentation of the draft report “The Commonwealth Accountability Testing System”.
Ms. McNees explained that there was widespread criticism of the Kentucky Instructional Results Information System (KIRIS), the testing system implemented after enactment of the Kentucky Education Reform Act (KERA). She said that the 1998 General Assembly passed House Bill 53, replacing KIRIS with the present testing system. Under the CATS system, a student is classified as novice (lowest level), apprentice, proficient or distinguished (highest level). Ms. McNees stated that the Kentucky Board of Education has set a goal for school to have the average score of all its students be proficient level by 2014. She stated that the assessment components of CATS include two different types of tests. The Kentucky Core Content Test is a criterion-referenced test, which tests students on what they can do and what they know based on a standard of performance. She said the Comprehensive Test of Basic Skills (CTBS) is a norm-referenced test, which can be used to compare Kentucky students’ progress in basic reading, math, and language arts to students in other states. She explained that the test items used under CATS are multiple choice, open response, and the writing portfolio. She said Kentucky and Vermont are the only two states that include portfolios in their statewide assessment system, and Kentucky is the only state that holds schools accountable for the writing portfolio.
Ms. McNees explained that to comply with the No Child Left Behind Act (NCLB), additional tests would have to be added by the 2005-2006 school year. NCLB requires students to be tested annually in reading and math for grades 3 through 8, and that Kentucky would have to add a reading test for grades 5 and 8, and math test for grades 4 and 7. She stated that an important distinction between the systems was that NCLB requires accountability for the progress of subgroups: economically disadvantaged students, major ethnic and racial groups, students with disabilities, and students with limited English proficiency. She said if one of the subgroups failed to meet the target, the school as a whole would be deemed as needing improvement.
Ms. McNees stated because of the differences between the federal and state testing systems, the Kentucky Board of Education would be implementing a dual accountability system, which would allow the state to keep CATS intact. She explained that the dual system could result in some schools being eligible for rewards under CATS, but being classified as needs assistance under NCLB, or vice versa. She stated that schools would be judged on more than reading and math under CATS. She said that a school is classified as meeting its goal, progressing, or needing assistance, and that the non-academic factors—attendance rate and retention rates—are assessed at all grade levels. She said that the middle and high school levels are held accountable for the dropout rate, and if the schools are meeting their goals or progressing, they would be eligible for financial rewards.
Rep. Arnold asked if the goal lines and assistance lines were a part of No Child Left Behind. Ms. McNees stated that the goal and assistance lines were under CATS.
Ms. McNees stated under a two-year accountability cycle, each school eligible for rewards must have a dropout rate below 5.3 percent or a two-year rate that is 6 percent if the school has declined by at least one-half percent since the previous biennium. She said that staff had researched how dropout rates were reported and validated, and that KDE did not sufficiently verify or validate self-reported school and district dropout numbers. She explained that KDE used 16 different codes to differentiate types of withdrawals, and that any error among the codes could affect a school’s self-reported dropout rate, which could affect the receipt of reward money. She said that dropout audits performed in other states found that the actual dropout rates were consistently higher than the reported rates. She stated that Program Review staff recommended that KDE should verify school districts’ reported dropout statistics, review the schools’ documentation that students coded as transfers are enrolled in other schools, and that dropout statistics should be corrected to reflect any inaccuracies found in the audit. She said that staff also recommended that KDE should consider sanctioning schools that underreport dropout statistics.
Sen. Karem asked if there had been communication between staff and the Kentucky Department of Education, and was it staff’s position that KDE was absolutely not auditing the self-reported dropout rates. Ms. McNees stated that staff had been in communication with KDE. She said KDE checks anything that seems out of place. She explained auditors did check codes as part of the attendance audit to make sure that documentation was attached, but with transfer codes, KDE did not verify that students enrolled in other schools.
Sen. Karem asked if the report was saying that KDE does not even do random audits to check the accuracy of the dropout rate. Ms. McNees stated that was correct.
Ms. McNees stated that staff recommended that KDE should consider sanctioning schools that underreport dropout statistics by lowering their accountability index by an additional amount or by making them ineligible for rewards that year. She stated that staff also recommended a statewide student information system, which would allow KDE to monitor schools, track individual students as they moved through the system, and track the number of students excluded from CATS testing.
Ms. McNees stated that NCLB would require all states to report graduation rates annually. She said that the National Center for Education Statistics completion rate formula, which is highly dependent on the dropout rate, would be used by Kentucky and 19 other states to calculate high school graduation rates.
Sen. Stine asked if there were a lot of differences between using graduation rates and dropout rate. Ms. McNees stated that dropout rates only measure students who actually leave school and do not return. She said that graduation rates were based on the number of students who started 9th grade and actually graduated four years later with a high school diploma. She explained that the formula to be used by Kentucky to meet an NCLB requirement was reliant on the dropout rates, meaning that its accuracy will only be as good as the accuracy of the dropout rate numbers.
Sen. Stine asked if you would get different figures if you calculated how many students in the whole class dropped out starting when they were 9th graders, compared to looking at how many students dropped out this year. Ms. McNees said that the figures would be different.
Ms. McNees explained that Program Review staff calculated the statewide high-school graduation rate for the years 1997 to 2002 using the NCES formula, and the rate had increased each year to reach a high of about 81 percent in 2002.
Lynn Aubrey explained that staff had been asked to estimate the cost of the CATS assessment to the local school districts. She said that staff estimated the cost to be approximately $10.6 million dollars, or $16 per pupil for the 2000-2001 school year.
Ms. Aubrey said that staff estimated the cost to the state for CATS assessment to be approximately $10.3 million dollars, or $15.75 per pupil in fiscal year 2003. She stated that knowing the costs associated with CATS each year would be useful information for legislators, district officials, and the public, and therefore staff recommended that KDE develop procedures to measure districts’ annual assessment expenditures.
Ms. Aubrey explained that Program Review staff had surveyed teachers, principals and superintendents to elicit their views on various aspects of CATS. She said that 800 teachers, 500 principals and 100 superintendents responded to the survey. She said that superintendents and principals tended to be more positive than teachers about CATS. She said educators were asked if the quality of education was now better, about the same, or worse compared to the prior accountability system. She said that at least 70 percent of each group responded that the quality of education was now about the same or better.
Sen. Stine asked that the responses as to whether education was about the same or better under CATS not be grouped together for presentation because this gives a false impression. Ms. Aubrey explained that when asked how the CATS test affected the way teachers taught, more than two-thirds of the principals and superintendents responded that the CATS test affected teaching somewhat or very positively. She stated that only 41 percent of teachers agreed that the effect of the CATS test was positive, and approximately one-third of the teachers felt that the impact was negative.
Sen. Stine stated that in the draft report there was a sentence that read, “while it was the most common response by teachers, only 41 percent of them agreed that the effect of the CATS test on teaching was positive.” She asked if that meant that out of all the responses, that this response occurred most frequently. Ms. Aubrey stated that was correct.
Ms. Aubrey stated that the majority of educators responded that CATS positively affected what students learned, but teachers were less positive than the other educators. She stated that more than 60 percent of teachers, principals, and superintendents indicated that teachers do not have adequate time to cover the core content. She stated each group was asked if preparation for the test was affected by repetition of questions from year to year. She said that slightly fewer than half the principals and teachers and a majority of superintendents responded that repetition affected preparation. She said that when teachers were asked to explain that answers, most said that repetition helped them to prepare students for the format of test and to prepare students for the content that was stressed in the test. She stated the survey responses did not indicate that that teachers were actually teaching using questions from the test.
Sen. Stine asked if the reference to repetition was talking about the statement that said approximately 80 percent of the Kentucky Core Content Test questions are repeated from one year to the next, and that the Comprehensive Test of Basic Skills had not had questions changed since 1997. Ms. Aubrey stated that was correct.
Sen. Stine asked if there were concerns about the questions not being changed. Ms. Aubrey stated that there were concerns about teachers writing down the questions from the previous year and then teaching to those questions.
Sen. Stine asked if that were occurring, would it not contribute to an inflation of the scores over a period of time? Ms. Aubrey stated that it could be a possibility.
Ms. Aubrey continued by stating that teachers were the least optimistic of the three groups when asked whether or not their school could reach proficiency by 2014. She stated that only 21 percent felt that their school could reach the goal. She said that teachers frequently responded that it was impossible to reach that goal because not all students were capable. She also said that the teachers felt that the test did not measure student progress from year to year, and that student characteristics were not incorporated into the goals. She stated that more than 40 percent of principals and superintendents responded that their schools could reach the goal by 2014. She said that a large number of educators reported that they were not sure if their school could reach the goal. She said that when educators were asked about consequences to schools that failed to meet their improvement goals, such as not receiving monetary rewards were appropriate, more than 55 percent of teachers and principals felt that the consequences were inappropriate. About 55 percent of superintendents indicated the consequences were appropriate. She stated that the majority of principals and teachers whose schools are now receiving or had received assistance in the past indicated that the assistance was somewhat or very helpful. She stated that almost half of the teachers and 30 percent of principals and superintendents thought that the writing portfolios were weighted too heavily on the accountability index, and that about 75 percent of each group responded that weights for reading, math, science and social studies were about right. She said that many educators expressed the need for some student-level accountability.
Sen. Seum asked if 100 percent of the principals and superintendents received the staff survey. Ms. Aubrey stated that was correct, and a sample of about 2100 teachers received the survey.
Sen. Seum asked how many responded. Ms. Aubrey stated that staff received responses from about 800 teachers, which would be about a 38 percent response rate, and 500 principals responded, which would be a 40 percent response rate.
Sen. Seum asked if only 40 percent of the principals and 55 percent of the superintendents responded to the question of whether they agreed or disagreed that the consequences to schools that fail to improve are appropriate. Ms. Aubrey stated that was correct.
Sen. Seum asked if something should be read into those that chose not to respond. Ms. Aubrey explained that it is impossible to know for sure how nonrespondents would have answered questions. She elaborated that based on statistical analysis, it appears that those who responded are representative of all teachers, principals, and superintendents in Kentucky.
Sen. Seum stated that he was concerned why some chose not to respond to the survey. Ms. Aubrey explained that on the front cover of the survey, there was a confidentiality clause stating that identifying information would not be reported by staff.
Sen. Seum asked if the survey had been mailed. Ms. Aubrey stated that the superintendents and principals were emailed the web address of the survey form and that teachers received their surveys in the mail.
Sen. Seum asked if any responded to the survey by email. Ms. Aubrey stated that principals and superintendents responded by completing an online form.
Sen. Seum stated that he would be concerned about responding to a survey via computer.
Ms. Aubrey stated that staff had been asked to compare CATS test results to other tests. Due to unavailability of other test data, staff were only able to compare ACT scores and CATS scores. She explained that there were a few limitations that should be noted, including differences in the students taking the tests, differences in what the tests are designed to measure, potential differences in student motivation, and different test formats. She said it was necessary to match each student’s ACT score with his or her CATS score, and a student needed to have participated in all three years of high school testing on the CATS assessment to have scores for all components. She stated that approximately 41,000 students in the data set had both sets of scores. She stated by examining the ACT and CATS scores from students who took the ACT from 1999 through 2002, staff were able to examine trends in the two sets of scores for the same students. She said that staff found that the statewide ACT scores for this group of students had not increased, but the CAT scores for the students were increasing.
Sen. Karem asked if the report used the ACT scores from this year. Ms. Aubrey stated that those scores were not available at the time of the analysis.
Sen. Karem asked if staff had taken into consideration the growth of the number of people taking the ACT test. Ms. Aubrey stated that she would address that issue.
Sen. Stine stated that she wanted to know how many of those students taking the ACT were in private school versus public school. Ms. Aubrey stated that only public school students were used since they were the only students with CATS scores.
Ms. Aubrey stated that it was not possible to say for certain why there was a relatively strong correlation between CATS scores and ACT scores, and but statewide scores were not moving in a similar pattern. She said that several states had seen an increase in the number of students taking the ACT, and some of those states had seen their scores increase, but in about half of those states, scores remained flat or decreased.
Rep. Baugh asked if ACT scores were flat nationally or just for Kentucky. Ms. Aubrey stated that the ACT scores she was using was from the group of students that were used in this analysis.
Rep. Baugh asked how CATS scores have increased and ACT scores remained flat. Ms. Aubrey cautioned that the question cannot be answered with much certainty with available data. She explained that the data indicates that more students who are scoring novice based on the math, science, and social studies components of the CATS assessment are now taking the ACT. She said that this increase in the number of students who would likely perform poorly on the ACT could be affecting the statewide average score.
Rep. Baugh stated that he thought that the students taking the ACT were primarily those who planned to attend college. He asked if the novice students were the ones taking the ACT test. Ms. Aubrey said students who take the ACT are typically the ones going to a four-year college or university, but that there has been an increase in novice students in the data set compiled by Program Review staff.
Ms. Aubrey said that the National Technical Advisory Panel on Assessment and Accountability (NTAPAA) issued a statement explaining that there was substantial evidence supporting the validity and reliability of CATS and student performance on CATS constitutes a valid basis for rewarding or identifying schools that need to improve.
Sen. Stine asked if any members of NTAPAA were on contract with the state to help develop the test. Ms. Aubrey stated that she thought that was correct.
Sen. Stine asked if the statement of validity released by NTAPAA was an expression of all the members. Ms. Aubrey stated that the statement was released by the entire group of NTAPAA members and that all the members are on contract with the state.
Sen. Westwood asked if he was correct in that there was some question about whether or not CATS was measuring school improvement. Ms. Aubrey stated that was correct.
Sen. Westwood asked if the NTAPAA statement said that the CATS test was not a valid measurement device for an individual student. Ms. Aubrey stated that was correct.
Sen. Westwood asked if the report was comparing the ACT and CATS scores of individual students. Ms. Aubrey explained that she used the same formula that high schools used to calculate their school level academic index, but she removed the nonacademic factors and CTBS scores. She said student-level information was necessary to correlate CATS and ACT scores.
Sen. Westwood asked if she was satisfied that the results were valid and reliable after removing those elements. Ms. Aubrey stated that she was satisfied in that she applied the same amount of weight to each of the components that would be applied at the school level to get as close as possible to a student-level CATS score.
Sen. Westwood asked if she considered that under CATS accommodations are given to students with certain academic deficiencies. Ms. Aubrey replied that reported scores were used.
Ms. Aubrey next discussed the writing portfolio and the on-demand writing components of the CATS assessment. Both are assessed in grades 4, 7, and 12 and award students one of four possible scores: novice, apprentice, proficient or distinguished. These student-level scores are then used to calculate the school’s writing index. The writing portfolio makes up about 80 percent of a school’s writing index with the on-demand component making up the other 20 percent. Writing portfolios in the 4th grade contain four writing pieces and in 7th and 8th grade they contain five.
She stated that according to the survey, 71 percent of teachers reported that the time it takes to prepare portfolios was not appropriate to the benefit that the students receive from doing them.
Rep. Coleman ask for clarification of the response rates to the surveys. Ms. Aubrey replied that the response rate for superintendents was about 70 percent and the rates for teachers and principals were about 40 percent.
Ms. Aubrey said that 48 percent of teachers, 31 percent of principals and 38 percent of superintendents indicated that the writing portfolios were weighted too highly on the accountability index. She said that the scores for writing portfolios were higher than on-demand scores for all grades, with apprentice being the most common score for both assessments.
Ms. Aubrey explained that staff compared students’ writing portfolio and on-demand writing scores over the past four years. Correlations between the two assessments were consistent across the grade levels—about .40 to .45. She stated that this is not a particularly strong relationship, possible because these are two different types of writing, students do not receive feedback for on-demand writing, and the two assessments are scored differently.
Ms. Aubrey said that KDE utilizes a trainer of trainers model to provide portfolio scoring training. She stated that KDE trains the regional writing consultants, who in term train the cluster leaders, who then train the scorers of portfolios. KDE requires that all scorers receive a minimum of three hours of training for the year in which they score portfolios, with six hours recommended. Teachers responding to the survey indicated that they received about 4.5 hours of training per year, on average, and 90 percent of scorers indicated the training received from cluster leader has been helpful.
Ms. Aubrey said that almost half the surveyed teachers responded that teachers have biases that affect scoring. A majority of principals reported that teachers do not have biases that affecting scoring. She said that according to the responses from teachers and principals, slightly more than 30 percent of teachers are grading their own students’ portfolios. She said that when teachers were asked how KDE could improve writing portfolio scoring training, the responses volunteered most often were that training should be required of all teachers in the school, that the benchmark portfolio should be changed, and that there should be more opportunity for teachers to practice scoring. Ms. Aubrey said that the report recommended that KDE work with schools and school districts to reduce as much as possible the practice of teachers scoring their own students portfolios, survey teachers to determine how their portfolio scoring training could be improved, regularly replace benchmark and exemplar portfolios, and encourage schools to provide more opportunities for teachers to practice scoring throughout the year.
Ms. Aubrey stated that each school decides which scoring option—out of six possible methods—it will use to score portfolios. She said the most popular option is double blind scoring, and that surveyed teachers reported that conflicting scores are assigned to portfolios initially about 18 percent of the time.
Ms. Aubrey explained that the portfolio audit is a review of local scoring accuracy and is used to monitor statewide scores as well as to correct inaccurate scores. There are two samples of schools that are chosen for the audit. She said the purposeful audit is of schools with the largest differences in their predicted and actual portfolio scores, and the a random audit is for the remaining schools. She said KDE audits about 100 schools per year, and that KDE regulations specify that a school cannot be included in the purposeful audit for two years in a row. She said the CATS scoring contractor, CTB McGraw-Hill, performs the audits, with scores assigned to portfolios during the audit replacing the original scores and the school’s writing index recalculated accordingly. She said that audit results for the 1999-2000 to 2001-2002 school years indicate that original and audited portfolio scores did not agree 20 to 40 percent of the time and typically the school’s original score was higher. She said it did not appear that the agreement rate between original and audited scores has been increasing.
Ms. Aubrey explained that the report recommended that KDE consider implementing a system to track the performance of portfolio scorers, establishing consequences for schools that have low portfolio agreement rates, reauditing schools that have had a high number of inaccuracies the prior year, and increasing the number of schools that are randomly selected for the audit.
Sen. Stine said that the report noted that a statewide audit had been conducted in Texas by the state auditor, which found that the actual dropout rate was twice the reported rate. She asked if staff had recommended that such an audit be done in Kentucky. Ms. McNees replied that the Texas audit was initiated by a lawsuit of individuals based on allegations of fraud. She said the report did not recommend action by the auditor and that the student information system should better track students.
Sen. Karem stated that the most disturbing thing in the report is the dropout issue. He said the legislature has been quite vocal about this issue, and we cannot take comfort in numbers if there is a potential that they are inaccurate.
Sen. Seum asked if dropouts are tracked differently now than in 1990. Ms. McNees said that she did not know for sure.
Sen. Seum suggested that Kentucky return to the 1990 system and apply those numbers today. I did not think you could, but I thought I would just throw it out there. So my suggestion would be that we just go back to the 1990 system and apply those numbers to today.
Sen. Westwood said that he was pleased to see the recommendation that if a school gets a bad audit, KDE needs to reaudit following year. He asked why the rule was implemented that a school could not be audited two years in a row. Ms. Aubrey replied that there is a chance that a school could be selected for the random audit, but that the probability of this happening is very low.
Sen. Westwood encouraged the department to change the policy.
Sen. Stine asked that the fact that NTAPAA is under state contract be noted in the final report.
Rep. Pullin asked if schools are penalized if their dropout rate is over a certain rate. Ms. McNees said this was correct.
Rep. Pullin asked if expulsions count as dropouts. Ms. McNees replied that an expulsion who is receiving some sort of educational service would not be considered a dropout, but that an expelled student not receiving educational services would be considered a dropout.
Rep. Pullin asked if a school’s assessment is penalized for the number of novices the school has. Ms. McNees said this was correct.
Rep. Pullin asked if there was any evidence that it is more advantageous to a school to allow a student to drop out than to allow a student to be tested at the novice level. Ms. McNees said that she did not know for sure.
Rep. Coleman commented that policy on measuring school’s dropout rates seems out of date considering the creation of alternative schools and the number of students pursuing GEDs.
Sen. McGaha asked if it is possible for a school to remain between the goal and assistance lines until 2014. Ms. McNees said this was correct.
Sen. McGaha asked for confirmation that a school can receive rewards but not meet the goal of proficiency by 2014, but also not be classified as needing assistance. Ms. McNees said this was correct.
Sen. McGaha said that, based on his experience, it takes a long time for some students to transfer. He questioned the recommendations to sanction schools for poor reporting of dropout statistics and for poor portfolio scoring agreement rates. Based on his experience, he said that a 20 percent disagreement rate is not bad. He also called attention to the report’s finding that 38 percent of schools have substitutes going in—3.5 substitutes for an average of 1.4 days of instruction while they are undergoing portfolio training and 56 percent of schools have 5.1 substitutes for 1.4 days of instruction for scoring. He said this disrupts continuity in education and that substitutes are not the same as regular teachers.
Sen. Stine welcomed Commissioner Wilhoit.
Commissioner Wilhoit said he appreciated the opportunity to appear before the committee and that he thought the report was fair. He said the report would be taken seriously and KDE will act accordingly. He said we are not where we need to be in terms of understanding the role of assessment in education in the state based on the teacher responses and the discussion at this meeting. He said that an assessment system like this should be a tool that teachers can use to assess student learning, to adjust what they have been doing in the classroom, to set higher expectations, and then to help those students reach higher levels. He expressed concern that a lot of teachers still perceive the assessment as being outside that critical cycle. He said it is clear from this that our teachers need additional support, particularly for low achieving students. He said the most notable teacher response to him was that many teachers are not sure they can reach the goal. He observed that there is more work to be done with the writing portfolio program. He said that portfolios are supposed to be part of the instructional process and that writing should occur across the curriculum. He said the report provides further evidence that teachers are often treating writing separately. He said that most teachers still believe that the writing portfolio is a highly valuable exercise. He said that many of the data issues and recommendations that are made in the report can be carried out and KDE will do so for dropout data and by creating a separate CATS assessment code. He said it will be difficult to capture all costs though and it is questionable whether this should be mandatory.
Sen. Karem asked Commissioner Wilhoit whether school districts are padding the numbers and what KDE does to check the numbers. Commissioner Wilhoit responded that he does not believe districts padded dropout numbers. He said that KDE does attendance audits but does not track students who are reported. He said tracking students across districts is very labor intensive and would be done by KDE if it appeared likely the reporting system was misused. He said that the General Assembly and the department have stressed the importance of the dropout rate.
Sen. Karem said that he assumed that the commissioner would welcome the state auditor coming in and lending assistance. He made a motion that the committee communicate to the State Auditor that an independent review of dropout rates is needed
Sen. Stine noted this is not an assertion that there is any fraud or abuse.
Sen. Karem agreed.
Motion made by Sen. Karem and seconded by Rep. Baugh that the committee communicate to the State Auditor that an independent review of dropouts rates was needed, was approved by roll call vote.
Sen. Seum asked the Commissioner if elimination of the CATS test would end KERA. Commissioner Wilhoit replied that CATS is a single component but it is essential that there be some barometer for measuring student achievement and holding schools accountable. He said that adjustments made in 1998 improved the system and that they should continue to improve it.
Rep. Arnold asked about potential conflicts between KERA and No Child Left Behind. Commissioner Wilhoit said that the federal government requires more testing and that districts are going to be under more scrutiny for subpopulation performance. He said that a school could be meeting its performance goal and performing at a high level under CATS, but failing to made adequate federal progress. He said this will result in some public confusion.
Sen. Stine noted that there was one member of the public who wished to be heard on this issue.
Richard Innes explained that he had recalculated the cost estimates of assessment to be about $45 per student tested. He said there is confusion in the state about ACT scores because when the scores are initially released, there is only a state score for all school systems: public, private, and home school. He said that based on data broken down by public and nonpublic school students, portfolio scores have increased and writing scores have risen consistently since KERA began, but ACT English scores for public school students have decreased.
Rep. Hoffman asked Mr. Innes to explain his background. Mr. Innes replied that he is not a professional educator but has been familiar for decades with the education theories Kentucky is implementing and is noted for his expertise after years of intensive study.
Mr. Innes noted that only about half the superintendents think that education is better under CATS, which is not an expression of good performance. He noted that CTBS-5 questions have not changed since 1997. He said that if questions are not changed over a reasonable interval of time, inflation of scores will occur. He said that the only testing that occurs in third grade, sixth grade and ninth grade is CTBS. He said that Kentucky’s CTBS scores are highly questionable because the same questions have been used so long. He said there has been no discernable improvement in correlation between on-demand and writing portfolio scores since this program began in 1992 to 1994. He said writing portfolios are a good instructional tool but a bad assessment tool. He said there was a technical error in the draft of the LRC report that he received and that state-level ACT scores are available. He said he agreed with what has been said about graduation rates and dropout rates. He said dropout rates are not reported accurately anywhere. He said the graduation rate is a better measure. He said the percentage of ninth graders who graduate four years later peaked in at 72.7 percent. He said the rate constantly declined until very recently. He said new data indicated that only 66 percent of the students who were enrolled in ninth grade graduated as part of the class of 2002.
Mr. Innes said that he was relieved to see the correct answer to a sample test question included on page seven of the draft report because he had discovered this is not always the case. He said that the sample question was fifth-grade level in Kentucky, but would be third-grade level in Virginia.
Sen. Seum asked for confirmation that Mr. Innes had said that the state dropout rate was eight percent. Mr. Innes said the figure of about 8.5 percent is based on a rough conversion from the graduation rate.
Sen. Seum asked whether this meant that some areas of the state could have 20 percent dropout rates. Mr. Innes said this was possible.
Rep. Palumbo asked staff to clarify the issue of statewide ACT scores. Ms. Aubrey replied that the report did not say that statewide ACT scores were unavailable. The report said that by including only students who took the ACT and CATS test, then any comparisons would be for the same students, which would not be true if statewide ACT scores were used.
Rep. Baugh said that another sample seemed complicated for a fifth grader. Mr. Innes said that it was an arcane question given how few questions are on the test. He also discussed other errors he had found on past tests.
Sen. Stine said that she would entertain a motion to accept the study, with clarifications to be made in the final report of the graph on page 36 and assessment costs per tested student.
The Commonwealth Accountability Testing System report and its recommendations were adopted by roll call vote, upon motion made by Sen. Karem and seconded by Rep. Baugh.
Sen. Stine asked members to provide staff with their top 10 of the suggested topics, which includes adult protective services. She said the next two meetings will be devoted to follow-up studies of past reports. She said that the co-chairs had discussed asking the committee to approve three topics at today’s meeting: the brokerage system for Medicaid transportation, improper usage of state computers, and adult protective services.
Rep. Hoffman asked that staff prepare summaries of the enterprise zone and campaign finance studies for the next meeting. He said that if the will of the body is to act on those reports or pass over them, that will be addressed.
Sen. Stine said that the next meeting, which be chaired by Rep. Hoffman, will be Thursday, September 11 at 1:00 o’clock in the afternoon. She asked if there was a motion to initiate the three studies the co-chairs had proposed.
Motion made by Rep. Baugh and seconded by Rep. Pullin to begin work on the Brokerage System for Medicaid Transportation study topic; Improper Usage of State Computers study topic and Improved Coordination of Adult Protective Services study topic was approved by roll call vote.
Rep. Palumbo asked that both sides be present for discussion of the enterprise zone study.
Rep. Hoffman said that it is important that we hear both sides and also that we clean up loose ends before proceeding.
Rep. Palumbo asked for confirmation that more information was supposed to be gathered for the campaign finance report.
Dr. Hager replied that a revised report would include updated data on party spending in states.
Meeting adjourned at 1:25 p.m.