The Education Gadfly Weekly

Jay P. Greene, Center for Civic Innovation at the Manhattan Institute
January 2002

Manhattan Institute senior fellow Jay P. Greene has just released the second edition of his state-by-state "freedom index." It ranks the states according to their levels of "education freedom" as measured by the availability of four kinds of education choices for families: charter schools, subsidized private schools, home-schooling and public-school choice. He then relates the extent of a state's educational freedom to its student achievement (as measured on the National Assessment of Educational Progress, using states' 8th grade math scores in 2000). While Greene does not prove (or claim to prove) that more educational freedom boosts academic achievement, he shows a "strong observable relationship" between them. He also examines the strength of a state's accountability system (using publicly available data) and relates it to academic achievement in a way that controls for prior test scores. As for education freedom itself, Greene finds that Arizona has the most (2.94 on his index), followed by New Jersey, Delaware, Florida and Minnesota. Hawaii has the least (0.88), with Utah, Rhode Island, West Virginia and Maryland just ahead of it. Vermont and Ohio are smack in the middle. You can obtain this 14-page report from the Manhattan Institute's Center for Civic Innovation at http://www.manhattan-institute.org/html/cr_24.htm.

Neal McCluskey, Center for Education Reform
January 2002

For historians of education, 1991 will stand out as the year the world of education was turned upside down. As is often the case with revolutions, it is only through the perspective of time that we can really appreciate just how dramatic these happenings really were. In 1991, computer programmers at the University of Minnesota came up with the Internet protocol "gopher," and thus were born the ubiquitous Internet and World Wide Web that are now part of our daily lives. That same year, the Minnesota State Legislature passed the country's first charter school law. A decade later, we can see that these seemingly disconnected events have spawned an educational movement that has the power to deliver high quality instructional services to all learners regardless of time, place, and personal background. Education is no longer synonymous with the neighborhood school. In his short report "Beyond Brick and Mortar," Neal McCluskey, a policy analyst for the Center for Education Reform, tracks this history and what it means for today's students, parents, and the education establishment. He notes that there are now 30 "cyber charter schools" operating in 12 states. Those who have benefited most from these on-line schools are home-schoolers and children unable to attend traditional schools because of physical or emotional disabilities. Those who feel most threatened by them are the people who run school districts made up of buildings, buses, teachers and administrators. They see these cyber charters as taking their pupils and their money, and they don't much like it. Hence they've been struggling to scuttle cyber charters through legislative and legal action. In the spring of 2001, the Pennsylvania School Boards Association (PSBA) filed a lawsuit challenging the requirement that districts release funds for their students to enroll in cyber charters. In May a judged ruled against the PSBA, and the Keystone State has seen a boom in its cyber charter population. Meanwhile a related case is heating up in Ohio. While the outcome of the larger battle cannot be foretold with confidence, it is clear that change is underway. For more information, see http://edreform.com/press/2002/cyber_charters.htm, or order a hard copy of the report for $3 by calling CER at 202-822-9000 or visiting http://www.edreform.com/pubs/index.html.

Gary Miron and Christopher Nelson, National Center for the Study of Privatization in Education, Teachers College, Columbia University
December 2001

The National Center for the Study of Privatization in Education at Teachers College, Columbia University, continues to churn out "occasional papers" of varying quality and significance. This one is #41 in the series and was prepared by Gary Miron and Christopher Nelson of Western Michigan University's Evaluation Center. It seeks to compile extant data on student achievement in charter schools-a worthy project if an inconclusive paper, due largely to the spottiness of available data. In 33 pages, they do a decent job of identifying "independent" studies of student achievement in charter schools in those states where such studies have been conducted in methodologically defensible ways (just 8 states and a total of 18 studies). The meta-analysis of those studies leads them to conclude that charters currently present a "mixed or very slightly positive picture" with respect to pupil achievement. Mostly, though, they lament the paucity of decent data and sound studies. They provide some interesting theories about why so little is known and end up with a strong-and nearly irrefutable-plea for more research to be done. This is a good paper to know about, if only to answer the many people who innocently ask "Well, are charter schools working?" Judged by the single criterion of student achievement gains, the answer seems to be "The news is slightly positive but basically inconclusive for the nation as a whole." (Some of the individual state studies are more conclusive about gains-or the absence thereof-associated with their charter schools.) You can view a PDF version or request an emailed copy at http://www.ncspe.org.

Pedro Reyes and Joy C. Phillips, University of Texas at Austin
August 2001

A lot is going on in Houston by way of school reform and there's plenty of interest in whether various initiatives are succeeding. A number of reform efforts are loosely clustered in the Houston Annenberg Challenge, which has been underway since 1997 with substantial ($60 million) five-year funding from Annenberg and local matchers. 88 schools are being directly supported in Houston and five smaller nearby districts. A number of other ventures (e.g. professional development, institutes for teachers) are also being underwritten by the Annenberg Challenge, and some of these have already grown into larger initiatives with support from elsewhere. To its credit, the Houston Annenberg program invited a research team led by the University of Texas's Pedro Reyes to conduct a "formative" evaluation. This report-dated August 2001, issued in December-is called the "year two summary report" but in fact it reviews the Houston Annenberg program through its 4th year (2000-2001). The researchers claim to have found laudable progress, both the soft kind (e.g. teacher satisfaction and parent involvement) and measurable test-score growth by Annenberg-funded schools. The press release says "Our Year Two research finds that Annenberg-funded schools have made progress-in the case of Beacon schools quite considerable progress-raising achievement levels for their students" and "Minority student are making even bigger gains." ("Beacon" schools are one of three subsets of participating schools.) The problem is this: the Annenberg schools were hand-picked and, in the case of the praised "Beacon" schools-11 such, in 5 districts, that entered the program first and were concluding their 4th year when this study was done-they were picked "because they have already demonstrated the capacity to engage in school reform." One might even say they were cherry-picked. This report contains reasonably strong achievement data emanating from the Beacon schools, but it also shows that, for the most part, they were doing better than the Houston average when they entered the program-and were still doing better, by about the same margins, 3 years later. For example, in middle school math, the Beacon Annenberg schools (the reader is not told how many of these there are, nor how many are in Houston proper) surpassed the HISD average on the statewide TAAS test by 8 points in 1997 and 7 points in 2001. The Beacon schools made greater gains than HISD in high school reading-but the other two categories of Annenberg high schools slipped in reading between 1997 and 2001 (as did HISD as a whole). As for gap-closing, two of the three Annenberg school categories made worthy gains for minority and poor kids (relative to white and middle-class youngsters) but here we're given no comparison data for the district as a whole. In general, it's really difficult to know for sure what to make of the Annenberg schools' progress. The schools were hand-picked. They were doing relatively better than the citywide average when they entered the program. There's no real control group other than the district as a whole. And much else was happening in Houston during this period of time, so we can scarcely tell what was caused by Annenberg and what may have been shaped by other influences. My beef isn't with the program. It's with this approach to program evaluation. I do not doubt that the adults involved in the Houston Annenberg Challenge have positive vibes. But is the program the cause of their students learning more? This study doesn't really shed much light on that. If you'd like to see it, surf to http://www.utexas.edu/projects/annenberg/index.html. Hard copies may be ordered by contacting The Houston Annenberg Challenge, First City Tower, 1001 Fannin, Suite 2210, Houston, TX 77002-6709; 713-658-1881.

James Catterall and Richard Chapleau, National Center for the Study of Privatization in Education, Teachers College, Columbia University
December 2001

In yet another "occasional paper" (#42) from the Teachers College center on education privatization, UCLA professor James Catterall and Richard Chapleau of Chapman University analyze voting patterns in the celebrated California voucher referendum of November 2000. Everybody knows that the measure was thrashed. The authors here try to examine who voted for and against it and why. Note, though, that their analysis is limited to Los Angeles County (about 28 percent of the state population), where Proposition 38 had even less support than statewide (26.9 percent of those voting versus 29.4 percent). Los Angeles also has distinctive demographics and its own raft of education problems (and reform initiatives). The authors do not try to make it representative of California as a whole. Neither should readers of this paper. The (highly technical) analysis also suffers from a number of data limitations and analytic problems. The findings, therefore, should probably just be seen as suggestive. Still, some of them are interesting, if rather predictable. It appears that relatively wealthier voters were more apt to favor vouchers and lower income voters were likelier to oppose them. Republicans were also more pro-voucher than Democrats. Those with children already in private schools were more pro-voucher than those with kids in public schools. Nobody, it seems, had much information about what vouchers are. So people were voting-as no doubt they usually do-more on the basis of ideology or perceived self-interest than cool, rational analysis of information. Note, too, that better-informed people (at least in their own view) were also slightly likelier to favor vouchers. Readers of this study might also want to acquaint themselves with Terry Moe's excellent book-titled Schools, Vouchers, and the American Public-on the complex attitudes that Americans have with respect to school choice. To locate the Moe book, surf to http://www.brook.edu/press/books/schools_vouchers.htm. To see the NCSPE paper, surf to http://www.ncspe.org, where you can view a PDF or request an emailed copy.

Important education insights sometimes arise from developments in other fields.

This happened to me twice in recent weeks. Both episodes bear on results-based accountability, how it works, what can go awry-and what's wrong with the usual substitutes.

First, a new study of hospital accreditation looked into whether it makes any difference for the quality of patient care. Note that 95% of U.S. hospital beds are in health care institutions accredited by the Joint Commission on Hospital Accreditation of Healthcare Organizations. Researchers at the University of Michigan School of Public Health looked to see whether such accreditation is a good predictor of the safety and quality of health care, according to a January 14 Wall Street Journal article. (The study itself-a technically sophisticated bit of data analysis-appears in the winter 2001 issue of the journal Quality Management in Health Care.) After studying 700 hospitals, they found "that even hospitals with higher-than-average rates of deaths and complications receive favorable scores" from the Joint Commission. No doubt that's related to the fact that the "commission almost exclusively relies on surveying a hospital's structure and processes to determine whether to accredit a hospital, and doesn't give any weight to performance measures such as the number of deaths or unexpected complications or the ability to adapt to the latest treatments." In other words, the accreditors look at inputs, programs and activities, not results.

Predictably, the Joint Commission fought back, insisting-watch the nuanced words-"that accreditation assures patients that a hospital 'complies with a set of standards identified by health-care professionals as important things that lead to safety and quality and care.'"

Sound familiar? "Standards identified by professionals as important things that lead to" the desired results? Trust us experts. We oughtn't be judged by what actually happens to patients. Judge us and our institutions by whether they do the things that we professionals believe contribute to the desired results.

This calls to mind the debates about whether teacher-training programs must be accredited by NCATE (which some states require), whether public school teachers themselves need to be licensed (which every state requires), and whether teachers certified by the National Board for Professional Teaching Standards should get extra pay and status on grounds that they're more effective (which more states are offering). In all these cases, the essential question is whether an inputs-and-process heavy evaluation system is a reliable way to assure satisfactory results.

When critics charge that such hoops, hurdles, input controls and professional judgments do not correlate with actual institutional effectiveness or student achievement, they get told that education should be more like medicine or it will never be a true profession. How many times have I heard that "You wouldn't want your brain operated on by an unlicensed doctor; why let your child be taught by a non-certified teacher or one who went through an unaccredited preparation program?"

Well, now we know that accreditation doesn't amount to much in hospitals, either. And it isn't just a few isolated scholars saying this. Three years ago, the Inspector General of the federal Department of Health and Human Services also faulted the Joint Commission's accreditation system for "major deficiencies" and said its reviews are "unlikely to detect substandard patterns of care or individual practitioners with questionable skills."

Second insight: as the Enron swamp deepens, we see that neither the company, nor its accountants at Arthur Anderson, nor sundry government oversight bodies did an acceptable job of protecting the public interest (which includes the interests of company shareholders and employees). But what's the remedy? And how does it differ from our approach to failing schools? I addressed this topic in the 12/13/01 Gadfly but it's worth another comment. In business, Americans take for granted that government has little direct ownership and management role and that private firms are the main actors. They're free to run themselves as they think best but not to keep secrets about how they're doing. Our economy depends not just on private ownership but also on a well-informed marketplace. Firms that are publicly traded must disclose all manner of financial and other information. Outside auditors are meant to keep the company treasurer honest and the stockholders informed. The accounting profession sets guidelines for how things are reported. And the government monitors and, when necessary, polices all this.

That system usually works fine. The economy thrives. Capitalism flourishes. And nobody proposes that the government should run the companies. In the Enron case, however, it broke down. People lied. What was disclosed wasn't the whole story. The accountants were, evidently, complicit in the misbehavior. And government oversight bodies sat on their hands.

Recriminations are now flying, Congressional hearings are underway, politics is heating up and from all sides we hear a clamor for reform of the accounting profession and its standards. Indeed, that clamor shows how important accurate information is to our economy and what a challenge it is to design and sustain (and modernize) systems that ensure that the public has (and can trust) the information it needs. It's impossible to bar all chicanery but we can do our best to create systems and rules that lead to the best data possible and to develop checks and balances that make it difficult for anyone to conceal the truth from the marketplace for long. Nobody doubts that government has a proper role in closing loopholes in reporting requirements and accounting practices that allow companies to mislead the public. (If Enron's questionable bookkeeping methods had been prohibited, we would have known far earlier about the company's problems and a lot of misery might have been averted.) But nobody, to my knowledge, has suggested that government should itself run-or replace-the companies themselves.

In K-12 education, on the other hand, we don't assume independent operation of schools in a marketplace supplied by government-mandated information. Rather, we assume direct government operation of the schools themselves by state and local "systems." We settle for bureaucracy instead of public information. That's one reason we seldom have good data about what's going on in individual schools or what results they're producing. Despite its flaws, American capitalism provides investors, customers and outside watchdogs with tons more detailed information about companies than parents, teachers or students have about their schools. In education, these data have traditionally been held close. Instead, we've been told that government operation of the schools will look after the public interest. We don't need to know much about how they're doing because supposedly we can trust the government to run them properly.

Then a few exceptions are made, such as charter schools, home schools, private schools, and outsourced public schools run by private firms.

Those exceptions don't always work well, to be sure. I've been to my share of bad charter schools and the evidence is mixed on outsourcing. In education, though, people tend to assume that, if one of the exceptions isn't working, the remedy is to put government back in direct control of the situation, or at least to make more government rules apply to it. That's because education has no effective market to solve the problem. If it did, we'd demand full information-and expect government to help us get that information-then let the market work its will. An Enron-like school would lose customers, file for educational bankruptcy and probably shut down.

Perhaps the greatest virtue of the new federal No Child Left Behind law is its earnest efforts to force better information from the education system via school report cards, "adequate yearly progress" reports at the building level, the disaggregation of test results, data on how many of a school's teachers are "highly qualified" (and how many are teaching out-of-field, etc.) and other mechanisms. Taken as a whole, it makes a valiant start at getting education consumers (and practitioners) the kinds of data that investors and customers have long expected from the companies they deal with. In that way, No Child Left Behind begins to redefine government's role in education, not just as bureaucratic overseer (though that's still there, too) but also as a marketplace facilitator and honest broker of vital data.

What's still missing in education, of course, is a true marketplace. That's where the federal legislation wimps out. So we continue with an awkward hybrid, a basically bureaucratic accountability system onto which is heaped the kinds of consumer information that would be needed by a market-style accountability system. One hopes we'll get that sorted out in the years to come.

"Hospital's Accreditation Status Proves a Poor Predictor of Care, Study Says," by Barbara Martinez, The Wall Street Journal, January 14, 2002. (available only to subscribers)

As they flew back to Washington earlier this month after celebrating their joint education bill, the No Child Left Behind Act, President Bush and Senator Edward M. Kennedy held an extended conversation about the need to boost early childhood education, and that conversation may soon lead to legislation, according to reporter Anne Kornblut of The Boston Globe. Last week, First Lady Laura Bush appeared as a star witness on early childhood education at a hearing before the Senate Health, Education, Labor and Pensions Committee, which Kennedy chairs. Though many Democrats may not be thrilled about giving Bush another chance to claim bipartisanship, the President is expected to announce the details of a new proposal for early learning in the coming weeks. Improving early childhood education is expected to have payoffs for both regular education and special education; a report released by the National Research Council earlier this month urges the government to increase its emphasis on early childhood education as a way of reducing the number of minorities in special education, among other things. The report, "Minority Students In Special and Gifted Education," recommends that states adopt a screening and intervention strategy for children at risk of developing reading problems and that the federal government support this. The mixed quality of preschool experienced by children today was brought home by a study published by the Massachusetts Department of Education last week which found that 65 percent of preschools and day-care centers in the Bay State fail to provide effective programs to develop language and thinking skills in their students. For more, see "Bush, Kennedy work on preschool plan," by Anne Kornblut, The Boston Globe, January 22, 2002 (article must be purchased); "Minority Students In Special and Gifted Education," Committee on Minority Representation in Special Education, M. Suzanne Donovan and Christopher T. Cross, Editors, National Research Council; and "Study: Many preschool programs fail," by Ed Hayward, Boston Herald, January 19, 2002 (article must be purchased).

In an earlier report and Gadfly editorial-available at http://www.edexcellence.net/gadfly/issue.cfm?issue=75#1062-the Manhattan Institute's Jay Greene explained that official high school graduation rates published by the federal government understate the problem of dropouts because they treat the General Education Development (GED) credential as the equivalent of a standard high school diploma. But is the GED a good substitute for the real thing? In an article in City Journal, Greene notes that earning a GED brings few of the benefits of earning a high school diploma-economists have found the life outcomes of GED-holders to be no better than those of high school dropouts-and proceeds to explore why that might be. One reason is that passing the GED requires very little academic knowledge; the average GED recipient passes the test after just 30 hours of class time and study. Preparing for the GED also requires none of the social discipline that sticking it out in high school demands. Greene argues that treating the GED as the equivalent of a high school diploma not only distorts our dropout statistics, it may even contribute to the problem; the existence of an easier route to a credential may actually encourage students to drop out. Greene cites a study by the Urban Institute's Duncan Chaplin that found that the easier a state makes it to get a GED, the higher the dropout rate. To eliminate this problem, Greene suggests that we make the GED harder and raise the minimum age for taking the test. "GEDs Aren't Worth the Paper They're Printed On," by Jay P. Greene, City Journal, Winter 2002.

Gary Miron and Christopher Nelson, National Center for the Study of Privatization in Education, Teachers College, Columbia University
December 2001

The National Center for the Study of Privatization in Education at Teachers College, Columbia University, continues to churn out "occasional papers" of varying quality and significance. This one is #41 in the series and was prepared by Gary Miron and Christopher Nelson of Western Michigan University's Evaluation Center. It seeks to compile extant data on student achievement in charter schools-a worthy project if an inconclusive paper, due largely to the spottiness of available data. In 33 pages, they do a decent job of identifying "independent" studies of student achievement in charter schools in those states where such studies have been conducted in methodologically defensible ways (just 8 states and a total of 18 studies). The meta-analysis of those studies leads them to conclude that charters currently present a "mixed or very slightly positive picture" with respect to pupil achievement. Mostly, though, they lament the paucity of decent data and sound studies. They provide some interesting theories about why so little is known and end up with a strong-and nearly irrefutable-plea for more research to be done. This is a good paper to know about, if only to answer the many people who innocently ask "Well, are charter schools working?" Judged by the single criterion of student achievement gains, the answer seems to be "The news is slightly positive but basically inconclusive for the nation as a whole." (Some of the individual state studies are more conclusive about gains-or the absence thereof-associated with their charter schools.) You can view a PDF version or request an emailed copy at http://www.ncspe.org.

James Catterall and Richard Chapleau, National Center for the Study of Privatization in Education, Teachers College, Columbia University
December 2001

In yet another "occasional paper" (#42) from the Teachers College center on education privatization, UCLA professor James Catterall and Richard Chapleau of Chapman University analyze voting patterns in the celebrated California voucher referendum of November 2000. Everybody knows that the measure was thrashed. The authors here try to examine who voted for and against it and why. Note, though, that their analysis is limited to Los Angeles County (about 28 percent of the state population), where Proposition 38 had even less support than statewide (26.9 percent of those voting versus 29.4 percent). Los Angeles also has distinctive demographics and its own raft of education problems (and reform initiatives). The authors do not try to make it representative of California as a whole. Neither should readers of this paper. The (highly technical) analysis also suffers from a number of data limitations and analytic problems. The findings, therefore, should probably just be seen as suggestive. Still, some of them are interesting, if rather predictable. It appears that relatively wealthier voters were more apt to favor vouchers and lower income voters were likelier to oppose them. Republicans were also more pro-voucher than Democrats. Those with children already in private schools were more pro-voucher than those with kids in public schools. Nobody, it seems, had much information about what vouchers are. So people were voting-as no doubt they usually do-more on the basis of ideology or perceived self-interest than cool, rational analysis of information. Note, too, that better-informed people (at least in their own view) were also slightly likelier to favor vouchers. Readers of this study might also want to acquaint themselves with Terry Moe's excellent book-titled Schools, Vouchers, and the American Public-on the complex attitudes that Americans have with respect to school choice. To locate the Moe book, surf to http://www.brook.edu/press/books/schools_vouchers.htm. To see the NCSPE paper, surf to http://www.ncspe.org, where you can view a PDF or request an emailed copy.

Jay P. Greene, Center for Civic Innovation at the Manhattan Institute
January 2002

Manhattan Institute senior fellow Jay P. Greene has just released the second edition of his state-by-state "freedom index." It ranks the states according to their levels of "education freedom" as measured by the availability of four kinds of education choices for families: charter schools, subsidized private schools, home-schooling and public-school choice. He then relates the extent of a state's educational freedom to its student achievement (as measured on the National Assessment of Educational Progress, using states' 8th grade math scores in 2000). While Greene does not prove (or claim to prove) that more educational freedom boosts academic achievement, he shows a "strong observable relationship" between them. He also examines the strength of a state's accountability system (using publicly available data) and relates it to academic achievement in a way that controls for prior test scores. As for education freedom itself, Greene finds that Arizona has the most (2.94 on his index), followed by New Jersey, Delaware, Florida and Minnesota. Hawaii has the least (0.88), with Utah, Rhode Island, West Virginia and Maryland just ahead of it. Vermont and Ohio are smack in the middle. You can obtain this 14-page report from the Manhattan Institute's Center for Civic Innovation at http://www.manhattan-institute.org/html/cr_24.htm.

Neal McCluskey, Center for Education Reform
January 2002

For historians of education, 1991 will stand out as the year the world of education was turned upside down. As is often the case with revolutions, it is only through the perspective of time that we can really appreciate just how dramatic these happenings really were. In 1991, computer programmers at the University of Minnesota came up with the Internet protocol "gopher," and thus were born the ubiquitous Internet and World Wide Web that are now part of our daily lives. That same year, the Minnesota State Legislature passed the country's first charter school law. A decade later, we can see that these seemingly disconnected events have spawned an educational movement that has the power to deliver high quality instructional services to all learners regardless of time, place, and personal background. Education is no longer synonymous with the neighborhood school. In his short report "Beyond Brick and Mortar," Neal McCluskey, a policy analyst for the Center for Education Reform, tracks this history and what it means for today's students, parents, and the education establishment. He notes that there are now 30 "cyber charter schools" operating in 12 states. Those who have benefited most from these on-line schools are home-schoolers and children unable to attend traditional schools because of physical or emotional disabilities. Those who feel most threatened by them are the people who run school districts made up of buildings, buses, teachers and administrators. They see these cyber charters as taking their pupils and their money, and they don't much like it. Hence they've been struggling to scuttle cyber charters through legislative and legal action. In the spring of 2001, the Pennsylvania School Boards Association (PSBA) filed a lawsuit challenging the requirement that districts release funds for their students to enroll in cyber charters. In May a judged ruled against the PSBA, and the Keystone State has seen a boom in its cyber charter population. Meanwhile a related case is heating up in Ohio. While the outcome of the larger battle cannot be foretold with confidence, it is clear that change is underway. For more information, see http://edreform.com/press/2002/cyber_charters.htm, or order a hard copy of the report for $3 by calling CER at 202-822-9000 or visiting http://www.edreform.com/pubs/index.html.

Pedro Reyes and Joy C. Phillips, University of Texas at Austin
August 2001

A lot is going on in Houston by way of school reform and there's plenty of interest in whether various initiatives are succeeding. A number of reform efforts are loosely clustered in the Houston Annenberg Challenge, which has been underway since 1997 with substantial ($60 million) five-year funding from Annenberg and local matchers. 88 schools are being directly supported in Houston and five smaller nearby districts. A number of other ventures (e.g. professional development, institutes for teachers) are also being underwritten by the Annenberg Challenge, and some of these have already grown into larger initiatives with support from elsewhere. To its credit, the Houston Annenberg program invited a research team led by the University of Texas's Pedro Reyes to conduct a "formative" evaluation. This report-dated August 2001, issued in December-is called the "year two summary report" but in fact it reviews the Houston Annenberg program through its 4th year (2000-2001). The researchers claim to have found laudable progress, both the soft kind (e.g. teacher satisfaction and parent involvement) and measurable test-score growth by Annenberg-funded schools. The press release says "Our Year Two research finds that Annenberg-funded schools have made progress-in the case of Beacon schools quite considerable progress-raising achievement levels for their students" and "Minority student are making even bigger gains." ("Beacon" schools are one of three subsets of participating schools.) The problem is this: the Annenberg schools were hand-picked and, in the case of the praised "Beacon" schools-11 such, in 5 districts, that entered the program first and were concluding their 4th year when this study was done-they were picked "because they have already demonstrated the capacity to engage in school reform." One might even say they were cherry-picked. This report contains reasonably strong achievement data emanating from the Beacon schools, but it also shows that, for the most part, they were doing better than the Houston average when they entered the program-and were still doing better, by about the same margins, 3 years later. For example, in middle school math, the Beacon Annenberg schools (the reader is not told how many of these there are, nor how many are in Houston proper) surpassed the HISD average on the statewide TAAS test by 8 points in 1997 and 7 points in 2001. The Beacon schools made greater gains than HISD in high school reading-but the other two categories of Annenberg high schools slipped in reading between 1997 and 2001 (as did HISD as a whole). As for gap-closing, two of the three Annenberg school categories made worthy gains for minority and poor kids (relative to white and middle-class youngsters) but here we're given no comparison data for the district as a whole. In general, it's really difficult to know for sure what to make of the Annenberg schools' progress. The schools were hand-picked. They were doing relatively better than the citywide average when they entered the program. There's no real control group other than the district as a whole. And much else was happening in Houston during this period of time, so we can scarcely tell what was caused by Annenberg and what may have been shaped by other influences. My beef isn't with the program. It's with this approach to program evaluation. I do not doubt that the adults involved in the Houston Annenberg Challenge have positive vibes. But is the program the cause of their students learning more? This study doesn't really shed much light on that. If you'd like to see it, surf to http://www.utexas.edu/projects/annenberg/index.html. Hard copies may be ordered by contacting The Houston Annenberg Challenge, First City Tower, 1001 Fannin, Suite 2210, Houston, TX 77002-6709; 713-658-1881.