Last week, I took the Star to task for its article about Pima County districts’ AzMERIT scores in the post, To Understand Pima County Test Scores, Follow the [Parents’] Money. (The post had more likes, shares and comments than most of my recent pieces, and a number of letters in the Star voiced similar criticisms, meaning the Star article bothered a lot of people.) Comparing Tucson Unified’s test scores with districts whose students come from more affluent homes where parents have more formal education makes little sense, I wrote. In Arizona, around the country and around the world, children from higher income families score higher on standardized tests than children from lower income families regardless of the quality of schools they attend.
I’ve written often that if you want to create a reasonable analysis of Tucson Unified’s AzMERIT scores, you have to compare them to scores in districts with similar demographics. Well, I’ve decided to put my keyboard where my mouth is. I’m beginning a rough study to see how test scores in Tucson Unified schools compare with scores of similar schools in similar districts. Why am I telling you what I’m planning to do even though I’ve only just finished the thinking process and haven’t begun the research? To keep myself honest, for one. If I put my approach in writing, I’ll be forced to stick to it and report the results as honestly as possible (which I’d try to do anyway, but it’s always tempting to fudge a bit). And to let readers know what my approach is before I write about my findings so you’re less likely to think I began with my conclusions and worked back to the data that “proved” what I already decided.
On my computer, I have two databases from the AZ Department of Education. One lists the total number of students in every district school in the state along with the percentage of students on free or reduced lunch. The other breaks down the 2017 AzMERIT scores of every school in detail, by gender, ethnicity, English Language proficiency and grade level. Looking at the two data sets, I can compare how schools with similar student bodies scored on the state tests, and I can compare the scores of subgroups in the schools.
Here’s my methodology. Scratch that. “Methodology” is to high fallutin’ a term for my crude analysis—I won’t be using any sophisticated statistical tools—of a blunt instrument—a high stakes test whose validity as a measure of student achievement is questionable. So, here’s what I’m gonna do.
I’m looking at districts which are similar to Tucson Unified in demographics, which means districts with a large percentage of low income and Hispanic students. I’m focusing on Southern Arizona districts, because I think it’s probable Hispanic students in districts farther north have been in the U.S. longer, which means they have a higher level of English proficiency than districts closer to the Mexico border and could have higher scores for that reason. Also, I’m focusing on larger districts, those with four or more elementary schools. That means along with Tucson Unified, I’ll be looking at schools in Sunnyside, Yuma, Nogales and Douglas — also Flowing Wells, though it has a larger white population than the other districts.
To make sure I’m comparing like with like as much as possible, I’ll only look at elementary schools, and I’ll group schools which have similar enrollment numbers and similar percentages of students on free/reduced lunch. I’ll create the groupings before I look at the test scores. The next step will be to compare the scores of schools in each of the groupings.
I honestly don’t know what I’m going to find. I expect the test scores of similar schools will be reasonably similar, though I know there will be variation. A few percentage points difference one way or the other isn’t very relevant, given my crude analysis of a blunt instrument, but if some schools are significantly above or below others, that could mean something. I also expect that most or all of the scores will be significantly lower than in schools in high income areas. I have no idea how Tucson Unified schools will fare in the comparison.
I’m not setting a timeline for myself, so don’t hold your breath waiting for the results. They’ll happen when they happen.
This article appears in Sep 14-20, 2017.
David, I don’t know what you learned in grad school, but the first thing I learned was a healthy skepticism about our ability to “know” (in the strict sense of the word) what the cause of any given effect may be. “Correlation DOES NOT prove causation” was drilled into us about as much as Strunk drilled “Omit needless words” into EB White and others.
You can compare test scores and break them down according to race, income, parental educational achievement, etc. until the cows come home and you will never be able to demonstrate anything other than that some of the factors you are examining are co-occurring variables. Perhaps one causes the other and perhaps two of them are caused by some third variable not examined in the study. Whatever the case, no causal relationships can be proved. Political writers frequently take advantage of the fact that many members of the general public can be tricked into assuming consistent correlation implies causation. It happens on both sides of the political divide. You’re always pulling correlations between poverty and low test scores out of your bag of tricks. Huppenthal regularly mentions correlation between increase in school choice policies in AZ and a drop in the juvenile murder rate.
Given your political affiliations, you no doubt hope to show in your forthcoming study that there is no statistically significant difference between the test score performance of districts serving the same kinds of populations. That proves nothing about what causes similarities in testing outcomes, and, more importantly, it is entirely irrelevant to the main thing worth examining, something that CAN be empirically studied and KNOWN a posteriori, but by means you do not like to employ, i.e. the tedious and maddening job of attending Board meetings, examining budgets, and submitting public records requests. What is that thing you can get at by days and weeks and years of the kind of tiresome work you do not like to do?
HOW THE MONEY ALLOCATED TO EACH DISTRICT IS BEING APPLIED. Is it being applied to benefit students and support teachers, or is it being applied to enrich already overpaid central administrators, lawyers, public relations consultants, and perhaps some private contractors like ESI? It’s a simple question, no doubt too simple for a sophisticate like you.
So enjoy noodling around on your computer in the comfort of your home office and best of wishes with the pending analysis of effects and attempts to imply that you are demonstrating something valid about causes. While you work on that, people who care more about improving services to students than they care about making excuses for irresponsible politicians will continue focusing their attention on the differences between how money is being applied (e.g., in Tucson “Unified” (?), on PR and “rebranding”) and what funding applications would be if the district’s budgeting were prioritizing giving students the basic raw materials needed if optimal learning is to take place (e.g., having adequate supplies, textbooks, and permanent, fully qualified teachers in every classroom).
(And please note that I did not say “give students the things that will cause them to learn optimally.” I said “give students the basic raw materials needed if optimal learning is to take place.” Some of us try to be cautious not to imply knowledge of causes that we know we can never have. We’d rather focus on advocating that students and teachers receive the support needed to do their respective jobs of teaching and learning as well as they possibly can, given the various circumstances we cannot or should not control. Public budgets, on the other hand, are things that the public can and should control or, at the very least, influence to the greatest extent possible, for the benefit of students enrolled in public schools.)
One of the things that is missed in this discussion is how the affluent, better educated parents got there. For example first generation Asians and Africans come here with nothing, yet their kids are among the highest scoring according to data I have seen at Cradle to Career. Then the second generation becomes affluent and better educated and legacy begins. My family was farmers and we had no one in my family that had a college degree until me. I served in the military and got the GI Bill. That’s what makes America great is opportunity not some egalitarian myth or utopian socialism. It is freedom and opportunity to climb the ladder and give your kids a better life than you have. That is what needs to be emphasized.
There still remain, Doug Martin, significant differences between the raw materials for learning available in different neighborhood schools in Tucson. When some schools have qualified teachers in every classroom, readily available textbooks and computers, and ample supplies (perhaps provided by parent fundraising or tax credits which are unavailable in other neighborhoods) and other schools are insufficiently supplied with the basic materials needed for student success to take place, there is not a level playing field. We are not giving every student regardless of income level the means by which they can pull themselves up by their own bootstraps, as you recommend. Some of them don’t have bootstraps in their schools.
The sad irony of commentators like Safier is that when they focus on the pervasive correlation (and implied causation) between poverty and low test scores, they distract the public from variables we CAN and SHOULD control. We absolutely can and should make sure schools serving lower SES populations have the staffing and supplies needed for student learning to take place. Funding levels are low, but in recent years poor decisions have been made about how to apply available funds and, as a result, conditions have worsened in some schools serving poor neighborhoods. In this context, to use a media platform to feed the public the line that “poverty causes low test scores” risks letting both the district administration and the public off the hook for taking a good hard look at the real conditions that need to change IN THE SCHOOLS (not in the homes, not in the parents’ income levels, which are much harder to affect) to give students what they need to succeed.
Safier remarks that his previous article on correlation between poverty and test scores was liked and shared more than any of his other recent pieces. I looked it up and saw over 700 likes. How many people have read it now and, as a result, are going to go around repeating the line we’ve heard so many times, “poor TUSD (or Tucson “Unified”) — of course they can’t do any better when the children in their schools are so disadvantaged.”
Imagine, David Safier, what positive changes might occur in local schools if you used your platform to build public pressure for positive change, instead of using it to let people who absolutely could be doing much better for the disadvantaged kids in their schools off the hook.
Doug, I don’t know your family’s situation, but I do know that all immigrants coming to the U.S. are not created equal from an economic or educational standpoint. Many of the immigrants from Asia come to the U.S. with college degrees, even advanced degrees. I don’t know if it’s still true, but at one point, Koreans were the most educated immigrants ever to set foot on these shores. So when you see an immigrant struggle to open a small market, for example, and build a family’s financial security from there, what you often don’t know is, that person came to this country with an MBA.
As for African immigrants, according to the Migration Policy Institute: “Compared to the total foreign-born population, sub-Saharan Africans were among the best educated immigrants as a group and were less likely to be Limited English Proficient (LEP). . . . Sub-Saharan immigrants have much higher educational attainment compared to the overall foreign- and native-born populations. In 2015, 39 percent of sub-Saharan Africans (ages 25 and over) had a bachelors degree or higher, compared to 29 percent of the total foreign-born population and 31 percent of the U.S.-born population. Nigerians and South Africans were the most highly educated, with 57 percent holding at least a bachelors degree, followed by Kenyans (44 percent), Ghanaians (40 percent), Liberians (32 percent), and Ethiopians (29 percent). Meanwhile, Somalis had the lowest educational attainment of all sub-Saharan Africans, with 11 percent having graduated from a four-year college.”
An Asian exception is found among immigrants from Hmong communities. Their groups don’t have much formal education in their home areas, and they struggle with school when they come to the U.S.
Mexican Americans who come to this country with little formal education are in a different category than immigrants who arrive with significant educational attainment.
So what do you think, David? Can Mexican Americans who come here with less educational attainment learn to the greatest extent possible, given their circumstances, in schools like Utterback in TUSD that do not have certified teachers in every classroom, are often lacking basic supplies like textbooks and paper? Or that (last school year, not sure whether the defect has now been remedied) had not used available desegregation funds to hire the social worker the desegregation plan required to work with families and help them learn how to provide support in the home for academic achievement?
Your commentary always avoids the concrete obstacles to achievement in our local schools that can absolutely be remedied by observation in the schools and consistent advocacy with the Board and administration. Why is that, David?
Don’t answer. We already know you have no good answer. “That’s not the kind of commentary that serves the interests of my political network” won’t get you 700+ likes and won’t get the arguments that excuse mismanagement more deeply entrenched in the minds of the public than they already are.
Sad.
I look forward to the results of Mr. Safier’s analysis.
Let me just advance, again, the thought I had last week. Mr. Safier observes, “In Arizona, around the country and around the world, children from higher income families score higher on standardized tests than children from lower income families regardless of the quality of schools they attend.”
Maybe this is because children from higher income families have higher IQs than children from lower income families, on average. We might expect this since IQ is correlated with a person’s income, and is substantially heritable. There’s an assumption underlying much of education policy that all children are blank slates that can be turned into geniuses if we just figure out the educational formula correctly. There’s very little data that would support this assumption.
There are top flight analysts in school districts replicating your analysis with powerful statistical software. Ed Sloan at Dysart is likely doing it to rank his district. Matt Strom at Chandler is another wizard. I first came across their work when in the Senate. Creighton, Alhambra, Mesa were the top three districts in the stae.
The student growth percentiles should perform the exact same function, wading out the demographic differences.
You enter a twilight zone effect when you compare the two results and observe huge differences in ranking outcomes.
I disagree with the comments above. You could see that the leadership and organization strength at Mesa, Alhambra and Creighton were clearly a cut above.
Quality schools do make a difference.
I think you will find Chandler at the top of the hill this time.
The notion that genetically inherited (not environmentally conditioned) lower IQs in poor populations CAUSE both lower test scores and poverty is not a new one, and it is no more demonstrable or valid than the notion that poverty CAUSES lower test scores. (Evidently Nathan K did not read the first post in this stream, or if he did, he did not understand the implications.)
No political faction’s self-serving and oversimplistic propaganda about the causative factors behind our social ills reflects the complexity of our actual social situation, in which it is obvious that some smart, hard working people are poor and some dumb and lazy people are wealthy, and some students succeed in school and in the work place in spite of growing up in poverty.
We cannot base SOUND public policy on dubious assumptions about causation. What we can and should do as a society in both our schools and our workplaces is provide everyone with the means to succeed, if they apply effort. Unfortunately, we have a lot of empirical, indisputable evidence that we are falling far short of that in our local schools, and we have shockingly few people locally in the media or among the citizenry who are willing and / or able to report on REAL CONDITIONS ON THE GROUND (not distributions of test scores in data bases) and comment in a way that can inform the public about how funding allocations can be improved in massive public districts serving lower-SES populations, so that conditions in the schools will begin to give all Tucson students the means to succeed if they choose to do so. The few reporters and citizens who try to provide this kind of commentary (thinking now of some of Huicochea and Steller’s pieces and of the activities of Putnam-Hidalgo, Fox, Morales, and Campoy) most often encounter blocks and / or disparagement, as politically motivated commenters on the right and the left continue to circulate flawed ideologies that both — not coincidentally — let public officials at the state and local level off the hook for improving services.
I hadn’t seen Huppenthal’s comment before I responded to Nathan K’s.
RE Huppenthal agreeing with the legitimacy of Safier’s methods: QED, with respect to much of what is said above about similarities between the (invalid) methods and assumptions of ideologues on both the left and the right.
This must be a proud moment for Safier, rivaled only by the fun moment in a previous comment stream when Matthew Ladner commended him on his laudable support of Prop 123.
It occurs to me that David should also look at school districts in California, which are demographically similar to Arizona districts (in terms of ethnic mix and family income), but in which per pupil spending is much higher. That would be an interesting comparison.
Yes, since the methods proposed already cannot get at what the public SHOULD be holding elected officials accountable for, by all means throw in a set of data generated by school systems hundreds of miles away about which we have no relevant knowledge of conditions on the ground.
Leaving aside the fact that they use different testing instruments, so no comparison to AZ Merit results can be made, there will be differences in the quality of teacher preparation programs available in publicly funded universities in CA and AZ, differences in teacher salary levels and benefits packages, a different legislative history relating to schools and school funding, different laws and policies about Spanish language instruction, different laws and policies on immigration and different local-level decisions about immigration enforcement, just to name a few of the variables that would make an AZ-CA comparison apples-to-rhinoceroses, not apples-to-apples.
Some education commenters give the impression that they would rather waste their time on any amateur at-home attempt to make “data analysis” into something that it is not rather than stepping into the difficult and unpleasant reality of what conditions are on the sites and in some of the public governance operations. I hadn’t checked the news on TUSD for some months, but after reading this BS blog piece and some of the BS comments in this stream, I read a few recent pieces in the Star and on the Three Sonorans (TSON) website. What a complete surprise: the TUSD Governing Board is a chaotic mess of discord, backbiting, mutual sabotage and mismanagement, as usual. Not at all what is needed to improve the quality of decision making about funding allocations and policy in the district.
You think poverty causes low test scores, David? Watch a TUSD Board meeting some time and see if you think the constructive management of a school system enrolling close to 50,000 students could ever come out of that ongoing train wreck. Instead of wasting days on data analysis, pick up the phone and call the Board members whose candidacies you endorsed. Give them some constructive advice about what it means to be a responsible public servant — if YOU know what it means, which I often doubt.
The long winded fuck strikes again.