Petrilli and Pondiscio discuss the fallen NAEP scores, debate the meaning of Obama’s pledge to reduce testing, and ponder school dress codes. Amber takes a look at NAEP’s alignment with Common Core math.
Amber's Research Minute
Transcript
Mike Petrilli: Hello. This is your host Mike Petrilli, of the Thomas B. Fordham Institute. Here at the Education Gadfly Show, and online at edexcellence.net. Now please join me welcoming my co-host the DJ Silento of Education reform. Robert Pondiscio.
RobertPondiscio: Watch me whip, watch me NAEP NAEP?
Mike Petrilli: Watch me NAEP NAEP. That's right! We've seen it on Twitter. Watch me whip, watch me NAEP NAEP. Hey, my second grader loves this song, especially the Superman part. Watch me Superman. Yes, NAEP NAEP, how perfect. This is our special all NAEP addition of the Education Gadfly show Robert. It's NAEP day, should be a national holiday.
RobertPondiscio: It is a national holiday, if you work at the Fordham Institute.
Mike Petrilli: Yes, that's right.
RobertPondiscio: One or two other places, but we're wonky like that.
Mike Petrilli: We are wonky. Used to be a time there were only a few of us organizations out there analyzing the NAEP results. Some would argue committing mis-NAEP-ry, but now there's all kinds of groups involved, and more scrutiny than ever, so we have to stick to the facts Robert. This is not causal data here.
RobertPondiscio: Watch me wonk, watch me NAEP NAEP.
Mike Petrilli: Good. See impressive. This from a man who has been staying up late watching his Mets get beat.
RobertPondiscio: Oh you had to work that in there, didn't you Mike? Go ahead.
Mike Petrilli: Let's play Pardon the Gaff. Clara get us started.
Clara: What do we make of the new NAEP scores? Are the declines bad news for Ed reform?
Mike Petrilli: Robert? You going to try to sugar coat this one Robert?
RobertPondiscio: When have I ever sugar coated anything Michael? Can we get out of the business, please, of ... how do I want to put this ... Of changing our minds and over reacting to ever little twitching nerve impulse that comes out of the data. Come on, let's look at some long term trends. Here's the best thing I read this week, and I read this before NAEP came out.
Mike Petrilli: I did.
RobertPondiscio: A nice report. I can't remember the title of it. From the Urban Institute looking at 10 year trend datas. One of the biggest complaints of mis-NAEP-ry which you talked about. Is that why don't we look at the demographically corrected versions of NAEP, we never do. Well matching [inaudible 00:02:22] has made it easy for us. We don't ever have to have this conversation again, and to me everybody was talking about how Texas and Florida look a lot better. They're right up there with Massachusetts when you look at through this prism, but the thing that nobody's talking about is all 50 states, if you look at 10 year trends are improving. That to me is the big news, and that happened even before this came out.
Mike Petrilli: Well that did happen before. One of the things that makes these scores so disheartening is that in math especially, we have seen for almost 20 years now, nothing but up, up, up. It's kind of like the crime rates. We're used to them going down, down, down, and when we. Hold on. When we see a little blip, it makes us nervous. I totally agree.
RobertPondiscio: Move away from the window, Mike. It's going to be okay.
Mike Petrilli: We can not make any conclusions. We got to see what happens next time.
RobertPondiscio: Right.
Mike Petrilli: We certainly need more analysis, but look you'd rather see tests scores up then go down. You'd rather see crime stats go down then go up, and when you see these trends, perhaps take a wrong turn, it's something that makes you nervous. All I'm saying is, of course we can't draw conclusions, but let's also not sugar coat it. Hey let's be honest. Some of our friends, 2 years ago, were out there crowing about the fact that the District of Columbia and Tennessee saw big gains.
RobertPondiscio: Yup.
Mike Petrilli: Claiming it was because of their implementation of Common Core and Teacher Evaluations.
RobertPondiscio: I just read and interesting op-ed, and I can't remember where I read it. It was written by a guy, it was an Italian last name, like Petrillii? Something like that.
Mike Petrilli: Yup, yup, yup, yup. It's a good one.
RobertPondiscio: It was good. It was about how.
Mike Petrilli: That's my cousins.
RobertPondiscio: Was it.
Mike Petrilli: Yeah.
RobertPondiscio: Okay. It was really well written. It was written by another guy who wrote, another Italian name I think. Anyway, it was about how Common Core test results were showing. Don't shoot the messenger.
Mike Petrilli: Yup.
RobertPondiscio: They're going down because all these changes going on in classrooms. Why did we think that when we're asking teachers to do so much more, change their practices in the last couple of years. Why did we think we wouldn't see a down turn?
Mike Petrilli: Right.
You're parodying Secretary Duncan's line. He says that.
RobertPondiscio: I'm parodying nothing. I'm my own man.
Mike Petrilli: You're just an army Duncan lackey. That's fine, but look. There's an implementation dip. A well known phenomenon. That may be true. Nobody knows, but hey, we would rather see these scores going up. Let's hope that next time that is the case, and let the true wonk's dig in, do some more sophisticated analysis.
RobertPondiscio: If a frog had wings, Mike, it wouldn't bump its butt on the ground. Come on.
Mike Petrilli: Okay. I have no idea what that means Clara. Why don't we go onto Topic Number 2?
Clara: What's our take on President Obama's plan to reduce testing in schools.
RobertPondiscio: This is the one where you just try to irritate me, isn't it?
Mike Petrilli: Well you are just irritated, Robert, this week. Now on this one you're not an administration lackey.
RobertPondiscio: Oh my god.
Mike Petrilli: You actually seem to disagree with them on this.
RobertPondiscio: Vehemently. I was actually kind of ... I would say Obama and Secretary Duncan sort of ruined my weekend on this one. This is just nonsense. I wrote a piece in U.S. News about this, this week. This idea that you're going to reduce the number of tests. Let's count the number of ways in which this is silly.
1 - How many tests are federally mandated Mike?
Mike Petrilli: Well depends on how you count them. 3 to 8. Math and reading, plus science, it comes out to something like 24, I think.
RobertPondiscio: In any given year, 2? Let's be not disingenuous. The over test ... look I think is a real phenomenon, and we have to be mindful of that. That is not federally driven. By the way, did my brain fall out of my head?, or do Obama and Duncan have virtually nothing to say about the number of tests that schools, and states, and districts assign. Right?
Why are they even talking about this? The real mischief here is the effect of testing. I've been a classroom teacher. I'm still a classroom teacher. Let me tell you if you're not paying attention to the effects of testing, not the number of tests, but the fact that the test and prep culture we have materially changes the experience of schooling for our children. That's the problem. It's not the number of freaking tests.
Mike Petrilli: So Robert. I hate to break it to you, but you're wrong and you're wrong, twice.
RobertPondiscio: Oh man.
Mike Petrilli: All right. So here's the first thing. This is being driven in part by the federal government, not just the requirements of reading and math tests, but especially the new teacher evaluation mandate.
RobertPondiscio: That's what I'm saying Mike!
Mike Petrilli: Well no. You're talking about the stakes and I agree with that part.
RobertPondiscio: That is the stake.
Mike Petrilli: I'm also saying that quite literally there is now more testing happening because there's a mandate to collect student achievement data. Including in the non tested subjects. What have states and districts done? They've gone and created pre and post test for gym class!
RobertPondiscio: Okay, but Mike come one.
Mike Petrilli: It's ridiculous.
RobertPondiscio: Whoa. If you take away every single test, except the annual test in math and English, you will have no material change in the way schools are run right now.
Mike Petrilli: No, but Robert, the point is this. Why are we having this big backlash right now? I think, part of it is, a bunch of new tests have popped up in just the last couple of years. Including in gym class, and it's because of the teacher evaluation thing. Which is ridiculous.
RobertPondiscio: Right. Do you think if that goes away that nobody's going to complain about tests anymore?
Mike Petrilli: No. Listen. All right. It should go away. Now the problem was, states like Ohio would like to get rid of those things, but then they're told by the federal government. Well then what you have to do for gym teachers, rather than just, I don't know, go watch them and see if they're good teachers. If the kids are doing push ups correctly, or whatever they do. No, no, no. Instead you should start to evaluate them based on reading and math scores.
RobertPondiscio: That's just silly.
Mike Petrilli: Right. That's ridiculous. This is because the federal government with ... By the way a condition on waivers that is illegal, and that by the way nobody in Congress supports.
RobertPondiscio: It's really, really simple.
Mike Petrilli: So don't let your friends over there in the administration, Mr. Lackey, off the hook so easily. Now the second thing is, on your whole stakes thing. Look, Robert, I share a lot of your concerns about the design of accountability systems, or the way we are doing this has not encouraged good practice. We got to fix that.
RobertPondiscio: Correct. Correct. Stop right there, you were done.
Mike Petrilli: However, that doesn't mean throwing the baby out with the bath water, or else you become Diana Ravitch.
RobertPondiscio: Okay. No, no, no. Look.
Mike Petrilli: What is the difference between your position and Diane Ravitch's position? I just want to know!
RobertPondiscio: I'm all for accountability. Here's what I say. I don't have to have the solution to identify the problem. The problem is our relationship with testing is just out of wack. We are asking these tests to do things they were not intended to do. I want the data, because the data gives us the engine that creates the appetite for choice, for charters, for all these things that you and I love. If we are blind to the fact that we are making a toxic environment around these tests to serve our needs, we are going to lose this battle. We will have neither test, nor accountability.
Mike Petrilli: Okay, thank you Dr. Ravitch. Okay. Topic Number 3.
RobertPondiscio: Why do I subject myself to this?
Mike Petrilli: I don't know, because our listeners enjoy it. Okay, Topic Number 3.
Clara: Some students feel like school dress codes unfairly crack down on girls for wearing, quote on quote, "distracting clothing." Are dress codes sexist?
RobertPondiscio: Now you're just messing with me!
Mike Petrilli: I should say Robert, I did lie. I said it was going to be all NAEP, but I couldn't help but pass this one. This has gotten some attention that these dress codes. Look here's the argument, Robert, is that some of these dress codes, rather than be gender neutral, they say things like. Girls may not show their shoulders, let's say, if they have some kind of strapless dress on, or whatever, and there's nothing in the dress code about boys showing off their shoulders. It should at least be gender neutral. If you're going to wear a skirt, whether you're a girl or a boy, here's how long it can be. People thinking that some of these dress codes are out of date. Is this something the American public should be concerned about?
RobertPondiscio: Actually for what it's worth, this is one of those things because everybody has a feeling about this. Parents this is an intuitive issue, everybody has strong feelings about this, right? This is something that you're going to have to be an expert to have a strong opinion about. Look I like school uniforms for exactly this reason, it solves this problem. It takes the bat out of the hands of parents and kids, and say that I work in a charter school, we have uniforms, great, problem solved.
Mike Petrilli: Right.
RobertPondiscio: That's my answer. I mean on the one hand this issue.
Mike Petrilli: So you really don't believe in parent power do you? That's very interesting. You just took the bat out of the hands of the parent. I'm just kidding. In this case, right. Instead of having a dress code that says well you can buy something you bought at the mall, but it's got to meet these specifications. Instead you say you have to have a uniform.
RobertPondiscio: The funny thing is that in most schools, many schools, that have uniforms. What I would argue, they do it precisely backwards. If you're in elementary school you wear a school uniform, in middle school you wear half uniform, in high school you have a dress code. They should do it just the opposite. Let little kids wear whatever they want, because they're not going to wear distracting, and I'm making air quotes everybody, distracting clothes.
Mike Petrilli: Right. Well now this is the other thing about distracting. The other argument is that these dress codes are all about the boys. That they are basically [sent 00:11:04] on this thing that teenage boys are going to be sent into these crazy fits if the girls, young women maybe we should say, are scantily clad. Therefore we must make the girls cover up. We might as well have them wear one of those, what do they call them the hijab's, like it's over, right, the burka's, right.
RobertPondiscio: Right.
Mike Petrilli: Like we're over in the middle east, lest bad things happen. Again, if that is the case we should also make the boys cover up, because look what could happen the opposite way.
RobertPondiscio: Here's what I'm going to say. There are so many problems that we have in schools right now, this is one I have a difficult time getting exercised about. I say that as a father of a volleyball player that plays her sport in spandex, and that upsets me, and beach volleyball players had to wear bikinis.
Mike Petrilli: What do you want her wearing?
RobertPondiscio: Anything, a uniform.
Mike Petrilli: Well that.
RobertPondiscio: I mean I'm sympathetic to this issue, but.
Mike Petrilli: A Scottish kilt.
RobertPondiscio: The biggest issue that we face in American education.
Mike Petrilli: I know, but look here's the thing. I think we still should give schools a lot of leeway on this one.
RobertPondiscio: Sure.
Mike Petrilli: I think that.
RobertPondiscio: They should be able to choose the color of the uniform.
Mike Petrilli: We have pushed back against the adult authority in the schools on too many occasions. It all started with that crazy Tinker decision in the Supreme Court that said that kids have rights. Kids don't have rights!
RobertPondiscio: Not in my home.
Mike Petrilli: Okay. When you're 18 you get rights. Until then, forget about it. I'm glad we finally agree on something.
RobertPondiscio: We had to eventually.
Mike Petrilli: Yes. All right.
RobertPondiscio: Now can we agree that this is done?
Mike Petrilli: It is.
RobertPondiscio: Can I leave now, Mike?
Mike Petrilli: Done. You may leave now, but not you listeners because this everyone's favorite segment. Amber's Research Minute.
Amber welcome back to the show.
Amber: Thank you, Mike.
Mike Petrilli: We're ready to watch you Wonk, watch you NAEP NAEP. You really like that don't you. Maybe we do need to cut a video about that. What do you think?
Amber: Well imagine that. I have a new study out by the NAEP validity studies panel this week.
Mike Petrilli: Yes!
Amber: It analyzes the alignment of the 2015 math items. This is great for NAEP to the Common Core standards formats. The study classifies the actual 150 items at each grade level that are in the 2015 NAEP math item poll. These are the real items, 150 of them on NAEP, okay?
Panelists are to tag each NAEP item to a Common Core standard, or determine that it does not match a Common Core standard. Okay, it's simple enough. 18, I had to look a little bit I wanted to see who they got, but 18 mathematicians, teachers, math educators, supervisor with familiarity of the Common Core. Okay, so 18 folks looked at this stuff and their different backgrounds. The bottom line, reviewers determined that the concordat’s. I thought the concordat’s, I guess they don't want to say alignment, right? Even though the name of the report has alignment in it, but people have different ideas of what alignment means, anyway.
Mike Petrilli: It's a-
Amber: I digress. The bottom line is to determine the alignment between Common Core standards and NAEP, to the reasonable, I'll give you some details in a minute.
Mike Petrilli: Okay.
Amber: At both great levels, especially since NAEP is by design supposed to be broader than Common Core.
Mike Petrilli: Mm-hmm (affirmative).
Amber: It's supposed to, this is a direct quote, "It's supposed to maintain a degree of independence relative to the current fashions in instruction and curriculum." Okay?
Mike Petrilli: Okay.
Amber: Besides that the Common Core writers had access to the NAEP frameworks when they were writing the common core.
Mike Petrilli: Right.
Amber: Right, so okay, anyway. The study results, more specifically, at grade 4 panelists find that 79% of NAEP items were matched to the content that appears in the Common Core at, or below, grade 4, or below, right? Not just at, thinking like below. 79%.
RobertPondiscio: 21% did not?
Amber: That's right. 12% because there's some category where they couldn't make sense of it, or they disagreed, but 12% of NAEP items were judged to assess non Common Core content or were found at Grade 5 or above. The content area with the lowest percentage of NAEP items covering it, that's found in the Common Core at, or below, grade 4. Any idea?
Mike Petrilli: Procedural fluency?
Amber: Nah. Data analysis, statistics, and probability.
Mike Petrilli: Oh.
Amber: Where only 47% of the items were classified as matches. At grade 8 the overall alignment of NAEP to Common Core, at or below grade 8, is 87%. Although there is much variation across the content areas, 7% of NAEP items assess something that's not in the Common Core or it's in the high school. A couple of other little factoids.
There is less agreement between Common Core and NAEP when it comes to algebra and geometry.
Mike Petrilli: Mm-hmm (affirmative).
Amber: Just over half of grade 3 and 4's Common Core's standards, and the Algebra domain, have at least 1 NAEP item match to it.
Mike Petrilli: Mm-hmm (affirmative).
Amber: Okay, so that's. Again the Data analysis, statistics, and probability as we found in grade 4, also had the greatest proportion of items judged to access content not covered in Common Core. Basically you got those. That's your one area where you don't have clear matches.
Mike Petrilli: In other words, there's more of that stuff in the NAEP framework then in the Common Core?
Amber: That's right.
Mike Petrilli: Right.
Amber: Then there's a sort of this weird ending that seemed kind of conflicting to me. On the one hand these authors say that it's been 10 years since NAEP has conducted a thorough review of its math framework. They recommend that we do so in light of Common Core, as well as other college and career ready standards, but I'm thinking didn't we just learn that independence is a really important thing and that we don't want to be caught up in the fad. By the way this alignment is pretty high already, right?
Mike Petrilli: Right, well.
Amber: I'm a little confused.
Mike Petrilli: Well, couple things to add. One is they try to keep the trend line, particularly for the long term trends, right?
Amber: Right.
Mike Petrilli: That pretty much stays standard, but they do tend to update these frameworks every once in awhile to be somewhat matching what kids are learning in schools so that is a tough decision. I did see some folks say today, that the areas where the kids really bombed on the 2015 test, were those that had the least amount of alignment with the Common Core. When you look at the sub scale scores, and get into some of those areas where basically save, under Common Core 4th graders now don't get to those topics until the 5th grade, and they did not do well on them. I don't know. I think it's a very slippery slope to make these arguments when we've held NAEP up as a gold standard. It could be the case that Common Core for good reasons waits until 5th grade, let's say to do some of these things, and the kids just haven't gotten to it yet.
RobertPondiscio: Does this explain some of the potential softness that we are seeing in 2015 data?
Mike Petrilli: Yes, it could. It absolutely.
RobertPondiscio: It means it’s worth learning more about.
Mike Petrilli: Yes, it is a hypothesis.
RobertPondiscio: Right.
Mike Petrilli: We will not claim that be the case because that would be mis-NAEPery 00:17:57].
Amber: We say we don't and then we just do mis-NAEPery 00:18:01] don't we?
Mike Petrilli: No. As long as you state it as a hypothesis Amber, it's not mis-NAEPery.
RobertPondiscio: Conjecture! So now you're a researcher. Go off and tell us what you find out.
Amber: Is this the role of the game?
Mike Petrilli: Yes, it's not a game! Amber.
Amber: Then you look high and mighty because you say that you aren't doing it, but you do it, and then you say but I didn't mean to do it? I don't really get it.
Mike Petrilli: Amber you clearly are missing the nuances here.
RobertPondiscio: She's missing out. She knows, she knows!
Amber: Anyhow. Hey, it was a good study. Honestly I thought it was a little bit more matching going on then I thought.
Mike Petrilli: Yeah, yeah. There is a lot of overlap. Look it is a difficult question. Should NAEP change? I don't know. I don't know. I think there's pros and cons on either side.
Amber: I'm with you.
Mike Petrilli: We do need some kind of benchmark that is not corrupted by the Common Core.
Amber: That's right.
Mike Petrilli: Alright. By the way Amber, I understand we may have some alignment studies coming out.
Amber: Maybe.
Mike Petrilli: As well.
Amber: Maybe this little tiny study that I've been working on for I don't know how long. Yes.
Mike Petrilli: Yes. With MCAS, PARCC, Smarter Balance and ACT aspire coming along.
Amber: Coming soon. Giving me many grey hairs to cover up with blonde dye.
Mike Petrilli: You do that so well. Okay. That is all the time we've got for this week. Until next week!
RobertPondiscio: I'm Robert Pondiscio.
Mike Petrilli: Watch him NAEP, and
RobertPondiscio: Watch me NAEP, NAEP.
Mike Petrilli: Watch him wonk, wonk. Okay. I'm Mike Petrilli. Nae, nae-ing off.
