 |  |  |
|
|||
|
|||
|
|||
| |
||
| |||
 |
            |
|
|
Paper
Transcript of Iris Weiss's talk
Good morning. I'm going to give you a kind of potpourri of data that you will see in the report when we get feedback and we get the report copied. Some of this is in the summary. The fact that the more professional development you provide to teachers, the more they like it, is an encouraging sign. Now it could be, but I think not, because this finding continues to be stable over time-- It could be that the ones that are participating first are the ones that are seeking it out, etc., etc. But it's not attenuating over time. So that's a nice finding.  We asked back in the early, early time-- cohort two is the first one that was pure on this one-- we asked them before the LSC to think about typical professional development in their district. And these are teachers that said they were likely to be given time to work with other teachers and receive support for implementation.  The first box is prior to the LSC. The second and third are with the LSC. Notice the receiving support for implementation. This is one of the hallmarks of the LSC, the year round professional development, not just the summer burst and go away. So this is a very positive sign. One of the things NSF has asked us to do in the core evaluation is to triangulate data, and in addition to the teachers' self-report data, your evaluators have been rating the support for implementation. That is looking pretty good as well on a five point scale.  Previewing what I'm going to say later, it's also the biggest concern of teachers, that they're not getting enough support. They would like more, and they recognize that more can be infinite, but that is a major area of concern. As many of you are aware, the teacher questionnaires have been used to create composites of individual items because individual items can be very unreliable. There is a composite for both science and math on content preparedness. This is, in fact, a significant finding when you compare the zero hours to the 40 or more. But it's a fairly small effect in the teachers' own perceptions of their content preparedness in science.   For pedagogical preparedness, you see that nice stair step pattern, and then we get to the 6-12. And if you look at the chart that compares the effect sizes, this is a very interesting phenomenon. According to the teachers' self report, we have consistent and large statistical differences with effect sizes consistently larger over time for K-8 science. By evaluator ratings, K-8 science is the lowest rating in terms of quality. So it may be that the need is so great, but it's a striking difference. But with NS being not significant, with 6-12 we're having trouble demonstrating as much impact. 
Professional Development on Teachers and Teaching
* Comparison between untreated teachers and teachers with 40 or more hours LSC professional development.
I'm not surprised on content preparedness. I am surprised in terms of pedagogical preparedness that we're not seeing more of a change in self report data. Okay. The next two slides are really important. In fact, NSF has been hammering us for the last several years to get more on the designated instructional materials into the core evaluation and really make it central. In the classroom observation protocol the last couple of years, we have had questions about whether the lesson is based on the instructional materials, whether it was taught essentially as designed, or modified, and whether the modification made the lesson better or worse.  I worry a little that since the evaluators know what the materials are that they might simply be rating it higher because the materials are good. We can't distinguish that, except for saying that 24 percent of the lessons with teachers who were untreated were observed to be using the instructional materials. And more than twice as many with treated teachers. So treated teachers are more likely to be using the designated materials than untreated. And that's not a surprise. And some of your projects, you don't even distribute the materials until the teachers have treatment. The single most important finding of the core evaluation is right here. When the evaluators rated lessons on this five point scale where four and five are high quality, effective lessons, when teachers were not using the designated materials, there was essentially no difference between treatment and no treatment. When teachers were using LSC materials, there was a large difference between treated and untreated.  So if you don't use the materials, your lessons are likely to be judged less effective. If you use the materials but have not received professional development around using the materials, your classroom is likely to be judged less effective. But the combination of using the materials and getting professional development is something that is leading to lessons that are judged to be more effective by trained evaluators. It was a very reassuring finding to me because I was worried that maybe the evaluators were walking in and seeing the materials used and just automatically saying, "This is a good lesson." So now you don't have to listen anymore. I just told you the most important finding. What are some of the issues that the evaluators have been reporting? The most challenging issues, they say, are finding time to do professional development during the academic year. Teacher after teacher in the interview data reported the lack of being able to find qualified substitutes. Having to prepare a lesson for the substitute teacher, then having it not done the way they had wanted it to be done. This creates a conflict, a very heavy tension for teachers. That they feel like they know the professional development is benefiting them, but they don't want to take it out of the hide of their students. That's been a problem. Evaluators report that addressing the diverse content needs among teachers has been a real difficult issue for a lot of projects. Some projects have addressed this by setting up menus. Sometimes that works better than others. There's not always a guarantee that teachers will elect based on what they need. They may elect based on what they want and in the spirit of professionalism, some projects are reluctant to require that you do something. The teacher should have some say. So this is an area that's proven thorny for a number of projects. One of the nice things about the core evaluation is that individual evaluators report things, and we get a chance to see across 70 projects what are the themes, and which are the idiosyncratic kinds of issues. Teachers have commented quite a bit in the interviews about the length of time it takes to implement the materials well, especially in elementary science. That is turning out to be a major problem. Remember I said the classroom observers look and see if the lesson was modified? In addition to modifying lessons, what teachers are doing, again especially in elementary science, is they're picking and choosing activities. And they're not necessarily picking and choosing activities that make conceptual sense as a whole.
As Deborah Ball said yesterday, when you modify activities, sometimes the mathematics of it gets lost, or it's a different mathematical task. That's what evaluators are reporting. If they either modify the unit by picking and choosing activities or modify the lesson, they may modify it to make it more engaging to kids, but not necessarily deepen the science and mathematics. And that's something that really needs to be addressed. Again, I mentioned the additional follow up support during academic year, and materials management, the Achilles heel of elementary science projects. When it goes well, the teacher's absolutely enthused. They are thrilled. They say things like, "For the first time, I have everything that I need. I can do this. It's there. It's in good shape." When the kit shows up or there's not enough of the materials, they're just stymied, and they're resentful.
At the project level, an issue that emerges as challenging for the LSCs is teacher turnover. You work very hard with teachers only to see them leave the district. This year, when you create your sampling frame, you're asked to leave the teachers in indefinitely so that you can record the hours that you have given to teachers when they have left, since you are helping the other districts around the country. Administrative support, administrative turnover, competing priorities, reading, reading, reading. Assessment at every level bedevils projects. We have teachers practically crying in the interviews that they believe what you're telling them is what's best for their kids. They try it, it's engaging. The kids are learning. But they're not learning what's on the district tests. And that's what all the pressure is. To get test scores up. The state test. All of that. That is a real issue for our whole profession. Assessment just seems to get in the way. We don't seem to know how to test reliably and inexpensively and well, the things that we want to test. And my personal feeling is until we do that, we're going to always be stymied in our efforts.
So what are some of the recommendations that flow from all of this? Based on what we read in the evaluator reports, addressing both content and pedagogy within the context of the designated instructional materials seems to be a winner. It's the basic design of the LSC that seems to be working. The old days when we used to do content in the morning and pedagogy in the afternoon seem to be replaced by something that is more effective. The transfer issues seem less daunting for teachers when they can learn both the content and the pedagogy.
But again, you need to help teachers see what the important content is. It's almost like you can't emphasize that enough. Teachers as practitioners are alert and already attuned to how to make it work with their kids. We have to help them be more conscious of what is the underlying content here. If you're going to shorten it, which activities should you leave out, and which ones are absolutely key? When you're going through activities in the modules, don't assume that because you know what the important content is, it's obvious to them. Because for learners, all of this is just sort of in the ether. When my son was not quite three, he asked me to write his name on some scribbling that he had done. His name is Jeremy, and I picked up a blue crayon and started to write a J. He said, "No! The J is purple." "Okay." I keep writing. "No, the E is blue." And this went on. I finished writing his name. And then I suddenly-- And I run downstairs. His magnetic alphabet letters. He had memorized the color. To the three year old, the color is more salient than the shape. He didn't know that that was not powerful, that it wasn't going to get him very far. And sometimes I think that with all of us learners, we're grabbing at features, and it's our job as professional developers to help them see which of the features is worth learning. And the color of the letters is not. Okay, this is that same thought. Focus on the big picture. What important mathematical and scientific concepts are being developed? We need to emphasize the use of the materials as the developers intended rather than having teachers develop or adapt the materials. This particularly goes against the grain to science people. For many years we've talked about mechanical use and getting beyond mechanical use into creative use. And based on what the LSC evaluators are reporting, it looks like competent mechanical use is a whole lot better than creative creative use that loses the content.
And this is really the same theme about making the lessons explicit. Most of your projects model effective pedagogy. But evaluators report that you sometimes miss the opportunity to make it explicit. We all hear that teachers teach as they were taught. You want to have some control over which features they're modeling.
This is stuff that all of your projects are trying to do. The evaluators report that teachers want more of it, and time for teachers to share experiences and insights, and to get help during the academic year. One of the issues that we're all struggling with is the time. A hundred hours, a hundred thirty hours is a lot more than we used to be able to give teachers. It's a lot less than we need. So it's constantly balancing that.
Projects find that teachers really do appreciate and make good use of time during the academic year while they're trying to implement this stuff to get their questions answered. Many of you are using teacher leaders who do have experience using the materials, and that is a real strength for the credibility that comes with having people actually use the materials in their classroom. If it is combined with those people helping them understand what's the important science and math, that's a winner. But as we know, we don't have this huge pool of teacher leaders out there who are already knowledgeable in content, knowledgeable in pedagogy, experienced using the materials, have leadership training, etc. So we pick the best we can and then it becomes very important to provide sufficient orientation and support to the teacher leaders and the scientists/mathematicians.
It's just as you would expect. When there are concerns, evaluators report that the teacher leaders don't know enough content, and the scientists/mathematicians don't know how to work with teachers. This is not a big surprise. And then comes the issue of focusing on all teachers. Many projects cannot logistically work with all of the teachers over the whole five years. If you can't, evaluators are reporting that it works better when you work not just with one volunteer from here, one volunteer from there, but with groups, whether it's by grade range, or groups of schools. Something that gives a critical mass that allows teachers to support each other.
I also believe, although we don't have evidence of this yet, that there is hope for getting the recalcitrant teachers caught up in the enthusiasm of their peers. Evaluators worry in some of these projects, where you're working with waves of people who volunteer, that the hard core that's left at the end, you're not going to know what to do with. Now, in some cases, it might be an intelligent strategy of they're going to retire and just wait them out. But there's not a one-to-one correspondence between resistance and age. Okay. I'm switching gears entirely and talking about the core evaluation system. These were just some of the results that you will read in your summary and you will read in the full technical report. We are changing some aspects of the core evaluation, and I thought it would be helpful to give you a chance to hear it as well as read it in your manuals. The core evaluation system includes teacher level data, as you know, the questionnaires, the classroom observations, and the interviews. When I did it this way, I was surprised myself to see that the only school level is the principal questionnaires which reinforces what Susan was saying about the importance of getting high response rates. Because that's really all we're getting. There are a few projects that are doing school documentation efforts. But for the core evaluation, this is the only data we get at the school level.
There's quite a bit at the project level. The PI descriptions of project strategies are what we would have worked on if we had had the pre-session. I haven't quite figured out how we're going to make up for having been snowed out of that. We have the project team interviews, where the evaluator tries to find out what you're thinking and how that's progressing.
You've been collecting data on district policies, and the evaluators do the professional development observations. I want to reiterate what Susan said. The core evaluation requires that your lead evaluator personally observe three sessions. Because it is that lead evaluator that's rating your program and needs to have first hand knowledge. And in some of the projects, that's not happening. Other people on the evaluation team are doing all of the professional development observations. The reason Susan has all that compliance information at her fingertips is they ask us to report it periodically. That's one of the things that we monitor. So if your lead evaluator has not been doing the observations, NSF knows that. Here are the changes to the core evaluation system that are going on right now. Some of the questionnaire and observation data are being collected less frequently, and they're being collected longitudinally. For the first time, NSF has decided to have impact data collected after you've done your major professional development, so that the impact has a chance to be seen.
The impact of this change is that your level of evaluation effort varies by year. So in year one and in the interim years, you do the professional development observations and the teacher interviews, a very short version of the principal questionnaire. It's two pages. The PI interviews. And then a much shorter evaluation report. The evaluators were getting tired of telling us the data every year that didn't change. And we were getting tired of reading it every year that didn't change. So this is a good win/win.
In the second year, and the final year, data collection is more extensive. Not only are you doing the professional development observations, you're doing the teacher questionnaires, the longer version of the principal questionnaire. Classroom observations. And then a fuller report, where they do look at both the quality and the impact, because they've collected impact data, teacher questionnaire data, and classroom observation data.
In terms of level of effort in the new system, it looks like year one and the interim years are small, and year two and the final year are large. And as you know, the core evaluation calendar goes from September 1 to August 31 each year. Your final year is defined as the data collection year following the project's last summer of professional development.
Every fall we're going to ask you to accept a data collection schedule confirming your final summer, because that changes. Projects have money left over and ask for no cost extensions. We don't want you doing your final year data collection before your final year.
Now, I was supposed to have staff with me who understand this far better than me, so what they told me to tell you was that existing projects are transitioning to the revised system, and that you should rely on project-specific schedules that are on the web. I see from the nods that that means something to you. I'm glad.
Okay. We've talked about teacher turnover and administrator turnover. There is also evaluator turnover, as you have discovered. Because of the transition to the longitudinal system where you do more observations less often, some projects are finding the need to have additional classroom observers trained. And we are offering, snow permitting, a session in Chapel Hill February 10 and 11, and in March, here in D.C., just before the new project orientation meeting. So if you need to train more classroom observers, these are those two opportunities.
If you are adding professional development observers, not lead evaluators, other professional development observers, we have prepared a video training package that your lead evaluator can use to train other professional development observers. And that's available from our office on request. It can only be used for training observers. Don't bother asking us if you can use it because you want to work with principals or teacher leaders or anybody else. We have permission to use the tapes only in training and your evaluator signs an agreement that that's what they're going to do with them. We are breaking the law if it is done otherwise, and you are breaking the law if it is done otherwise.
If your project is losing its lead evaluator, then you will need to send that new lead evaluator to get trained with the new project meeting in March. So if you have a new lead evaluator who also wants to learn to be a classroom observer, get certified, they would come the whole time, the two days before the March 1-2 and then the 2-4.
Pat Campbell gets credit for this next slide. The projects are concerned about if you went in the first year and said to the teachers, "Each year it's a new random sample of observations, the chances of you being observed again are very small," and now we're coming in and we're observing them again. NSF asks that we provide from the core evaluation contract some incentive for teachers each time they're observed. We feel that giving money is sort of insulting. The amount of money you could give feels like it's too small an amount to compensate them for their effort. So instead what we've worked out is a voucher system where the evaluator will be given a piece of paper and as soon as they finish observing that teacher, will give them that. It's an order form. There's a catalogue on line-- we haven't worked out all the details-- where they can order science and math materials.
Okay. Sampling frames are different this year, and you may not open your notebook, but in your notebook, as well as in your manual, is the information about creating the sampling frame. There are now going to be two sampling frames. One of the things that was a total disaster last year, as many of you remember, was we were told by OMB that we had to collect demographic information using the new census definitions. It didn't make any sense to the principals. It was a disaster.
Many of you said, "Is there any way we can take this off the backs of our principals?" And there is. You put it on your backs. We've gone back to the old definitions. We checked out the law, and we're now collecting data the way you collect it. And when the government gets its act together and all that, we'll change it. But for now, you're going to provide the demographic data as part of the school sampling frame and update it each year. And that way we can take it off the principal questionnaire. Many of you said, "We've got to take the burden off our principals. We're at the edge anyway in asking them to do stuff. And if we keep asking them for more they're going to rebel." So the school sampling frame will have a list of all participating districts and schools and will provide current demographic information for each school. All schools remain in the sampling frame, even if for whatever reason they drop out, so that the sampling frame will become a record. We're not asking you to go and backtrack. We're saying from now on-- __: So it's not really a sampling frame? IW: No, for the principals, in fact, that's right. It is a population frame. It is a sampling frame where we will sample with the rate of one. Probability of one. The math people just gave me a hard time.
Okay. For the teacher-sampling frame, again, we're asking that all teachers remain in the sampling frame even if they're no longer participating in the project. It now contains more detailed information about a teacher's treatment status, where you're given some ranges and for each teacher, you write this year how much treatment they've had since the beginning of the project. Next year, you say how much treatment they've had since the beginning of the project. So it provides a way for you to keep track of that and for NSF to report to GPRA about the treatment levels. So for all of this to work, it's important that there be consistency across the two sampling frames and from year to year. That if a school is school three, it's always school three. And any teacher in that school is also school three.
Sampling frames are due 30 days prior to the date you need the samples and the questionnaires. And everybody look. (Laughter) There is an infinite number of ways to do the sampling frame, only one of which is right. If it's wrong, we send it back.
Now, you may notice that we are more than once a year asking you the number of targeted teachers, schools, and districts. And that may seem repetitive. If you ever got it twice in a row the same, we might not have to do that. We use that information for sampling, for weighting the data, for computing response rates, for reporting to GPRA. So we keep asking it. And we're going to continue to ask it. Maybe if it's three times in a row the same we'll drop you off the radar screen.
__: For those of us in large districts, the district we're working with, that number will stay the same. But every year, schools open, schools close. Some of our schools subdivide, the number of teachers is never constant, even within a building. I mean, we'd love it if it was constant, because we could keep track of people. But it's not. IW: We're still working on an answer to the e-mail we got from Pittsburgh, which totally cracked us up. This school that used to be called this is-- All the faculty from that school went over here. A new school has opened with the name that the old school used to-- We thought it was a joke the first few times we read it. (Laughter) __: And it's true! IW: If in doubt, check it out. We will try always. For these issues go to your NSF program officer. We don't make these decisions, they tell us how they want to do it. We implement. We are staff to NSF in this thing. Fran? __: I know that Claudia is working on one of my questions, which was-- Teachers who are teaching half time in two schools? (Laughter) IW: Listen. I have had enough grief for one week. I have an answer to that. E-mail Claudia! I am blessed in Claudia Templeton and Joan Pasley, two of the brightest, most hard working and constantly gracious people. I'll sometimes listen to them and think, how are they having the patience? I want to grab the phone away and say, "Grr." But they don't let me. The core evaluation. What does NSF expect of you? They expect an evaluation team with content expertise, understanding of classroom instruction, knowledge of effective professional development. They expect the school and targeted teacher sampling frames. This is all in your notebook. I suspect all of this is going to go up on LSC-Net as well.
NSF expects the professional development observations to be done throughout the year. This has been an issue in some projects, especially those of you who have distance evaluators. If they do six observations all in one week, that is not fair to your project. You're not getting the breadth, you're not getting the opportunity to demonstrate. It's too much of a crapshoot. You ought to be insisting that they observe throughout the year and somehow have the lead evaluator do the three and work it out. But you really oughtn't to get it all bunched at once. We have already said that you should pick sessions with the evaluator that are representative of what you're doing. If you're doing some school based, they should be observing some school based, etc. We hadn't said spread throughout the year. But that has turned out to be an issue.
As Susan mentioned, NSF expects high response rates. At least 80 percent for the teacher questionnaires. At least 90 percent for the principal questionnaires, and the full complement of classroom observations and teacher interviews. The classroom observations and teacher interviews are going swimmingly and I think it's because NSF told us to add to the sample the next year if they didn't get them the previous year. So I think that they've gotten the message that they might as well do it.
NSF expects that the evaluator reports are responsive to the reporting guidelines. This has not always been the case. There have been evaluators who are answering last year's questions, and that doesn't help us do this year's reports. So that's important.
The evaluator reports should identify the project's strengths and weaknesses with supporting evidence. Saying that this was good or bad and not saying why they think so is not helpful. It's not fair to you, and it's not information that we can use. And they should be providing sound recommendations for project improvement. This is actually what you should expect of your evaluator. NSF expects the projects' PIs to take responsibility for ensuring thorough and timely core evaluation data collection and reporting. "Our evaluator didn't do a good job" is not an acceptable response to not doing this.
What NSF expects of us is development of valid and reliable data collection instruments that measure important constructs with triangulation for professional development, for teacher attitudes and preparation, and for classroom practice. They expect us to prepare and disseminate the data collection manual, the newsletters, the reporting guidelines, and your project data tables. They expect us to provide training for the lead evaluators, the classroom observers, and the professional development observers. They expect us to provide data for GPRA and timely preparation of the cross site report, and that requires timely submission of individual project reports.
Identification of lessons learned-- that's the final thing I want to talk about. This is a major focus for us this year. In a quantitative sense, we are trying to collect information from you folks on project strategies so we can begin to identify lessons, to work on building a theory here of what works in what context, under what circumstances. Does it work better to work with all of the teachers? To work with a subset of schools? To do content before you do the pedagogy? That was what the pre-session was going to help us refine, what might turn into a project strategies questionnaire. We will probably send something out as a pilot that will ask far more questions than we intend to use later. What we have to do is find out where there's variation. If all of the projects are doing something the same, then putting it into-- HLM is hierarchical linear modeling. It's a sophisticated regression procedure. But if there's no variation, then we won't get any effect. The qualitative documentation of lessons learned is happening in two ways. One is we have some funding for individual projects. Some have been doing mini grants to work on various aspects of this, and we'll tell you more on that as we work that out.
And we're going to have a lessons learned conference. It will be in D.C. June 5th and 6th. We did not have, and are not having, a lead evaluator meeting this year. So the money you put in your budget for the lead evaluator will pay to bring the evaluator to that meeting. We will pay for one other person from the project. You must come as a team. And one of the reasons is we didn't want to just get the evaluators together because they may feel some sense of their not able to share honestly and openly because they report to you. So we want you there. We want both perspectives.
I want to end by saying that when I walked in yesterday, I was really struck by the family feeling that I was feeling as I walked in. That this is a really exciting enterprise to be part of and I appreciate the opportunity to work with all of you on this. I'm learning a lot, and I think we're making a difference out there. Thank you.
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
