|
posted by:
|
Mark St. John
on April 23, 1998
at 10:51AM
|
|
subject:
|
Reflections
|
This is a relatively long message. I post it as a way to explain my
thinking re the evaluation of professional development in terms of
student outcomes. I think it reflects the deep and interwoven nature of
learning, and, indeed of the "system" we are trying to improve and
reform... In what follows I write down the stream of thought that has
emerged for me as I struggle with the reality of trying to assess the
value of a professional development program by measuring student
achievement in the classes of those who participate:
As an "evaluator" of privately and publicly funded programs, I am always
being asked to determine 1) "if the project has met its goals" or 2) if
the project has succeeded in achieving its ultimate goal of "improving
student achievement". After many years, I find that both of these
questions leave me flat. While at first glance they appear perfectly
reasonable, they both actually are not deep or intelligent questions,
and, worse, they may even be counterproductive.
Take the first question: ... Who cares if the project has met its goals?
Maybe the goals were dumb to begin with... Or maybe they were too
lofty.. And If the answer is yes, then what? If no, then what? There
is no way at present to use the measured success of one project (which
is very specific to those who implemented in a specific set of
curcumstances at a specific time) to create "replicas" which can be
widely disseminated... Projects are not pills...and use the careful
measurement of the achievement of goals creates a kind of knowledge
that does not take one very far...
Take the second question, and apply it to a professional development
program like the LSCs... That is, ask the following question: Can you
PROVE that this professional development program was directly
responsible for improving student achievement in the classrooms of the
participating teachers?
Let me make some points about why this question is essentially a poor
one to ask ... and that the very asking of it demonstrates a lack of
understanding (or a willingness to understand) the actual complexity of
"student achievement"...(In fact, in my more cynical moments I would say
this question is a very good indicator of a very weak understanding of
the educational process.)
1) It is very difficult to measure "what a student knows and is able to
do".
There are great efforts being made to develop test items that are valid
and reliable measures of very simple types of knowledge and skills.. It
is much more difficult to measure the "higher order" knowledges and
skills of the type that are espoused in the national standards, for
example... Thus, simply assessing what one student knows in a given
domain at a given time is very difficult to do ...
2) To measure the increase in a student's knowledge or skills over time
is even more difficult... That is, one needs to know the INCREASE in
knowledge and skills, not jsut what a student knows at an absolute
level.... The measurement of change in knowledge inevitably involves
subtracting some sort of pre-test measure from a post test measure...
(Or equally difficult trying to compare a "treated group" with an
equivalent "untreated" group) And when one subtracts (or compares) two
numbers that have large uncertainties in them, one gets a number with a
VERY large uncertainty in it... That makes it almost impossible to
accurately assess small changes in higher order skills.... Also, the
assessments themselves are typically designed to measure absolute levels
of knowledge and skills, and they may be very insensitive to small
changes.... Thus the point is that it is very very hard to measure
accurately small changes in achievement...
..The signal to noise ratio here is very very small...
3) In spite of this, let us pretend that we are able to determine that a
student has in fact grown, over the course of a year, in his or her
abilities and knowledge... Now what accounts for a student's growth ?
In fact, there are many many possible inputs that may have contributed
to this growth... there is the natural maturation of the student...
there are many influences outside of school that both promote and
inhibit growth.. and there are many experiences within school and within
the classroom that contribute to changes in ability... Thus, even if we
could measure accurately the growth of a student's ability in, say,
doing inquiry, or communicating in mathematics, or even carrying out
long division... there are many many possible contributors to that
growth....
4) Also, let us be clear that a child's knowledge is cumulative... Most
of what a student knows and is able to do in seventh grade is not due to
his or her experience in seventh grade... Rather it is due to what
happens both in and out of school over the time up to seventh grade...
Thus, it is a very real (and very common mistake) to assume that a
student's achievement is somehow due to the experiences he or she had
during the past several months in that grade with that teacher... This
is only the tip of the iceberg....Thus, assigning attribution for the
development of a particular skill to a particular classroom experience
is to deny the long term, cumulative, and non linear nature of
learning... Perhaps what a child is suddenly able to do in seventh grade
is the slow maturation of experiences that took place in grades 5 and
6.... Thus, what a child know and is able to do ...is only very
partially a product of the experience they are having in a particular
classroom with a particular teacher in a particular year...
5) BUT lets even pretend that it is possible to assign credit to a
particular classroom for a particular increase in student learning.
Then we still need to know what it was about that classroom that made it
effective. And here again the number of contributors is huge... That
is, the instruction that takes place in a given classroom on a given
day is the product of very many factors... .. (This is the very essence
of the argument for Systemic Change).. The instruction that actually
happens is influenced by the state and local curriculum, the available
texts and materials, the time of the year, the state and local
assessments, the school culture, the department or grade level
priorities etc... Perhaps we should attribute increase in student
achievement to a reduced class size, to the introduction of new state or
district standards, to the presence of a new assessment system, to a new
curriculum in use, to an effort to restructure schools etc... All of
these are in addition to the variables that are associated with the
teacher -- his or her content knowledge and background, their pre
service experiences, their teaching history, their involvement in
professional associations, and, yes, perhaps even some professional
development experiences they may have had....
SO.... let us review the argument made this far... increases in student
achievement on important concepts and skills is very hard to measure...
and there are very many determinants of such achievement and of any
increases in achievement... and one possible contributor is the
classroom of the current year ... and there are many many factors that
contribute to the nature and quality of the instruction in that
classroom .... only on of which is what the teacher chooses and is able
to do....
6) And if we keep going backwards into this causal ladder... we will
find that the teacher's practice is shaped, in part, by his or her
knowledge and beliefs.. .. and these are in turn formed over many years
by many experiences.... Quite frankly, one professional development
program, no matter how wonderful, comprises only one small part of what
that teachers does, knows, and believes...
7) Just as with student learning, teacher learning is cumulative and non
linear. .. When exactly does a professional development experience "kick
in"? Does it improve instruction immediately? Does it take a year or
two? How much and what kind of PD experience counts here? What if a
professional development experience leads a teacher to other programs
and other experiences? Who gets the "credit"? What happens when a
teacher involved in an LSC uses a new curriculum at a grade level that
is interested in thematic teaching in a school that is undergoing
restructuring in a district that has a new scope and sequence in a state
with new assessments? What is causing what here?
Professional learning follows what I think of as a "waring blendor"
process... Each new experiences is ground up in the waring blender of
past experiences and knowledge, and it becomes impossible to extract the
sources of knowledge and experience that contribute to the final
stew.... And, I think, paradoxically, the more profound an experience
is, the more it becomes assimilated, and the more unindentifiable the
source becomes....
8) Thus, professional development may well contribute strongly to a
teacher's knowledge and believes but it is only one input... and the
inputs are highly interactive with each other.... so that classroom
instruction is a stew that is uniquely and inexplicably shaped by all of
these interactive factors...
Let us say we had a classroom situation where student achievement of the
kinds of skills and knowledge espoused in the national standards seemed
very high... What is likely to be the cause ??? My guess is that you
would find a WHOLE FABRIC of high quality ... good schools, good
curriculum, professional teachers, supportive community etc... Not to
mention a likelyhood of an upper SES community, and very strong family
support system... etc... This positive situation might also likely
involve a strong professional development system... But it is one set of
threads that is adding to the strength of the fabric... Without the
other threads.. there is nothing there...no matter how strong the prof
dev threads...
ANALOGIES...
Let me end this diatribe with some analogies to make the point
further... And remember.. all I am doing here is trying to dispel the
idea that it is possible or even sensible to assess professional
development by measuring student achievement... (Note that this is very
different from saying that the ULTIMATE goal of professional development
is to improve student achievement. )
If a teacher is meant to be a "guide" or a "coach" that assists student
learning , then we ought to be able to use that metaphor to look at
other domains... Let us take, for example, a baseball coach... If a
baseball coach attended a 3 day clinic on the latest techniques in
hitting, would we assess the quality of that clinic by measuring the
team's win/loss record in the next year.... (Remember again there may
be a few other variables that affect how the team is doing...)
If a cook goes to a cooking class, does that mean that the patrons will
be happier at the restaurant? (Somehow we always picture these
situations in our minds by holding the constraints and other variables
constant...) Could we evaluate the quality of the cooking class by
asking the patrons about the quality of the restaurant? (What if the
restaurant changed "superintendents" every two years, couldnt get the
materials he chef wanted, and patrons came and went on a frequent
basis, and there was an overwhelming demand for hamburgers and
appreciation of little else?)
If a pilot attends a flight training clinic, does it mean that the
airline will have a better safety record? Seems logical... but it does
not necessarily follow... Can we evaluate the training clinic by looking
at the airline safety record? (Consider a situation where the planes
are designed by pilots on task force during the summer, and passengers
are asked to bring maps from home, and where lead pilots are asked to
train the other pilots on Saturday mornings...)
So, going back to the whole fabric notion, professional development is
a NECESSARY BUT NOT SUFFICIENT, condition for a healthy and robust
educational system... trying to evaluate it impact in the absence of the
other critical factors makes little sense....
........................................................................
What I have said up to this point argues that it is simplistic to try to
assess professional development by looking at student achievment,
particularly in those cases where other critical dimensions of the
system are lacking or are not aligned... But this does not mean that
one can not look in productive ways at the ways in which and the extent
to which professional development can contribute to the overall fabric
of the system that determines the quality of instruction. That is what
I hope we can do as we think about how the best ways to identify and
assess the real contributions of professional development.
The fact that professional development should not be evaluated on the
basis of student achievement does not mean that there can not be a
productive focus on students in the conceptualization and design of
professional development... Hence professional development that focuses
of real classrooms, on student work, on the postulates of national
standards are all quite legitimate...
AND it may be possible to identify the ways in which PD adds to the
capacity of the system and enhances the probability that high quality
instruction will occur ...
OR it may be possible to assess the QUALITY of professional development
per se.. ..and use that assessment to make judgments about its merit....
I was in a school the other day where the teachers were all quite
motivated and doing quite a good job in a difficult situation. They
were working with students from a nearby housing project and using many
innovative strategies to help them learn the basics as well as
experience inquiry, critical thinking etc. .. But these teachers were
discouraged ... their school had just been put on a list of district
schools that were "inadequate"... The reason for this was that only 38%
of the fourth graders had scored above adequacy on the district math
test... (They needed more than 40% to avoid the label of inadequate...)
Note that there was no attempt to determine the "value added" of this
school to the students. (Perhaps only 5% would have scored above
adequate without the programs this school offered.) And note that there
are probably other schools who score better but who add little or
nothing to the knowledge their students bring with them...
I relate this story because it shows the great tendency for state and
district administrators (politicians?) to use over simple and just plain
wrong assessment procedures... Had they assessed value-added or quality
of instruction or improvement over time I suspect they would have come
up with very different findings....
Thus, I hope we can work together to think hard and carefully about
assessment and the inferences that are to be made from various
measures... I am always astounded that in science education there are
methods of assessment and conclusions drawn that would be seen as
flawed, if not fraudelent, in the scientific disciplines....
I think it was Alfred North Whitehead who said..." For every complex
problem there is a simple, and wrong, solution..." ]
I urge us all to be more careful and thoughtful as we proceed in what
are essentially murky political waters.
|
|
