Questions are Expensive!

A passage from this great posting on standards-based grading caught my attention:

Am I tempted to include one question on my test to send the message “HEY! We spent a day on this in class and we had a homework assignment on it, so you better do it because I said it would be on the test!” Yah, I’d probably do this. But the real message it sends is “I use my tests to reinforce that you should be doing my homework for arbitrary reasons and to punish you when you don’t”

What the author realized can be summed up in the title of this post:  questions are expensive!

In my work on a test development committee, one of the first things I learned was that a standardized test  ("item") that appears on the ACT and SAT represents a substantial investment of cash and time.  It costs upwards of $1,000 a question (in fact, by some estimates, more than $10,000) to develop, vet, and pre-test a single item of the hundred-plus items that appear on a typical SAT.  In the context of test creation and administration, this fact makes test editors somewhat conservative: deciding to change (and re-vet) or throw out a test item in the late stages is actually a major financial commitment.  But what I'm suggesting here is that, as teachers, we should all be somewhat conservative about what we put on tests, because every test (or quiz, or homework, or project) item is expensive.

"Expensive how?" you ask.  The costs abound.  It takes time to write, proofread, and format the test.  It takes time for your students to do the test, time that could be spent in doing other questions or assignments or just (imagine!) having fun.  It takes time to check and grade the test, and then you have to figure out what to do with the scores and information about student performance.  Most important, every item you include on a test represents a decision not to include something else:  you can't give a class of fifth-graders a six-hour math exam.  Unless you teach the most boring class ever, the chances are that in the course of a single unit, you've had your students work on many different skills in literally dozens of tasks and contexts.  You can't rehash all of that on a test, so every item that gets on the test has pushed four or five or six more off.

So you need to be a little conservative.  By that I don't mean that you can only assign items that you've already reviewed in class, or that you should never change a test--quite the opposite.  You do need to choose your items carefully, thinking about them more as an incredibly expensive data sample -- or a trip to a very expensive gym or tourist destination -- rather than as simply a collection of objects that more-or-less mimics some of the things you've done in class.  You need to ask:

• What will I learn from doing this about what my students know?  What skills and concepts does this item assess?  How are my students likely to respond to the item, and what will I learn about my students from those responses?
  
• What will my students learn from doing this?  Is this an opportunity for them to grow and stretch in some interesting ways, or just a check that they can spit out what we've put in?  Will students come away from the experience with a better sense of what they themselves know and can do?  And will they come away with a better sense of what it is I'm trying to teach them?

My friend and mentor Diane Herrmann speaks sarcastically about the "sponge theory" of teaching:  you start the term with a dry sponge and spend the term pouring water into it.  At the end of the semester, you squeeze out the sponge into a measuring cup (graduated cylinder, whatever): the student's grade is the percentage of the poured-in water that you can successfully squeeze out.  I think that theory drives a lot of the garbage-y tests kids wind up taking--tests with 50 or 75 or 100 items to be done in 45 minutes, tests that ask similar questions again and again.  By contrast, if you think about teaching as developing a kind of mental fitness--with certain types of habits, strengths, and skills--then you realize that a test is not just a way to find out what a kid can do, it's a chance for you to provide the kid another growth experience.  And that's the real value added.

How Not to Return a Quiz

This is how not to return a quiz.

1. Walk around the room handing each student his or her quiz while everyone else is coming in or sitting down doing nothing.  Even better, wait until after the class has done your opener (or "bell ringer") and fully settled down.
2. Don't post the answers, either in class or online.  That way, students will have to follow your in-class explanations to learn how to fix their mistakes.  
3. Go over every problem that anyone got wrong.
4. When you go over a problem, make sure that you're the one giving the explanation.  If possible, give the same explanation you gave the first time.  Don't let students who got the problem right give an explanation at length, and if a student does start explaining a problem, make sure you talk over him or restate his explanation.  Don't give students who got the problem wrong the opportunity to explain their misconceptions to the rest of the class.
5. After going over a problem, don't give students an opportunity to do a similar problem.  You already have an assessment of what they know and don't need another one.

There are lots of ways to return quizzes.  Some things I've learned:

1. Get mailboxes for your room.  Use them.  Put papers in them before or after school and let kids pick up their work on their way into class.
2. Post or hand out solutions; don't go over them in class. Suppose eight students got question #3 wrong.    Of those eight, two have errors they can see immediately from your corrections, and don't need further explanation.  Of the other six, whatever it was that they didn't get the first time, they're unlikely to get by having the same explanation a second time.  Either watching someone else do the problem doesn't work for them (does it work for anyone?), or they've got some underlying misconception that made the first time through not so effective.  Whatever the cause, it's unrealistic to expect more than half of them to actually correct their error when you go over it.  So you spend four or five minutes doing something that only benefits three students.  What's the point?
3. If there's a problem that a majority of students couldn't do, briefly illustrate the main point or issue, then give a followup problem.  Or if students got wrong answers, post some popular wrong answers and have students explain what's wrong with them.
4. Anything that's important enough to talk about in class is important enough to re-assess, sooner rather than later.

Adolescence, and other mental "disease or defect"

My non-teaching friends who knew me "back when" are probably surprised that I love teaching adolescents as much as I do:  as a high school student, I often felt disconnected--even alienated--from my peers, and certainly my best friends were typically a few years older, not kids my age.  And yet here I am, loving being with kids: on field trips, I tend to sit in the back of the bus, with the students.

Part of that is probably my mild but ongoing sense of alienation from my "peers":  I'd often rather spend time with high school kids than with people my own age, partly because I don't necessarily like people my own age.  But the major shift is perspective: I'm able to enjoy the company of adolescents much more now than I did when I was an adolescent precisely because I'm not one of them.  The things that kids do that I found weird or incomprehensible or eneverating are now more curiosities than anything else.  I can understand and appreciate where they're coming from, even if it's not where I was coming from at that age.  (Every so often I'll say to kid X about kid Y, "I totally get that kid Y is driving you crazy, but ... ")  The grade-panics, the living from dramatic cliffhanger to dramatic cliffhanger, the sometimes near-total oblivion about the "big picture"--I now see these things as part of the pathology of adolescence, not personal defects.  They're not bad people, and I've come to realize that the many of the things they do and say--even to me--aren't really about me.  Those behaviors are about where the kid is at this moment.  There are some truly antisocial behaviors, but often I can do something about those.

In the big picture, adolescence doesn't bother me, because it's just adolescence.

I find myself thinking about this appreciation-with-distance as I think about teaching and working with students who have learning disabilities or mental illness.  A kid who can't get it together to get work done and in on time, who can't be relied upon to seek out extra help (even after multiple suggestions), or with whom I find myself having the same conversation for the fifth or sixth time--that kid can get under my skin.  As teachers, it's much easier for us to accommodate obvious physical disabilities.  I can teach a blind kid geometry: I provide raised-print materials and manipulatives, allow him to talk through problems that would be too hard to write out, give him a partner that can help with the manipulative stuff or describing diagrams.  (Actually, my two blind geometry students have been among my strongest, perhaps because they expend so much effort on retaining and adjusting a mental representation of the physical space around them.  But I digress....)  I know what's causing his problem, and I know what I can do to work around it.  But the etiology of these just-not-enough-effort-applied-in-the-right-way behaviors -- I don't know it, I can't see it, and I don't know what to do with it.  And that's frustrating.

I don't think I'm alone.  While the teachers at my school do a great job with students with autism, or with disabilities that come with clear-cut accommodations (written directions, access to technology or reference materials during tests, etc.), I find our hardest conversations revolve around kids who "just aren't doing it"--even when we know that those kids have processing disorders, or are struggling with major depression.

One reason is that these kids are capable of consuming an almost infinite amount of one-on-one resources, which are scarce at the best of times.  That scarcity comes from our basic teaching model:  somewhere between 20 and 35 kids together in a room, working on roughly the same thing at the same time.  (The fundamental inadequacy of this model is why I find resources like Khan Academy worth investigating and thinking about, despite their overemphasis on rote or procedural knowledge.)  So I have a choice about whether to spend fifteen minutes or half an hour with a single kid, often with no obvious results beyond the task at hand, or to spend that time doing something that will help the other 19-34 kids in the room, or the other 119-150 kids in my courses.  And so it's hard to put that time in with that one kid and not feel like I'm taking something away from everyone else.

But a more fundamental problem is that it's hard to see the student's disability--whether a learning/cognitive disability, or part of a mental illness--as a symptom rather than as a character flaw.  That same breakthrough that I've had about my adolescents' adolescence is harder to attain.  Part of that is background knowledge:  unless you've had or spent considerable time with someone undergoing clinical depression, it's hard to see how debilitating that condition can be.  But part is that we work so hard on communicating the news about these proactive student behaviors to our classes that when someone seems like they're "not catching on", we can't really understand why.  "What's the mystery?  Just get it done!" we say, although I'd never say to a kid struggling with quadratics "What's the mystery?  Just use the quadratic formula!"

Kids with these problems can be almost infinitely frustrating--I want to say "annoying", even though I know that's not fair.  And that frustration makes the whole problem harder, because it makes it harder for me to achieve that distance where I say "This is not about me, or the assignment, or even the kid.  It's about this disease."  It makes it harder for me to devote that time to that student without feeling like I'm (or he is) ripping my other students off.  And when I do that spend that time, two things tend to happen.  First, the student's behavior doesn't change much right away, and then it's hard not to take the continuing "apathy" personally:  I spent all this time on you, and you won't even meet me halfway.  Second, it's hard to know when or where to stop:  I find myself having the same conversations over and over again, without any evidence of progress.

I don't have an answer to this problem as a learning problem, but I do know this:  the kid isn't just the symptom, or the collection of symptoms.  Every kid wants to feel like a whole person.  So the one thing I can do is communicate that fact to the kid, that regardless of how they're behaving now, I still think they're a whole, valuable person; that I know they're not just this one set of symptoms; that I still love them.  I'm not sure how much saying that helps in the short term, or even in the medium term.  But over the long term, I think it's the only thing that can.

Catch-all Catch-up

I've been delinquent for a couple of weeks, but here are a few things that have crossed my path that are worth thinking about.

• The New York Times reported last Friday that EdX, the MOOC consortium run by Harvard and MIT, has created and plans to release open-source software that professors can use to grade college essays.  John Markoff's thoughtful article points out that, in addition to the obvious cost savings over "regular" TAs, computer-grading would make it possible for professors to assign more writing and for students to resubmit papers multiple times, possibly increasing learning.  He quotes the usual cries of "Pattern recognition is different from grading!" (not according to the Church-Turing Hypothesis, but I digress), but then closes with the following astute observation:  

"Mark D. Shermis, a professor at the University of Akron in Ohio, supervised the Hewlett Foundation’s contest on automated essay scoring and wrote a paper about the experiment. In his view, the technology — though imperfect — has a place in educational settings.
"With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation’s best universities, where the level of pedagogy is much better than at most schools.
" 'Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could,' Dr. Shermis said. 'There seems to be a lack of appreciation of what is actually going on in the real world.' "

• On first take, Sir Ken Robinson's famous TED Talk about creativity in education can sound like a standard "Infuse more arts into the curriculum."  But reading his book,  Out of Our Minds: Learning to Be Creative gives a different story.  "At the same time, other disciplines, including science and mathematics can be just as creative as music and dance. Creativity is possible whenever we’re using our intelligence."  I'm really enjoying the rest of the book -- it's smooth, a quick read, lots of fun, and very thought provoking.

• On a related note, John's former colleague Zach Herrmann writes in his blog that it's not just what we teach but whether we teach it in a way that fosters creative, actual thought:  "How often do we as teachers deprive our students of the excitement of learning by the way we ask our students to learn within our classrooms? I believe we can positively impact students’ perceptions of learning by being mindful of the problems we give and the way we ask them to participate, while rethinking our role as a teacher in their learning process."

That's all for this week!

Quick thought about Kahn

I am looking at the Kahn discussion from the outside since I have been out of the classroom for several years. I admit my first look was one of dissapointment as it does appear to be procedural. It does a good job of procedural, however. They seem to get the math right. Were it riddled with errors, it would deserve severs criticism.

I am once again impressed by P.J.'s take on this and it got me to thinking about print paterials that are similar.

I am willing to bet, if I beleived in betting,  that most of you who are over thirty posses(ed) copies of Schaum's outline for something.  Calculus was a big seller. There was not much more than worked out examples, but many students found tham very helpful. I suppose they still exist. They served a purpose. I think there was a similar product for novels, Cliff's Notes, I think. Cliff's Notes presented a summary of books like War and Peace in thirty pages.  This was certainly not the same as reading War and Peace, but it did help some students wade through the novel and keep track of who was who and what was going on. Scahum's did the same thing for math and science. No one that I know ever used one as a replacement for a textbook, but thousands of students learned important procedures from them. 

Am I too far from being accurate to say that Kahn Academy presentations are a digital version of Cliff's Notes/ Schaum's outline, or am I missing something important? 

By the way, I found Scham's usefull as a source of worked out examples. I could give my students one of their problems to work without having to create one and work it out to makes sure it "worked out nice".   

What value Khan?

It's almost become a party game among my math educator friends to talk smack about Khan Academy.  "The lessons are just procedural!" (not always true--I've seen some conceptual explanations).  "There's no effort to build in the Standards for Mathematical Practice!" (Mostly true.)  "Some lessons reinforce common underlying misconceptions." (I haven't seen them, but it's plausible.) And so on.

What follows is an open letter to those friends of mine -- and superstar math educators around the country -- who take these positions.  I don't think they're wrong.  But they are short-sighted.  Read on to find out why.

Friends--

At the risk of stating the obvious, you aren't  run-of-the-mill math teachers.  Of course you can envision--indeed, give daily--lessons that are more in-depth, challenging, authentic, inquiry-based, etc., than Khan Academy.  Indeed, I would be shocked if you couldn't.

But that's not the question.  Khan Academy wasn't created for yourstudents.  It was created for kids whose teachers, in many cases, don't even know the content, much less how to present it clearly or explain it well.  Have you been to the elementary schools in my district?  Because (as many of you know) something like half the freshmen who come to my school can't use a protractor to measure an obtuse angle --- they tell me it's 61 degrees or something cockamamie like that --- and they TOOK A TEST to get into my school (indeed, the cutoff score for my school is over 800 out of 900 possible points; we rejected more than 2000 kids out of the 2400 who applied).  Those kids will get more effective instruction from Khan Academy than they can get in a regular classroom, because right now they aren't getting effective instruction in their regular classrooms, period.  (I'm not blaming anyone in particular here, simply making the tautological claim that instruction that doesn't result in kids being able to do the things they are being instructed in how to do is, by definition, not effective.)

It works for other kids too: my daughter was far ahead of her class last year, and for the first half of the year did worksheets in the back of the room.  For the second half of the year, she and two friends got to go on Khan Academy and pick their own lesson every day, and she grew more (as measured by NWEA/MAP scores) in that semester than in the previous 1.5 years combined.  And she got about a quarter of the way through a standard Algebra I course.

Finally, I'd say this--about flipped classroom stuff generally and KA in particular.  Right now, I'm cooking up a pot of Cincinnati Chili (mmm...can you smell it?).  It's delicious, nutritious (yay low-fat turkey!), and my kids love it.  But there's a diner down the street from my house, and any day I want, I can go there and get a reasonably tasty, reasonably healthy meal at a reasonably low price.  And so every week or two--when I'm too tired, or we have nothing in the refrigerator--we go there for dinner.  It's not Tru, or Topolobampo, or any of the other great restaurants Chicago is known for--but it's a reasonable way to get fed once in a while.  I think KA and other online videos are like that:  not as good as the best (although maybe if you watch the first lecture of the Udacity physics series, on Eratosthenes' measure of the circumference of the earth, you'd be surprised).  But KA delivers reasonably clear, correct instruction to people who might not otherwise have access to it.  Friends who have expressed skepticism about the "All Khan, All the Time" approach:  I agree wholeheartedly.  Let's give our kids a balanced diet of different kinds of instruction and different ways of thinking about problems.  But I don't think that's a reason to trash on Khan altogether.

What's Wrong With Grade "Inflation"?

At dinner last night, I was talking to a friend involved in education, and she was pressing me hard on what she perceived to be grade inflation at my school.  I could have argued the facts more vigorously:  the kids she meets are applicants for an ultra-elite college (she's an interviewer), so when they say "I'm getting A's and so are my friends," that's hardly a representative sample of the class.  But I admitted that our bell curve is centered on a B--probably a high B--rather than on a C.  And then we started talking past each other: her interpretation seemed to be that, because our kids are really smart and do their work, our attitude was something like "They probably deserve A's, so why not just give them A's?"  Her response to this hypothetical motive was to ask me "What do you do to differentiate students?"

In fact, it's more complicated than that, and at 24 hours remove, I feel more clearly the need to challenge the entire premise of her argument.  (And, to be fair to my friend, this is a common argument.  So even if I've misattributed it to her, it's an argument worth discussing.)  You see, my goal isn't primarily, or even partially, to differentiate students.  I understand that that's something colleges wish I would do, that the entire college admissions system depends on using grades (and test scores) as differentiators.  But I'm a teacher, and so my goal is, primarily, to teach. And at some level, that's the opposite goal.

At the beginning of my course, I have to figure out two essential questions.  The first--really the a pair--looks at the present and immediate past:  what do my students know and what can they do?  The second pair looks towards the future: at the end of my course, what do I want my students to know, and what should they be able to do?  If I'm waxing philosophical, I add a third essential question:  ten years from now, what do I want them to retain of the experience of having been in my class?

Once I've articulated my standards, students who meet those standards get good grades:  A's and B's.  In fact, when my students get mostly A's, I generally feel like I've done a good job:  it means that I've gotten most of my students to master all or almost all of my standards.  When my students get C's, D's, or F's, that's  supposed to tell them that there's room for improvement.  And it tells me that there's room for me to improve, too: because especially when you're working with children, you can't take their attitudes and behaviors as a given.  If a kid struggles and doesn't do homework, I wonder what I could do to convince him or her that the homework is worthwhile, that time spent doing these problems will be fruitful in some crucial way.

Now I'm not saying that every time a class gets all A's, everything's hunky-dory.  Sometimes it's a sign that the time has come to raise standards, to demand more of students.  If you have kids who are currently scoring at the 40th percentile, then when they get all A's, you have external evidence that they have room to grow (that 40th percentile score), and internal evidence that they have the capacity to do it (they're doing everything you ask and are being successful).  So then you raise standards.  The same might be true if your kids are scoring at the 80th percentile, or even at the 90th--it depends on what your overall goals are.  You might change your standards:  spend less class time on the stuff they're clearly mastering, and add in projects or more exploratory work.  (Some of these changes might decrease test scores but reflect important long-term goals that standardized tests rarely assess.)  Partly it depends, too, on what it takes for your students to meet those standards:  are they getting A's easily, or are they doing multiple retakes, asking lots of good questions, etc.?

Where do classroom standards come from?  In many cases, the common core, or state directives.  In the case of the class I'm teaching, they come from my reading of comparable classes taught at the University of Chicago and the University of Illinois.  I have external validation--from those schools--that the things I want my kids to be able to do are reasonable goals for honors first-year math majors.  I have internal validation that few, if any, of my students find the coursework easy;  they all make some mistakes, and many come back for multiple retakes of my quizzes and tests.  So yeah, if at the end of the semester almost all of my kids can do those things, then almost all of my kids will get A's.  I'm not going to run around trying to ratchet up standards and lower the number of A's.  I'm going to be glad that they, and I, have done our jobs.  And if colleges can't tell the difference between them without actually reading the two-page single-spaced recommendations I write for the majority of my students--well, that's their problem.