Flipping Over

We are done. The course is over, except for the data analysis, which I expect to entertain me over the summer. This last entry will be a collection of random topics - things that happened over the course of the final exam and grading or observations I've made.

End of year party

Long ago, in 1993, we started the tradition of having an ice cream bash after the take home final. I don't believe we've maintained the tradition, but I decided to resurrect it. In addition, we continued the tradition of class T-shirts. The winning design was:

In addition, since I had artwork for each group, I made that available to students to construct their own team patch, which they could then iron onto their shirts. Groups who didn't have the time to do anything fancy (21 of 22 groups), got the plain hand-drawn black and white animal.

So, on Wednesday afternoon after spending a lot of time on the final, we all gathered in a much too small room to eat Toscanini's ice cream, collect T-shirts, and iron on patches. It was a total blast. Pretty much everyone ironed their patches. The one group who augmented their animal was the marmots, who added color and appropriate attire:

We had a few ironing disasters which prompted a return to QRSTs for a few additional shirts.

Had I been thinking, I would have tried to get a group shot, but I didn't. Next time! One of my students, did take this small group picture:

Students seemed relatively happy and relaxed, although a few still had multiple final projects to complete. Much ice cream was consumed. The pear chardonnay sorbet was not consumed, but I have to say, it was outstanding (it came home and disappeared stunningly quickly). It was a good end to the course!

The final

For the curious, here is the final exam. I thought it was pretty tough; it was certainly time consuming to grade (it took me the better part of a week to grade 45 exams -- I'm guessing over an hour per exam). The students did exceptionally well -- a typical Margo-exam ends up with a median around 67%; this one had a median and mean over 80%!

The first problem was a design exercise and those are notoriously difficult to grade; after two days of grading I wondered why I continued to give these kinds of questions. Then a colleague, Kryzstof Gajos, explained how we graded his final exam, and I am excited to contemplate how I can apply this technique. I've invited him to write a guest entry on the topic, but in the meantime, here is my understanding.

The course he was teaching is called CS 179: Design of Usable Interactive Systems. It is very much a course all about design. So for the final exam, he split the period into two parts. During the first part, students took the (written) exam using a "uniquely" colored pen (provided by Professor Gajos). Then, he collected the pens and the class collectively graded (their own) exams. Thus, in real time, the class discussed the problems, the supposedly right answers, alternative answers, etc. When there was ambiguity, they discussed it. Afterwards, he collected the exams, had the TFs check that the self-grading was accurate and he was done.

This was pure brilliance! I'm not saying that because it reduces grading time (which it does), but because it turns a purely summative evaluation process and makes it partially formative. I try to do this when I grade exams -- I essentially engage in a typewritten discussion about their answers (which is one reason exams take so long to grade). I always have second thoughts about this, because I know that students merely search for the final number and ignore everything I write. Or so I thought (more on this later). The idea of discussing the exam right after they've finished it and being able to address confusion immediately is overwhelmingly appealing to me. It's not entirely clear how to apply this to my exam -- I like pushing students to think about complex issues for which mulling the ideas and questions over is useful (this is why I like the takehome exam format). I don't know how I'd fit both the testing and the grading into a standard 3-hour test slot. If I stick with the 24-hour format, then I'd have to make the exam be, say, a 20 hour exam with a scheduled 2-hour class which sounds like a logistic nightmare. I'm not sure what I'm going to do here, but I see further experimentation in my future.

In addition to a design question, I asked people to read a research paper ( the Barrelfish SOSP paper), and then I asked them questions on it. I'm really looking forward to having some of these students in my graduate OS class, because the high bar they set for a system is quite interesting. Many were accepting of a prototype, but a surprising number took a more business perspective, while perhaps not appropriate for a research paper, indicated to me a much better sense of how the world works than many tech-savvy people are. I found it fascinating.

In another question, I asked them to respond to a (rather poor) technical suggestion, but I asked them to do it in the context of an email to an engineer who worked for them (I put the students in the role of being the project architect). I was quite impressed with the care most students took in crafting a reply that was both technically correct as well as polite.

The post final

Although I write fairly long comments on my students' exams, I know deep inside that they never read them, and yet I continue to do it, because I know that there is the potential for learning there. This year, I was pleasantly surprised -- for the first time in twenty years, I had a couple of people engage with me after the exam to discuss answers. They weren't looking for points but were instead, trying to make sure they understood what a right answer to a few questions would look like. In one case, a student wrote up about 5-6 paragraphs as an answer to one problem to make sure that the student had a complete grasp on the answer. I was (pleasantly) astonished. While I can't make any causal statements between new pedagogy and this behavior, I could certainly believe that because I am much more closely engaged with students, they felt more comfortable doing this. In any case, I was quite pleased.

The end

And so, one woman's adventure with flipping comes to an end. I'm looking forward to crunching data, and I'll certainly post here if/when I find something interesting to say. In the meantime, here are my parting thoughts.

• It's good for an old dog to learn new tricks.
• Flipping lets me spend time with those students for whom the material is challenging.
• Learning takes place by doing, not by listening to me.
• TF engagement is critical.
• It takes a lot of effort to come up with effective in-class work.
• Pre class web forms are AWESOME. They let me engage with students in an entirely different way and to gather lots of interesting data.
• CS161 is even more time intensive than I thought.
• It would be useful to help students learn what it really means to design something.
• Flipping is a great equalizer when students enter with different experience levels or exposure to different topics.
• Fully integrated and coordinated materials take real effort but pay off tremendously.