Flip N-1

The penulatimate week of the semester was a busy one! I have some fun data on how the students view assignments, some experience with the partially-flipped classroom, an interesting discussion of a flipped exam, and finally the totally awesome creativity of my students as expressed via T-Shirt designs.

The Pain/Gain Scale

One of our former PhD students is now on the faculty at Swarthmore and he suggested to me the pain/gain scale as a way to determine if the degree of difficulty in assignments was perceived to be worth it. This seemed like a fine idea, so I polled students on the two major assignments completed so far, asking them to rate on a scale of 1-5 the amount of pain the assignment caused and the amount of gain they derived. The results are fascinating.

Here is the histogram of the responses:

At first I thought that Assignment 3 was really nice correlated, but then I calculated the ratio for each student and graphed that -- it tells a different story:

While most people found the ratio about 1:1, there were some who clearly suffered and others who had a blast. I have not done further analysis, correlating these ratings with how well groups did, but I found it interesting nonetheless.

The Partially Flipped Class

The astute reader will have noticed that while upholding my deal that I will not make students prepare for class while they are also implementing processes, building a VM system, or making a file system recoverable, it's been difficult to maintain the energy and enthusiasm of the flipped classroom. As I entered the homestretch, I decided to try an experiment with what I am calling the partially flipped classroom. I take my lecture notes and I break them into two parts -- a short intro (6-8 slides) and the rest (10-20 slides on technical content). I present the intro and then I dispatch the class for 15-20 minutes deriving from first principles some of the challenging aspects of the particular topic and approaches for addressing those aspects. For example, on Tuesday I was teaching virtualization. Now that they had built major pieces of an operating system and become intimate with other parts of the OS, I let them figure out what the big problems were in building a virtual machine-based environment. The groups each came up with excellent lists, which we then pooled. When this was all done, we'd covered much of the material in the rest of the slide deck. So, I was able to breeze through the rest of the deck relatively quickly and I knew that they had a much more complete picture of the topics, because they had largely derived them themselves. It felt very good. I repeated the process on Thursday with security. I introduce the three A's of security: Authentication, Authorization, and Access Control and then let them play around with identifying what can go wrong if you fail to address each aspect properly and then proposing approaches to solving it. This week, we'll be able to breeze through my slides on the topic relatively quickly, because most of the ideas came up in discussion. That will leave us time to talk about a few real world exploits.

I think this is the approach I want to try for all the material that I had been teaching "the old way." I'm pretty sure that with some careful thought I can make it work and we may be able to maintain a higher level of energy and engagement during the roughest parts of the semester. We shall see!

The Flipped Exam

Thanks to the wonders of Facebook and my friend Jill Levien, I was introduced to the idea of a Flipped Exam. Now, I don't teach game theory and I'm not entirely sure how this would work for my class, but I found it intriguing, so I posted it to our class discussion board with the directions "Read/Discuss." And my fabulous students engaged!

Student1: We should have such a final exam: the entire class: write
as much as possible of an operating system in 24 hours together.
editdelete

Student2: While I do think there is a place for individual examinations,
the current educational system overvalues such tests. My basic
understanding of the "real world" is that most projects of any real
significance are done in teams, be it research, commercial products,
etc. Sure, there have been cases of lone wolfs that create paradigm
shifts or fantastic projects (and this should be encouraged as
well!), but teamwork is ultimately a skill that should be encouraged,
not subdued. (24 hours might be a bit short to get much functionality
done on an OS even with a full class of people, but I certainly
wouldn't mind a collaborative written final).

Student3: Synchronizing >40 people would be so much fun...

Student4: I really like that idea. It will be a fun, collaborative
experience, and we'll end up synthesizing the entire semester
together into one glorious product.

Student5: This idea was the first thing I thought about as well. I
think it would be fun, maybe not necessarily writing a whole new
operating system, but writing another subsystem for OS/161.

TF1: Rob Bowden: Networking!!

Student5: Indeed that would be fun.

Student6: so punny

Student7: You get a synchro bug! And you get a synchro bug! Everybody
gets a synchro bug!

Student8: I find this much more reflective of how the world actually
works and I think it reminds people that the best answer is 99% of
the time not produced in solitude.

I'll second Student2's final suggestion as well.

Student9: If it's a coding assignment, maybe we'd need (want) groups
of a smaller size. 45 people can't all be coding at the same time,
so it seems as if the exercise would likely devolve into "Okay, you
two/three/four code, we'll just watch...and write a DesDoc? And,
uh, run tests!"

What about five partitions of nine, sorted day-of? You've got 24
hours to do what you can on [subsystem], working with some, all,
or none of your tablemates. Coordinate sleep, food, and synchronize
your watches...here's an assignment spec, go!

Right now I think I'm still going with the individual exam, but I'll definitely have to think about the concept of collaborative exams for the future. I am glad the class (or at least a subset of it) thinks that working together would be fun.

T shirt Designs

Even in the midst of a pretty heavy workload, my students know how to have fun. Check out the T-shirt designs on our home page. I'll let you know which design won after the T-shirts have been delivered.

Next: Flipping Over (May 24, 2013)

Flipped Out

It's Friday, April 19 -- my mother's birthday; my colleague's birthday.

I sit at home, working from the relative safety of a town six or seven miles way from the mayhem that erupted in the greater Boston area last night. As is old news by now, within roughly 48 hours of the Marathon Bombings, FBI and police had identified two "persons of interest" who became suspects last night after engaging in a robbery, a shooting, a police chase, and explosives. As of right now (8:13 AM on Friday, 4/19), one of the suspects is dead; the other is the target of an enormous manhunt that has much of the greater Boston area on lockdown. All the local universities are closed; my kids' schools are closed; people in six towns are being told to stay home and leave doors locked. I've certainly never experienced anything like this, and I'm guessing the vast majority of my students, friends, and acquaintances haven't either. So, as I did Monday, I turn to blogging.

The end of the class is in sight! Because many groups took late days on the virtual memory assignment, we pushed the design reviews for assignment 4 (make a file system recoverable by adding journaling) a few days. This gave students a much needed bit of breathing room and as a result they came to class Tuesday pretty energetic.

Rather than continuing with new material, I made Tuesday a flipped class. They had a few short pre-class questions to make sure they'd read the assignment and understood what they needed to do. We then spent class time letting them code read to figure out how the existing file system worked, what kinds of things were going to need to log, and how they might recover those log records. On one hand, we could have done a structured code walkthrough, but I'm pretty convinced that letting them do the walkthrough in small teams results in a better understanding than a guided walkthrough. (Although I'm beginning to wonder if we shouldn't offer code walkthrough sections each week ... that might be a really nice supplement. Or perhaps we could make a series of audios to walk through the code ... oh I'm liking that idea a great deal.)

We spent the first chunk of class time letting students merge new code into their trees. My theory is that this would be a relatively quick process and anyone who ran into trouble would have immediate assistance. I'm not sure that part worked very well. Some students got the merge done quickly, but others spent a significant chunk of time on it, even though they didn't need our help.

They were then supposed to pick a couple of operations and walk through them to figure out what they'd need to log and what recovery would look like. Some groups found this useful; other groups felt lost. Even for those who felt lost, my sense is that it was a decent use of time, as the students really need to understand the process of figuring out how to determine what needs to get logged and what needs to be recovered.

This class was followed up by the peer design review. The room sounded very animated during the review and I heard a lot of good discussions. I think the extra days after A3 really paid off here. We need to work with the timing to make the A3 peer design a more useful endeavor. That's going to require some serious rethinking about what we do in-class versus outside of class.

All in all, this week was a classic flipped classroom and everyone seemed much happier. I also took the opportunity to snap photos of my students with their group placards. For those of you who've never been involved in Harvard's CS161, we ask each group to select an animal as their team name. We interpret the term animal quite broadly and we've had everything from unicorns to meowbears in past years.

This year, I had someone who shall remain nameless draw each team's animals and then we printed them on bright yellow paper, laminated them, and but them in stands to assign groups to tables. With permission from the people photographed, I bring you a subset of the images of 2013 CS161!

The Blobfish

The Anteaters

The Caribouyah

The Centaurs

The Falcons

The Sphinx

The Squids

The Baby Velociraptor (singular)

The Dingos

The Koalas

Next: Flip N-1 (April 30, 2013)

Anxiously Flipped

It is 4:57 on Marathon Monday, and by the time you read this, I'm sure you all know what transpired today in Boston at the finish line of the Boston Marathon. I am grateful to be safely home and am at my keyboard waiting for all my students to check in. As Harvard has classes today, I am optimistic that my students were all in Cambridge, safely out of harm's way. Even so, I'll be happier when I see their messages. If only everyone else would be so lucky.

I figured that blogging might be a distraction from the day's events, but nonetheless, my suspicion is that this will be short and perhaps scattered.

It was a rough week to be a CS161 student. The virtual memory assignment was due on Friday and it is a particularly devilish assignment -- the number of lines of code to write is not huge, but getting the synchronization correct is just downright hard. This sets the students up for a lot of late night debugging, perhaps an all-nighter, and then perhaps a couple of late days. This has all played out reliably over the past many years.

So, this week's status has much less to do with flipping and much more to do with CS161, virtual memory, and why anyone would put themselves through this. For your reading pleasure, I bring to you a Facebook discussion that took place after I posted a note about how guilty I felt coming in at 8:00 AM and finding my students (surprisingly happily) working away, having been up all night. Let me just say that I never looked that happy after an all-nighter.

My original post: There is nothing quite so guilt-inducing as walking into your office at 8:00 am and seeing your students sitting in the lounge working on your problem set after having been working on it all night. Oh wait -- there is -- seeing multiple groups later that day around 4:00 PM -- all of whom have been up all night and are still working. Many of them now have operating systems that mostly support virtual memory, though!

In the following, I have converted names to initials, to provide some shred of anonymity.

• GK: That last bit, "mostly support virtual memory," reminded me of this and this.
• MO: that's guilt-inducing? I thought that's what you live for!
• RH: And yet, it didn't keep you from giving them the assignment in the first place. Hmm! Actually, IIRC, that was the easy assignment. It was the stupid filesystem one that almost killed me. I wonder if these assignments would still seem as hard today as they did 15+ years ago when I did them (am I really that old?).
• GK: Tell them that in 24 hours Andy Sudduth finished his operating system, finished putting together his HeathKit computer (this was in the early-to-mid 1980s), wrote a full screen editor, and then wrote a paper due the next day in his editor on the computer he built running the OS he wrote. Or, you can tell them what I try to tell myself—start sooner. Editorial note: Yes, Andy Sudduth, Harvard legend and Olympic rower is also a graduate of CS161
• Me: We've tried many, many things over the years to attempt to make the problem sets less time consuming, but at the end of the day, learning to build and debug asynchronous, multi-threaded programs is just plain hard. And we continue to get the feedback, even years later, that the time is well spent. I have an awesome class of 45 who, although exhausted, still seem deeply happy with what they are doing.
• GK: And since I mentioned Andy, rest his soul, he also loved that course.
• PA: I have fond memories of programming such things when using an ascii terminal to an under-powered VAX11/750 running BSD4.2 with it crashing every now and again... your students should be happy their development environment is pretty stable!
• RH: @Margo: I agree. It was a good class. It retains, however, the distinction of being the only 40-hour-a-week class I ever took. Second place was about half that. @GK: If those students are anything like me, not starting soon enough wasn't the problem.
• MO: it's no fun without an all-nighter once in a while
• MO: Of course, I never took the course, just had the enjoyment of watching BF and others suffer through it
• MW: Wow, 45? That's awesome.
• PR: It's a public service. Think of of the trouble those students could be getting into if they weren't locked up building an OS.
• NM: I wonder how many techies at the likes of Google, Facebook, etc. would still get a chill down their spine at the mere mention of "assignment 3"?
• DK: I still fondly remember our all-nighters in CS161 (with CY). Professor Hsu said we built the class's most elegant OS (everything was based on a common semaphore implementation). And it always ran for at *least* 30 seconds before crashing.
• CY: I remember the class being a 9 to 5 affair. PM to AM, as that was the only time M, D, and I had in common. And I fear that the elegant semaphores were woefully slow--the guys that used atomic ops to access the priority queue were twice as fast as our version where the semaphores also managed priorities.
• DK: I think that was also the class where we wondered "what happens if you tell the unix shell while (1) fork() ?" and took down the department compute server for the night.
• CY: I think it was only microvax-6.husc.harvard.edu, out of of a fleet of 10 or so microvaxen, and not the file server (thank goodness). The Magican's Apprentice moment was priceless, though, as we tried and failed to kill the parent process and started getting "no more processes" messages from our other shells. The next morning, the sysadmins were more worried that we were doing some important computation than about our abuse of the machine--they just power cycled it and everything returned to normal.
• CY: Actually, the real thing I should say about Prof. Hsu's CS161/261 class is that, of all the hacking classes I took at Harvard, it was by far the most useful in the rest of my career. I think it was the construction of synchronization primitives in a raw environment and figuring out how to debug code built on top of those primitives--there's nothing like it for combining power and danger, and for whatever reason being able to build such things remains a rare skill.
• DK: I think the "whatever reason" is that we have done such a good job of creating layers that shield students from the "raw environment"---students building a web application in Django have no idea how many layers of complexity are hiding between them and the hardware, and wouldn't know what to do down there. The hard question is whether this is a good thing or bad----given that we have already built the layers, do we want to force every student to really understand them, or is it enough for them to use them?
• And to make my colleague MM happy, I won't cut this one out DK: As for most useful classes, I've got to flag Umesh Vazirani's CS124, at least to the extent that "useful" means "determined where I am now".

And to top it all off -- this week's picture, straight from China: Tao Stein's students at midnight working on their ray tracer.

Next: Flipped Out (April 22, 2013)

Missing Flipping

Since I'm back in more conventional lecturing, there isn't much newsworthy to report, so I want to highlight just two things.

First, I recently learned that one of my students is blogging the course! I was thrilled when she said I could link to her blog. She is far more poetic than I, and I love reading her entries.

Second, I was chatting with a colleague recently who said that she couldn't tell whether I liked flipping or not. I thought I had been making that clear, but just in case, the point isn't coming across. I'm going to rant for a bit on just how excited I am about flipping.

I am completely sold on the flipped classroom. I will talk at great length to anyone who will listen; I recently used flipping as the centerpiece in the session I did for our teaching practicum; I have yet to hear a compelling argument for why you would not want to flip (as long as you could figure out space issues for a large class). I believe that I have covered all the reasons I am so enthusiastic about it, but just to put them all in one place:

• The teaching staff can focus time and energy on those students who need it the most rather than those who are most vocal.
• Pre-class work gives me concrete data on how students are doing -- what they know, what confuses them, what they don't know.
• Because the pre-class work is built into the class structure, I have a mechanism to obtain immediate feedback on most everything. I believe you could do this in a more conventional class as well, but it falls out naturally here.
• The pre-class work also gives me a way of staying connected with each student. I regularly check in on how partnerships are doing. We had one pair on the brink of divorce, but it appears to have been salvaged. I will point out that I have never taught this course without having at least one divorce, and it currently appears that we will have no divorces, even though we have more students than we've had in a decade.
• The physicists demonstrated that peer instruction works, and this confirms it. The students learn a lot from each other.
• Flipping can be the great equalizer -- if students already know half the material, they can skip the audio decks on those topics rather than having to sit through a lecture on them, just to get what is in the second half of that lecture.
• Preparing the coordinated materials makes me think much more deeply about what I'm teaching, how I'm teaching, and why I'm teaching. The end product is therefore better thought out.
• I've discovered that one becomes very sloppy while lecturing. You can walk into a class with lecture notes and wing it. However, I cannot wing recording an audio track. Because I can revise each slide track by itself, I will re-record until I have a very crisp, tight, clear explanation. I think this is a big win.

All that said, flipping, like any other mode of teaching, can be done badly. I've certainly experienced that. I believe that the work to be done outside of class, the pre-class quizzes, and the in-class work must be tightly coordinated. I also believe that the instructional staff needs to be fully engaged during class.

Next: Anxiously Flipped (April 15, 2013)

Inadvertent Flip

We're back after spring break, and it does appear that most of my students took a much-needed break. We kicked off our first week back with our second peer design review and then on Thursday we dug into file systems. So, today's post will revolve around two topics: observations from the peer design review and a discussion about class participation.

Peer Design Review

This was our second peer design review and I was struck by two things. First, it seemed that it took longer for the students to read through the design doc of their peer group. I speculated that perhaps the students had started writing more in-depth detailed designs, so I decided to ask about that in the week's web work (which are just short surveys at this point). Turns out that this was not what I was seeing -- about half the class claimed that the A3 design they reviewed was about the same quality as the A2 design they reviewed. And the other half were split as to which was better (there was a small tilt towards the A3 designs being better, but nothing huge).

The second thing I noticed was an overall spirit coming out of the discussions. They seemed genuinely happy to be talking about fault handling and page allocation. They also seemed surprised that when they asked the staff certain questions, they got multiple answers, "I did it this way." or "Oh yeah, I didn't do it that way, I did it some other way." I thought this was good -- there are rarely right or wrong answers in design but instead a set of tradeoffs to be considered. And seeing that the staff didn't necessarily agree but could engage in a discussion of tradeoffs and relative merits was valuable in and of itself.

Participation

I have always graded on class participation, believing all the research indicating that people learn better when they are engaged. I know that class participation is difficult for some, and I always let students know that they can earn participation points by sending email before or after class to discuss things that they found either challenging or interesting. At the same time, I tell those students that my goal is that by the end of the semester they do feel comfortable speaking up in class. And nearly every semester, they get there, and it's a wonderful thing for all of us.

Anyway, I always have students who never speak, never email, and never say anything about it. I've come to accept that as normal. Now, this year, I figured that participation grades were easy to come by because I consider web work and small group work in class part of participation. So I didn't worry too much about even my quietest students, because as long as they were in class, there was no way for them not to participate. Imagine my surprise when in the very same week, two of my students brought up the topic of participation. I was shocked. I'm convinced that this never would have happened if I'd been teaching this as a conventional lecture, but somehow these students, who were doing everything asked of them, somehow felt that they should be engaging more.

I assured both of them that from a grade standpoint everything was well, but I used it as an opportunity to engage in the greater discussion about how to participate, what I could do to make it easier, why participation was a good thing, etc. I'm curious to see what happens, but I was so (pleasantly) surprised to be able to engage in these discussions that I had to write about it.

By the way ...

I just want to say, "My teaching staff Rocks!"

Next: Missing Flipping (April 8, 2013)