An Index to my Flipping Blog Postings

I've had multiple requests for a more convenient format in which to read about my experience flipping CS161, my undergraduate operating systems class, that I've put together this index page. Enjoy.
One Woman's Adventure with a Flipped Classroom (February 4, 2013)
Flipped Again (February 7, 2013)
Not Flipping Out (February 16, 2013)
Addicted to flipping (February 21, 2013)
Unflipped! (March 4, 2013)
Half Flip (March 9, 2013)
Flipping and Testing (March 18, 2013)
Inadvertent Flip (April 1, 2013)
Missing Flipping (April 8, 2013)
Anxiously Flipped (April 15, 2013)
Flipped Out (April 22, 2013)
Flip N-1 (April 30, 2013)
Flipping Over (May 24, 2013)

Anxiously Flipped

It is 4:57 on Marathon Monday, and by the time you read this, I'm sure you all know what transpired today in Boston at the finish line of the Boston Marathon. I am grateful to be safely home and am at my keyboard waiting for all my students to check in. As Harvard has classes today, I am optimistic that my students were all in Cambridge, safely out of harm's way. Even so, I'll be happier when I see their messages. If only everyone else would be so lucky.

I figured that blogging might be a distraction from the day's events, but nonetheless, my suspicion is that this will be short and perhaps scattered.

It was a rough week to be a CS161 student. The virtual memory assignment was due on Friday and it is a particularly devilish assignment -- the number of lines of code to write is not huge, but getting the synchronization correct is just downright hard. This sets the students up for a lot of late night debugging, perhaps an all-nighter, and then perhaps a couple of late days. This has all played out reliably over the past many years.

So, this week's status has much less to do with flipping and much more to do with CS161, virtual memory, and why anyone would put themselves through this. For your reading pleasure, I bring to you a Facebook discussion that took place after I posted a note about how guilty I felt coming in at 8:00 AM and finding my students (surprisingly happily) working away, having been up all night. Let me just say that I never looked that happy after an all-nighter.

My original post: There is nothing quite so guilt-inducing as walking into your office at 8:00 am and seeing your students sitting in the lounge working on your problem set after having been working on it all night. Oh wait -- there is -- seeing multiple groups later that day around 4:00 PM -- all of whom have been up all night and are still working. Many of them now have operating systems that mostly support virtual memory, though!

In the following, I have converted names to initials, to provide some shred of anonymity.

• GK: That last bit, "mostly support virtual memory," reminded me of this and this.
• MO: that's guilt-inducing? I thought that's what you live for!
• RH: And yet, it didn't keep you from giving them the assignment in the first place. Hmm! Actually, IIRC, that was the easy assignment. It was the stupid filesystem one that almost killed me. I wonder if these assignments would still seem as hard today as they did 15+ years ago when I did them (am I really that old?).
• GK: Tell them that in 24 hours Andy Sudduth finished his operating system, finished putting together his HeathKit computer (this was in the early-to-mid 1980s), wrote a full screen editor, and then wrote a paper due the next day in his editor on the computer he built running the OS he wrote. Or, you can tell them what I try to tell myself—start sooner. Editorial note: Yes, Andy Sudduth, Harvard legend and Olympic rower is also a graduate of CS161
• Me: We've tried many, many things over the years to attempt to make the problem sets less time consuming, but at the end of the day, learning to build and debug asynchronous, multi-threaded programs is just plain hard. And we continue to get the feedback, even years later, that the time is well spent. I have an awesome class of 45 who, although exhausted, still seem deeply happy with what they are doing.
• GK: And since I mentioned Andy, rest his soul, he also loved that course.
• PA: I have fond memories of programming such things when using an ascii terminal to an under-powered VAX11/750 running BSD4.2 with it crashing every now and again... your students should be happy their development environment is pretty stable!
• RH: @Margo: I agree. It was a good class. It retains, however, the distinction of being the only 40-hour-a-week class I ever took. Second place was about half that. @GK: If those students are anything like me, not starting soon enough wasn't the problem.
• MO: it's no fun without an all-nighter once in a while
• MO: Of course, I never took the course, just had the enjoyment of watching BF and others suffer through it
• MW: Wow, 45? That's awesome.
• PR: It's a public service. Think of of the trouble those students could be getting into if they weren't locked up building an OS.
• NM: I wonder how many techies at the likes of Google, Facebook, etc. would still get a chill down their spine at the mere mention of "assignment 3"?
• DK: I still fondly remember our all-nighters in CS161 (with CY). Professor Hsu said we built the class's most elegant OS (everything was based on a common semaphore implementation). And it always ran for at *least* 30 seconds before crashing.
• CY: I remember the class being a 9 to 5 affair. PM to AM, as that was the only time M, D, and I had in common. And I fear that the elegant semaphores were woefully slow--the guys that used atomic ops to access the priority queue were twice as fast as our version where the semaphores also managed priorities.
• DK: I think that was also the class where we wondered "what happens if you tell the unix shell while (1) fork() ?" and took down the department compute server for the night.
• CY: I think it was only microvax-6.husc.harvard.edu, out of of a fleet of 10 or so microvaxen, and not the file server (thank goodness). The Magican's Apprentice moment was priceless, though, as we tried and failed to kill the parent process and started getting "no more processes" messages from our other shells. The next morning, the sysadmins were more worried that we were doing some important computation than about our abuse of the machine--they just power cycled it and everything returned to normal.
• CY: Actually, the real thing I should say about Prof. Hsu's CS161/261 class is that, of all the hacking classes I took at Harvard, it was by far the most useful in the rest of my career. I think it was the construction of synchronization primitives in a raw environment and figuring out how to debug code built on top of those primitives--there's nothing like it for combining power and danger, and for whatever reason being able to build such things remains a rare skill.
• DK: I think the "whatever reason" is that we have done such a good job of creating layers that shield students from the "raw environment"---students building a web application in Django have no idea how many layers of complexity are hiding between them and the hardware, and wouldn't know what to do down there. The hard question is whether this is a good thing or bad----given that we have already built the layers, do we want to force every student to really understand them, or is it enough for them to use them?
• And to make my colleague MM happy, I won't cut this one out DK: As for most useful classes, I've got to flag Umesh Vazirani's CS124, at least to the extent that "useful" means "determined where I am now".

And to top it all off -- this week's picture, straight from China: Tao Stein's students at midnight working on their ray tracer.

Next: Flipped Out (April 22, 2013)

Unflipped!

We've entered that time of the semester when the students are busily creating user level processes in their very own operating systems. It is both exhilarating and exhausting. Due to that exhausting part, this is the part of the course where I promised not to make them do pre-class work -- there would be no videos to view and no web work (other than quick surveys about how much time they are spending). That means that, for the most part, I have had to revert to a more traditional class structure -- I talk and call on people and we try to collectively learn stuff. It feels dull; it feels unsatisfying; it feels as though students are not engaged. I am looking forward to polling them about this class structure so we can decide how to move forward during the next several assignments.

Since we're not flipped, I have little novel pedagogy to talk about nor do I feel the exhilaration I've felt throughout most of the semester (and any of you who have run into me and dared ask about the course know that I can now babble incessantly about how fantastic flipping is). So, today's entry is about two things: how it feels to be lecturing again and thoughts on what I would do if I had 250 students instead of 50.

The Traditional Lecture

It sure does take less time to prepare for class! Once I finish class notes, I have finished class notes. I don't have to think up pre-class activities; I don't have to think up in-class activities; I do spend a bit more time working through the notes, deciding when I can engage the class and which activities can be converted into class exercises.

As has historically been the case, I find myself interacting with about 20% of the class, instead of all of them. This is rather disappointing. I coax, I tease, I cajole, but some people just don't want to speak up in class. I do find it effective to ask for a raise of hands to vote on whether answers are correct or not and I seem to get better participation than I have in the past, and my sense is that even when I'm trying to extract thoughtful answers from people, I have a slighly larger fraction of the class engaged, but it's nowhere near the 100% that are engaged when the students are completing in-class work.

I also have found it effective to try to convert some of the open-ended questions I typically ask into exercises that the students can complete in groups. Even if it only takes a minute or two, it means that more people are engaged. It is this last tidbit that got me thinking about our next topic ...

Flipping 250 Students

A few of my colleagues have indicated that they have been thinking of how to flip significantly larger courses and it poses an interesting question. What would I do if I were teaching CS51, which is a heavy-duty programming class and has roughly 250 students?

Classroom space is the first obstacle. I love teaching in our new state-of-the-art classroom that has reconfigurable, mobile tables, traveling whiteboards, screens on both ends of the classroom, etc. However, it holds only 54 students, and even if you could build a bigger one, I think it would rapidly become sufficiently cavernous that you'd lose any semblance of a class. So, I would constrain my thinking to what would work for a more traditional lecture hall.

Since the students won't be able to cluster around tables, I'd probably try to use some kind of online shared whiteboard-like tool. That way students could bring laptops and work together in groups of 2-4. I would also want some way to do real-time data collection. I don't think you need any special clickers -- I would see if Google Forms work well enough in real time that people could submit answers and I could see the overall statistical picture of how students are answering.

The physical layout is going to make it difficult for the teaching staff to circulate -- you can get to the edges of rows, but getting into the middle would be tricky. I'm not entire sure how to deal with that -- I suppose the staff could join the online discussion, but it's not the same as interacting with the students directly. This would seem to be one of the biggest obstacles -- I just can't see the teaching staff climbing over students to get into the rows; having the students get out of the rows to talk to the staff doesn't sound better. I'm also not completely sure how the noise level is going to play out in a room with that many students. You really want them to be able to talk to one another, but a room of 50 sounds pretty chatty; I'm not sure how loud a room of 250 will sound.

So, those are my thoughts, but I think that the best thing would be to run a experiment. Perhaps if one of my colleagues invited me to guest lecture in his class, we could prep a single class and give it a try!

Next: Half Flip (March 9, 2013)

Flipped Again (Week 2)

Last week I was full of euphoria; this week not so much. I'll cut to the chase before diving into details. First, teaching this way takes at least three times as much preparation as teaching "the regular way." I still think it's worth it, but I can tell that it's going to take its toll. Second, in a programming course, preparing pre-class work and in-class work is a detail-oriented process, and if it goes wrong, it can go badly wrong. Third, I am hoping that full transparency, honesty, and a lot of humility will help to mitigate the disasters that can happen as a result of the previous item.

Part One: Time Intensity

Regardless of how I teach, I try to prepare thoroughly. When I lecture traditionally, I hand out notes, so students have the outline of what we're covering, but I leave a lot of blank space that I try to fill in collaboratively with the class. However, I always have the main points jotted down on my notes, so I make sure to cover the things I want to cover. After class, when I discover errors or things that didn't go well, I either fix the notes then and there or leave myself notes for next time. In either case, my notes and lectures are constantly evolving and changing.

So, how does that kind of preparation translate into my current situation? First, taking last year's notes, updating them, and then preparing the audio track requires at least a factor of two more time, perhaps even more now that I'm thinking deeply about it. Applying the same fixes I would have applied were I teaching conventionally takes about the same amount of time it would have. But then I start recording audio tracks. In my case, I'm using discrete audio files per slide, attaching them to the powerpoint, and then converting that powerpoint deck to a movie. The logistics of all this are tedious and take time, but that's not interesting (e.g., the PPTX->movie conversion really only works well using software that runs only on Windows; it produces enormous files, which I then translate using a different tool; these large files get moved around, etc).

The more interesting is what goes into preparing the audio bits. Since students will be listening to these on their own, they don't have the opportunity to interrupt and ask clarifying questions. We do have an interactive message board, and the students are taking good advantage of it, and the TFs and I are working hard to be responsive, but it's not the same as saying, "Excuse me, I didn't understand that." OK, few students ever actually say that in class, but when you can see them, you can frequently figure out that they are thinking that. In lieu of that, I think very carefully about what I want to say, how I want to say it, and then I record. I'd say right now about one in three times I am happy with my first take. One in three times I need a second take. One in three times I do many, many takes. So far, the videos have mostly come in around 30 minutes, so we're talking a couple of hours when you add in recording with incorporating the audio track, setting options, fixing things, etc.

But once I've completed the audio track, then I can really begin. Next I think about what I want them to take away from the video and I translate that into pre-class work. I think I am posing somewhat more challenging pre-class questions than one normally associates with this task, because I use those results to help me identify what issues are really confusing people. (I spend an hour or two in the morning before lecture reviewing answers, analyzing responses to the various questions, discussing them with my staff, and figuring out whether I want to a) simply post answers, b) go over problems in class, or c) do nothing.) In any case, I spend somewhere between 30 and 60 minutes thinking about and writing pre-class work.

Then I begin working on the in-class component. In our case, this involves writing code. Writing code always (always, always) takes longer than you expect. Testing is key. However, since my pedagogical thinking evolves as I am working through the assignment, sometimes I don't always use the practices. This brings me to item two on today's list.

When Disaster Happens

We had no fewer than four different types of disasters happen all in the span of a single 90 minute class (of which 30 minutes was consumed discussing pre-class work). There are many things to learn from this experience, so I'll walk you through what happened in potentially gory detail.

1. Pre-class work

It turns out that unless students were clairvoyant and could read my mind, there was much ambiguity in the questions as I wrote them. Both TF feedback and the analyzed data supported that claim. So I took the first 30 minutes of class to go through the questions and discuss the different interpretations and how those interpretations led to different answers. My assessment is that while it didn't feel good to be me, the end result was actually pedagogically constructive. We ended up talking about a number of important concepts and assumptions as a group, and I think we raised issues that, in past years, never surface, are never addressed, and manifest in difficult-to-debug code later in the semester. It will be difficult to do any kind of hard empirical analysis of this (although if people have ideas, I'm all ears), but my sense is that we had a deeper discussion about synchronization than typically happens.

2. Preparing for in-class work

After that, we turned to the class work. I had prepared the exercises as a separate branch, with the intention that they could play with it in class, and it would not in any way interfere with their own work, and afterwards they could either keep it or toss it without any effect. Great idea in theory. In practice, due to a combination of miscommunication, my own experience with the particular source code control system we're using (there were good reasons to change, but it's not one I've used extensively), and pretesting the process on a tree that was not identical to what the student's had, we ended up having them execute a command that, well, um, kind of messed up their tree.

3. Dealing with the in-class disaster, or "Another Disaster in the Making"

When then insued was sheer insanity. Over the course of the next 15 minutes, we frantically proposed multiple different ways of fixing people's trees. The TFs, students, everyone had an opinion, and we kept posting them. This was just dumb. We should have stayed calm, figured it out, and then dealt with it. But no, we freaked out -- all of us!

Finally, I went around the room and checked on every person to make sure that they had a working tree. That seemed like a good idea. Hoewver, then the fun really began.

4. The Crowning Touch

As they started working on the problems, they discovered truly, horrifically idiotic bugs in the code. How did that happen? I really tested things. I really did. Honest.

Ah yes, I'd done a couple of those last minute, "Hey -- this will make things easier to debug!" fixes and of course did not rerun the tests. That will teach me. Class ended with me feeling like I'd wasted a lot of truly valuable time and the students probably wondering what on earth they'd signed up for.

Transparency

So what do you do after a class like that? Well, I tried really hard to do better on Thursday. Although I did forget to have people put their names on the pre-class work, I didn't make any monumental errors. I did however, address the situation head one. This is what appeared at the beginning of Thursday's work:

First, let me start out with an apology -- we did not use your time efficiently on Tuesday, and that's a violation of the "contract" I made. I hope you all believe it certainly wasn't my intention, but that doesn't excuse it. I must, and will, do better. There are a number of ways in which things went badly, and rather than focusing on what they all were, I'd like to focus on how to avoid them. Thanks to Carl, we have a nice simple mechanism in place to help us prevent git shenanigans. Please refer to the Piazza posting. All that said, one takeaway is that at some point in the semester, if you find yourself in the midst of a git disaster (e.g., you and your partner cannot figure out how to resolve conflicts, get each other's changes in exactly the place and way you want them, etc), don't panic. There are ways out. Stay calm, learn to read git commit logs so you can figure out where things have gone astray and patiently try to fix them (we did not demonstrate the "patiently" part yesterday -- we frantically tried N fixes in a course of N-1 minutes trying to muscle through it; not a good strategy). So, as promised, today's exercise is simply to finish what we started Tuesday, ideally with significantly less drama!

I also tried to use the experience as a teaching moment, even if it was at my expense, so I also wrote:

There was another bone-headed error that got in here -- I will explain how this happened, just as a warning about "last minute changes." I had completed the problem, implemented a solution, tested it, and was happy. I then removed the synchronization to commit. "But wait!", I thought, "It's too difficult to debug, because there was no way to determine which TF is handling which student's question. The printout simply isn't helpful enough. I'll add a tf array so I can print out a TF number and that will be easier to debug." So I tossed it in the TF array at the last minute, and (here's the boneheaded part) did not rerun all my tests. So, take this as, "Don't do that!" No matter how minor the change you think you're making is, you have to rerun your tests. Especially if you're tired, that "simple" fix is undoubtedly wrong in at least three ways. Mea culpa. This was the second patch (semprob2.patch ) sent via email. Apply that patch too.

The Conclusion

So after all that, how did Thursday go? Remarkably well. The students seemed to accept my direct apology and commitment to do better. Perhaps some of them will chime in on their perspective (perhaps not; the one who are really upset will wait until the semester is over, and I understand that). And, by the end of class, all the students were animatedly and happily chatting about whale-mating. It was good to hear!

Next: Not Flipping Out (February 16, 2013)

CS161: One Woman's Adventure with a Flipped Classroom

For many years I have wanted to make my operating systems course, CS161, more interactive. I try very hard to engage students in the classroom, but let's be honest -- when they have been up all night trying to get their operating system to spawn processes, they just aren't really up to talking much in class. And when they do, it's the same set of characters and it's difficult to figure out if the rest of the class is really following along or is totally lost. At the same time, we know some typical student behavior that makes the course more difficult than it needs to be. Shockingly, even though we give students three weeks to complete assignments, they leave 99% of the work until the last week (day). That makes for a rather painful semester. Similarly, we ask them to write design documents, but many sit down to write them the night before without fully understanding what they are trying to do. And when we ask them to read the code and answer questions, they tend to gloss over most of the code and whip off quick answers.

I've always been tempted to try some newer classroom techniques, such as Peer Instruction and Flipped Classroom. I've just always been a bit afraid of balancing the potential for deriving greater benefit from class time versus dumping too much work on them.

So this year I took the plunge. I decided that I would flip the classroom for the first two to three weeks, return to a more conventional classroom while the students are deep in the most challenging programming assignments, but flip at key times, when I believe that there is real value to doing so.

I decided that I wanted to blog about this to record my observations, but also to invite students to chime in if they are so inclined. I also welcome comments from alums of the course or other others with concrete experience with related activities.

I am now one week into the semester, so we've done this for two classes. So far, I am wildly enthusiastic. It's scary, but it seems that we've already observed some tangible benefit (unfortunately, I do not have a good way to quantify/evaluate this).

Day 1

Historically I try to do a class-wide exercise where I draw the POSIX API or the programmer view at the top of a page and the reality of the hardware at the bottom and together we fill in what has to happen in between. My goal is to help students understand what the operating system must do and what they will be building over the course of the rest of the semester. As per many class exercises, a few students dominate such discussions.

So, this year I presented about 20 minutes of overview -- what the course was about, what I expected of them, what I promised in return, and then I turned them loose on three instances of the design problem (corresponding to the three main programming assignments). The TFs and I then wandered about the room, checking in, asking them questions, and helping out when necessary.

I quickly identified group(s) that were confused and were having difficult getting started. As other groups were happily making progress, and there were TFs talking with other groups, I was able to devote a significant and focused amount of time working with the groups that needed or wanted my attention. Instead of spreading attention around a class, dominated by the most confident and extroverted, I could provide focused attention where it could do the most good. Different groups addressed questions at different levels of detail, and that was OK. When I heard groups say things like, "You need page tables here," I stopped them and asked, "But what is a page table?" I wanted to force them to think through words they may have heard and make sure they really understood what had to happen.

After they worked for 30-40 minutes, I brought the class back together again and had groups present what they'd done, building upon each other, with my translating to some extent. I also assured them that we didn't expect them to complete this exercise perfectly -- after all, if they already knew how to do it, they'd have little use for the rest of the semester.

To me, it felt pretty good. The only tangible data I have is that a few students, who started off saying that they weren't really planning on taking the course because there were three other CS courses meeting at the same time, signed up. Pre-term planning number pegged the course at 37; we had about 60 students the first day; 47 registered. This is larger than the course has been in 15 years! And, it's important to realize that 15 years ago, there were no other undergraduate systems courses to speak of, so choices were significantly more limited.

Day 2

I took the material that I usually cover the first week and turned it into two videos (all the materials I'm talking about are linked from our syllabus here). For pre-class work, I had them do part of what we ask them to do for the first assignment -- get their work environment set up. Then, we spent class time letting them configure and build kernels for the first time. A few people ran into difficulty, and this was the beauty of doing it in class -- we could help them right then and there, making sure that they weren't stuck. So, within 10-15 minutes, every single student had configured and built a kernel. How great is that?

Next, we let them start answering code-reading questions (typically included as part of the first assignment as well), encouraging them to discuss the trickier ones amongst those at their table. Here is the thinking: most student spend little time actually reading the code, and when they do read it, if they are confused, there isn't much they can do except post a question (always a bit scary), email a TF, call a friend, beat their heads against the wall. However, by having them start in class, they had friends right there to work through the code with them. Some groups worked pretty independently, other groups worked completely together (two people in front of one terminal), others did a combination. Once again, the TFs and I circulated, answering questions, asking questions, encouraging discussion, etc.

They are each supposed to write up the answers to the questions to turn in, but they should have made significant progress on the assignment already.

On to week two!

Next: Flipped Again (February 7, 2013)