Thursday, August 15, 2013
An Index to my Flipping Blog Postings
One Woman's Adventure with a Flipped Classroom (February 4, 2013)
Flipped Again (February 7, 2013)
Not Flipping Out (February 16, 2013)
Addicted to flipping (February 21, 2013)
Unflipped! (March 4, 2013)
Half Flip (March 9, 2013)
Flipping and Testing (March 18, 2013)
Inadvertent Flip (April 1, 2013)
Missing Flipping (April 8, 2013)
Anxiously Flipped (April 15, 2013)
Flipped Out (April 22, 2013)
Flip N-1 (April 30, 2013)
Flipping Over (May 24, 2013)
Friday, August 2, 2013
101 Uses for Summer Squash
My old standby that is quite popular with the younger crowd are chocolate zucchini muffins (thanks to Jill Levien).
This year, I branched out and went for Sue LoVerso's chocolate zucchini bread. I did this one both for home and for a potluck; it was popular.
I also signed up to feed a film crew worth of 15-year-olds one weekend. Not knowing if all present were carnivoes, I opted for one vegetarian zucchini pasta casserole in addition to the very meaty version I usually make.
There were many dinners that included stir fried zucchini (or summer squash) with garlic.
But I was still feeling overwhelmed, so I went for a variant of the lindentree casserole, which I'll call How to use up a ton of squash in something vaguely reminescent of lasagna.
Next I went to some of my favorite cookbooks and found a fabulous recipe for zucchini pizza crust from Moosewood's original cookbook. I put cheese and sauce and onions and other goodies on it and thought it was fabulous!
I may try some zucchini noodles this weekend (use a vegetable peeler to turn a zucchini into noodles; blanch them and then serve with pesto or tomato sauce or any pasta sauce you'd like).
Last year, I had a couple of other recipes I used, but didn't seem to write down:
- Squash fritters: basically take some shredded squash, an egg or two, some parmesan cheese and enough flour to give you something you can shape into patties. Then fry them or bake them. I love these as a late night, post-soccer game snack.
- Stuffed Squash: scoop out the gust of the squash and saute with onion, garlic, olive oil, maybe tomato, whatever spices you got from the farm and then enough bread crumbs to dry it out. Put it in the squashes and bake until the shells are soft. Tomato sauce is nice on these.
- Squashed Slaw: Make something like cole slaw, but use shredded zucchini instead of cabbage.
- And finally, this amazingly fudgey chocolate zucchini cake is shocking.
Don't forget old standbys such as ratatouille, sauted squash with garlic and basil, lasagna with sliced squash instead of pasta/noodles.
Enjoy!
Thursday, June 6, 2013
Advising Junior Colleagues
Let me begin with a disclaimer: I can't guarantee anyone success. When I talk to my junior colleagues, whether at Harvard or elsewhere, I tell them things I really believe, but (unfortunately) I don't make the rules, I don't make decisions, and what I believe may or may not be right for any particular person or institution. So, this is my opinion on what I think one should tell Junior colleagues. But hey, this is my blog, and everything in it is just my opinion.
Let me just start out by putting it bluntly: being a professor is a bit of a crazy profession. It's one that has worked for me and offers many features (perhaps the topic of a later blog), but it's not the kind of job any sane person would make up. You get into this job because you're used to being really good at everything you do. Unfortunately, there are way too many things for a professor to do that you can continue to do all of them at the same quality level that makes you comfortable This is perhaps the greatest challenge that we have to face -- be willing to do things at less than peak performance. I repeat, "You must be willing to do some things at less than your peak performance." This seems totally counterintuitive in a highly competitive job, where you feel that everything you do is highly scrutinized, but if you try to excel in everything simultaneously, you will drive yourself totally crazy.
So, what is an over-achiever to do?
Pick and choose carefully. The key here is that you can't do everything well all the time. Instead, you pick what you are going to do well at any point in time. As an academic, I like to think in terms of semesters. At the beginning of each semester, you have to decide which are the things for which you need to produce A work and for which can you slide through with B work. Each semester, the lists change slightly so that at some point, every functional area gets some A work and some B work and every semester gets some A work and some B work.
Let's make this more concrete.
Let's say that you've just started your first faculty position and you want to make a good impression -- how do you get started? My advice typically goes something like this. No one expects you to crank out much research your first year. If you have some straggling papers from your dissertation or post-doc, that's fantastic -- get those out. But, your real focus in year one is to get your teaching under control and get your research agenda planned. Note that planning a research agenda is different from doing research, but we'll get to that in a minute.
If you are in a kind department, then you can assume that you'll be teaching the same courses for several years. That means that if you invest time this year in developing your courses the way you want to teach them (i.e., you make teaching an "A" priority), then you can leverage that investment over the next several years (expending "B" effort, while ideally obtaining "A" results). So, while many will tell you that any time you invest in teaching is wasted, I disagree. Good teachers get good students. And good students will help you do your research. That said, invest time wisely. Time spent automating grading is time well spent. Time spent developing four thousand step animations is not. (And trust me, it is very easy to fall into this trap. I've done it.) I could go on for quite some time about what is and is not a good use of your time with respect to teaching, but that's another topic. For some detail, I recommend my colleague Harry Lewis's article on the topic or my more recent experience elsewhere in this blog.
And then remember that what works for me may not work for you. The key ideas are:
- Figure out what works for you.
- Invest time in things that are reusable.
- Remember, the quality of your teaching is best measured by how much the students learn, not how entertained they are or how much they think you or your slides rock.
- True learning takes place when students do things and get feedback -- anything you can do to improve the quality of feedback students get on work increases the chances that students are actually learning something.
OK, so picking up the main thread here -- invest time in your courses in year one so that in the next two to three years, you can teach the same courses with minimal effort. You hire good teaching assistants, you pull out your notes (and perhaps assignments) from the previous year and you spend almost no time preparing. Sure, in those future years, you're giving a B effort in teaching, but because of the prep you did in year one, you ought to be able to invest B effort and get A- or B+ results.
The second part of year one is planning your research program. That probably means figuring out what you want to do and then writing the proposals to get funding for those projects. This is the time to renew acquaintances with all your friends who are now working in places that fund academic research. See if you can get some supporters lined up for corporate grants -- these are small grants, but require significantly less effort than your classic NSF grant. You'll want to submit one or two of those as well (and I mean one or two, not six or eight), but your main goal is to have a plan for the research you want to do and convince people to give you money to do it. If you have your ideas well in place by the end of the first semester, that puts you in good shape to identify the grad students you want to admit.
Getting a handle on what projects/research you want to undertake also sets you up in semester two and over the summer to write high quality grant proposals. This is the time to squelch your inner procrastinator. Get your proposals written early enough to get feedback. Ask your senior colleagues for critical review -- if you are in a good department, they want you to succeed and will help you submit the best proposal you can. It's also worth teaming up with another colleague or two for some bigger proposals
So, if all goes well, by the end of year one, you will have lots of reusable materials for your teaching, a research agenda on which you can begin to execute, a couple of proposals in the pipeline, and some incoming grad students whose interests are well-aligned with your research agenda. Begin year two.
Year two is all about getting your research firing on all cylinders. Over the last several years I have become a strong believer in knowing what paper you are writing when you begin a project. If it's real research, then you had better not know the results, but you had better have some hypotheses. Write those down. Really. Honest, write the opening paragraph that motivates what you are doing and why. If you can't write it now, why are you doing the research you're doing? Sure, you don't know how it will turn out, but if you don't have an hypothesis, you don't really know why you're doing what you're doing.
Concurrently with getting that research started, you're also starting to be a graduate advisor. That is a whole other topic, but in the context of this discussion, the message is you have to realize that this is another one of those activities that you need to learn to do. And every student needs a different advisor -- you might be able to play that role for multiple students, but if you walk in believing that you can advise every student identically, you will not be a successful advisor. You need to have some basic advising methodology and a philosophy, but you need to get to know your students, understand their strengths and weaknesses and figure out how to most effectively advise each student. This will take time.
After year two, you should be a on a role of focusing mostly on your research and students, and then periodically having to step back and brush up your teaching (i.e., make it an A task for a semester or two), whether to update a course, update your teaching methodology, teach a new one, develop a new course, translate your course to the web, etc. However, if you use that time effectively, then you can still make research an A priority for many of your semesters. Do not confuse grant writing with research -- grant writing is a necessity (at least in my field). If you are successful, you can secure several year's funding with a semester or two of concentrated effort and then forget about funding for awhile. However, you have to keep track of where you are in the funding cycle and know when it's time to elevate grant writing to an A activity, so you don't suffer from a gap in funding. Personally, this is a huge challenge for me as I'd much rather do research than write grant proposals (I'm kind of guessing I'm not alone here). As a result, I can end up leaving grant writing a low priority for too long -- don't let this happen to you.
Now, some of you are getting ready to scream at me and say, "What about all that other stuff? What about program committees and NSF panels and guest lectures and attending conferences and departmental committees, and undergrad advising and all those other things that our institutions expect us to do????" If your colleagues aren't already ready to fire me for saying that you can sometimes relegate teaching to a B activity, I'm sure some will cringe at the next section.
High order bit: you do not have to do as much of this stuff as you think you do.
There, I said it. Seriously, the closest I've ever come to "yelling" at my Junior colleagues (you know who you are) is when I see they have signed up to do six program committees in one year. Yes, program committees are useful. Yes, they keep you up to date on what people are doing. Yes, they get you known. But six? No! It's not a good use of time.
My advice is no more than two program committees each
Now, how do you pick the two? You want to work your way up to getting on the program committee of
Note: I do not always heed my own advice, and I almost always regret it.
OK, that's program committees, what about NSF panels? I find that serving on those is extremely helpful right before I am about to submit a proposal, because they remind me how panels work and what's important in a proposal. So, you should do a few of these, but not a lot -- perhaps one every other year? That feels about right to me. And yes, you'll get asked to do more. Fortunately, those invitations are always accompanied by specific dates, so you can use your, "Oh, I'm so sorry, but that day/week/month is simply not possible for me."
How about committees in your own department? This is a bit tricky. If you are in a nice, nurturing department, they won't ask you to do too much, because they don't want to distract you. However, there are some committees that are worth the time -- in my opinion graduate admissions and hiring. The former because it gives you firsthand knowledge of the pool and puts you in good recruiting territory. The latter, because you know who's coming out in the few years after you, who you might want for colleagues, and at whom you should be looking. However, moderation is your friend -- don't do both in one year -- each of these is very labor intensive (admissions a huge burst at one time in the year; search a more steady time sink throughout the year). Just like with your PC invitations, you can be polite and constructive in turning down committee invitations, "Dear Chair You-Who-Controls-My-Destiny: While I would be very excited to serve on the suck-all-my-time-out-of-day committee, I'm afraid that I cannot do it this semester/year. I've already scheduled all my committee time on other activities. I would however be happy to serve next year (and will reserve time now if that's feasible)." What's a chair to do? They can't really dispute what your schedule looks like and you have, in fact, signed up to do it next year. Of course, if it's an activity that you feel you never want to do unless someone is holding a close relative hostage, you'll need to tweak that letter slightly and leave off the part about wanting to do it in the future.
Finally, there are those friendly invitations to "Come give a talk." Giving talks is a good thing; it can be fun; it gets you visibility and gives you a chance to find out what's happening in a department. But, it also takes time and means "yet another trip." So, be strategic. There are times during your pre-tenure years that giving talks is more important: the semester before your department is going to write to every researcher on the planet asking their opinion of you. Yup, the year before tenure and, if you have a formal mid-tenure-track review, the year before that. Even then, you do not need to give 4000 talks. Be strategic. You can probably guess at least half of the people to whom your department is likely to write. Pick the very top departments from that list. Rather than visiting every institution on your list, leverage the most prestigious departments in your field. Be visible -- maybe, just maybe, if you're up for tenure, you give the talk rather than having your student give it. Even if you don't do that, use your conference time wisely: talk to colleagues, let them know what you're up to. Invite them out to lunch/dinner with your students. Finally, rather than traveling to give a talk, you can ask your department to invite them as colloquia speakers. It's funny how many invitations I get to come give talks that are followed up a year later by letter requests. You think it's purely coincidence? I don't.
The other way to make sure people know what you're up to is to convene a mini-workshop on YOUR RESEARCH. You have to get this right, but if you have the funding and can really take advantage of the event and you can persuade people it's worth their time, there is nothing better than inviting 15-20 bigwigs in your field to spend a couple of days at your institution engaged in thoughtful, interesting conversation. If they also happen to come away with the big picture of your research agenda, won't you feel like the star? I believe that it is becoming more common for departments to allocate at least some funds to help you do something like this. It can save you a lot of travel.
Above all, learn to say, "No." You can be polite, but you must be firm. After one particularly brutal year, I adopted the mantra, "It's all my fault." The message to myself was supposed to be that I control my work life and if I'm too busy, it's because I've said yes to too many things. Surprisingly, rather than being a defeatist attitude, it's actually quite empowering. If it's all my fault, then I have the power to change it.
One of the trickiest things for me is saying no to students. I truly love spending time with students -- after all, if I didn't like students, why would I choose to be at a University? When they invite me to a faculty dinner, I'd love to go, but guess what. Dinner time is family time in my life. I already miss way too many dinners when I travel and I frequently have some child-transportation responsibilities in the late afternoon or early evening, so it's quite difficult to really make faculty dinners go. So, what do I do when students ask? Typically I explain that evenings are rough for me, but suggest that we do lunch -- either during the semester, or more frequently during reading period when both our schedules are a bit more flexible. This past year, I had evening duty in Cambridge with one of my kids and that let me attend a faculty dinner or two or use those evenings to take students out. Yes, it cost more than a faculty dinner, but the students are more interested in just hanging out with you and talking than with trying the local 5-star restaurant (should you have one). The hours I've spent over lunch, dinner or coffee with students are among the most fun of the semester -- it doesn't matter if it's in the formal setting of a faculty dinner. In fact, think of the faculty dinners as mechanisms for you to figure out which students might enjoy just chatting and then do that!
OK, those are the things I like to tell my colleagues. Some have found these words useful; I'm sure others will find that they don't work. In any case, you get to make decisions about how you spend your time. If you make those decisions wisely, so that they simultaneously help you be productive and keep you sane, you will either end up tenured or you'll end up not-tenured, but reasonably content with your life, instead of bitter and resentful.
A Note on Powerpoint and Teaching
I'd like to talk about powerpoint (or any other presentation software), animation and teaching. For years, I avoided teaching with powerpoint, because I like to scribble on slides -- my slides are typically the outline of what I want to talk about and are mostly reminders for *me* so that I remember what I want to tell the students. However, there is information on them that you don't want students to have to frantically scribble down, so you think, "Hey, I can give them copies of the slides, and then they can listen to me instead of scribbling frantically." This is the beginning of the slippery slope. Once you hand out notes, the students believe that they are proxies for lecture, and if they are not, they will complain bitterly. Even though I very clearly explain that the notes are for me and are provided as a service and are not a replacement for lecture, I've been totally reamed for "having four blank bullets on a slide and only filling in three of them during the discussion in class." I kid you not.
Anyway, I finally transitioned to powerpoint when I got a tablet, because then I could have pretty slides and scribble on them. The first year I did this for my OS course, I spent way way way too many hours building meticulous animations of a context switch. You know ... I am 100% convinced that those animations wasted a ton of my time and benefited absolutely no student and probably harmed some. All the literature confirms that students do not learn by passively watching or listening to you -- passively watching you step through the animation, even if you explain it ever-so-eloquently, is absolutely not the same as forcing them to figure out what the steps are. Yup -- it will take a lot longer if you make them figure things out, but you know what -- they'll actually understand it better. (You may argue that with the picture perfect animation they can step through them later and without you there actually understand what's happening. While that may be true, one in twenty students will do it, while pretty much everyone sitting in a classroom has to pay some amount of attention if you're constantly making them explain what is going on.)
I do devote time to trying to make my slides and explanation as clear as possible when I am recording material for students to screen in advance, but I am working hard to avoid spending significant fractions of my waking moments building fancy animations. In my opinion, animation bad; making student think and discuss what things must happen, how they happen, why they happen, etc. is significantly better.
Saturday, May 25, 2013
Chasing Mia
As an admitted groupie of the US Women's National soccer team and a Meadowbrook parent, two unrelated postings make me want to write.
As this year's soccer season began for the women's national team, I struggled with the emotional conflict I felt as I prepared to watch Abby Wamback overtake Mia Hamm's goal-scoring record. (I predicted last year it would happen during the Algarve cup and I was wrong, but it will certainly happen this year.) Then, the head of our Middle School wrote a nice piece about why Meadowbrook gives out Spirit Cups for their sports teams.
I have always watched in stunned disbelief as the media and most of our society turn atheletes into role models. Just because someone is good at riding a bicycle (quickly) or throwing a ball into a hoop, why do we immediately assume that they are people our kids should emulate? At the same time, year after year, I look at the women who play for our US team, and I think, "Now there are role models!" These are nearly uniformly amazing women who have dedicated themselves to an underappreciated and underpaying sport. They work hard on the field and off the field, reaching out to fans, working with young people, and dedicating themselves to important causes, without a lot of fanfare, without the trappings of superstardom, and without the negative actions that draw negative publicity. I can hear some of you, "Oh, they are just women soccer players." No, they are world-class professional atheletes. They are among the absolute best in the world at what they do. And they do it in a sport that is important to more of the world than any of our conventional American pasttimes.
At the risk of missing someone deserving, let me run through the list of past and present US Women's National Team players who are extraordinary individuals as well as extraordinary atheletes.
- Joy Fawcett, thank you for embracing your professional atheleticism and motherhood simultaneously, for showing us that the two are not mutually exclusive, that you can bring your kids to work and still perform at the highest levels.
- Michelle Akers, thank you for showing us what it means to give your all -- through illness, injury, and perhaps even beyond the point of what was good for you personally, you gave every last bit of yourself on the field.
- Heather O'Reilly, thank you for showing us what a work ethic is. In every minute of every game that you play, you are 100% engaged on the field, chasing down every last ball or player.
- Amy Rodriguez, thank you for being approachable, for understanding that just talking with and teasing the young people who admire you makes them feel special.
- Carli Lloyd, thank you for helping me never question whether I'd play soccer again after breaking my ankle. As I rehabbed, I watched you rehab. As you returned to the national team, I knew I too would return to my (not quite so competitive) team.
- Shannon Boxx, thank you for demonstrating the importance of having a professional league in this country.
- Cat Whitehill, thank you for seeing into the heart and soul of a recreational soccer team and treating them with the same respect you gave your national teammates.
- Brianna Scurry, thank you for always demanding the best, for handling difficult situations with grace, and of course, for stopping PKs!
- Tiffany Milbrett, thank you for making me proud to wear #16.
- Kristine Lilly, thank you for many things -- for redefining what age means in soccer, for demonstrating what commitment and passion really are, and for showing the world that it's not only OK, but good for women to wield power.
- Thank you Abby -- for everything -- for the role that's been thrust upon you as the bridge between the past and the future. More than anyone else, you have been the transition between "the 91ers" and the future of US Women's soccer. You have provided leadership through both good and challenging times; you graciously accepted the mentoring of your predecessors and pass that on to your successors. And, thank you for putting Rochester on the map!
- Last, but never least, thank you Mia -- for becoming the face of women's soccer, although it seemed you would have always preferred not to be in the limelight. Women's soccer needed a superstar, a spokeswoman, an idol. You showed us that teamwork was as important as personal performance by wracking up almost as many assists as you did goals (144 to 158). Regardless of whose name appears in what categories in the record books, you will always be Mia -- the first face of women's soccer in this country.
Friday, May 24, 2013
Flipping Over
End of year party
Long ago, in 1993, we started the tradition of having an ice cream bash after the take home final. I don't believe we've maintained the tradition, but I decided to resurrect it. In addition, we continued the tradition of class T-shirts. The winning design was:In addition, since I had artwork for each group, I made that available to students to construct their own team patch, which they could then iron onto their shirts. Groups who didn't have the time to do anything fancy (21 of 22 groups), got the plain hand-drawn black and white animal.
So, on Wednesday afternoon after spending a lot of time on the final,
we all gathered in a much too small room to eat Toscanini's ice cream,
collect T-shirts, and iron on patches.
It was a total blast.
Pretty much everyone ironed their patches. The one group who augmented
their animal was the marmots, who added color and appropriate attire:
We had a few ironing disasters which prompted a return to QRSTs for a few additional shirts.
Had I been thinking, I would have tried to get a group shot, but I didn't.
Next time!
One of my students, did take this small group picture:
Students seemed relatively happy and relaxed, although a few still had multiple final projects to complete. Much ice cream was consumed. The pear chardonnay sorbet was not consumed, but I have to say, it was outstanding (it came home and disappeared stunningly quickly). It was a good end to the course!
The final
For the curious, here is the final exam. I thought it was pretty tough; it was certainly time consuming to grade (it took me the better part of a week to grade 45 exams -- I'm guessing over an hour per exam). The students did exceptionally well -- a typical Margo-exam ends up with a median around 67%; this one had a median and mean over 80%!The first problem was a design exercise and those are notoriously difficult to grade; after two days of grading I wondered why I continued to give these kinds of questions. Then a colleague, Kryzstof Gajos, explained how we graded his final exam, and I am excited to contemplate how I can apply this technique. I've invited him to write a guest entry on the topic, but in the meantime, here is my understanding.
The course he was teaching is called CS 179: Design of Usable Interactive Systems. It is very much a course all about design. So for the final exam, he split the period into two parts. During the first part, students took the (written) exam using a "uniquely" colored pen (provided by Professor Gajos). Then, he collected the pens and the class collectively graded (their own) exams. Thus, in real time, the class discussed the problems, the supposedly right answers, alternative answers, etc. When there was ambiguity, they discussed it. Afterwards, he collected the exams, had the TFs check that the self-grading was accurate and he was done.
This was pure brilliance! I'm not saying that because it reduces grading time (which it does), but because it turns a purely summative evaluation process and makes it partially formative. I try to do this when I grade exams -- I essentially engage in a typewritten discussion about their answers (which is one reason exams take so long to grade). I always have second thoughts about this, because I know that students merely search for the final number and ignore everything I write. Or so I thought (more on this later). The idea of discussing the exam right after they've finished it and being able to address confusion immediately is overwhelmingly appealing to me. It's not entirely clear how to apply this to my exam -- I like pushing students to think about complex issues for which mulling the ideas and questions over is useful (this is why I like the takehome exam format). I don't know how I'd fit both the testing and the grading into a standard 3-hour test slot. If I stick with the 24-hour format, then I'd have to make the exam be, say, a 20 hour exam with a scheduled 2-hour class which sounds like a logistic nightmare. I'm not sure what I'm going to do here, but I see further experimentation in my future.
In addition to a design question, I asked people to read a research paper ( the Barrelfish SOSP paper), and then I asked them questions on it. I'm really looking forward to having some of these students in my graduate OS class, because the high bar they set for a system is quite interesting. Many were accepting of a prototype, but a surprising number took a more business perspective, while perhaps not appropriate for a research paper, indicated to me a much better sense of how the world works than many tech-savvy people are. I found it fascinating.
In another question, I asked them to respond to a (rather poor) technical suggestion, but I asked them to do it in the context of an email to an engineer who worked for them (I put the students in the role of being the project architect). I was quite impressed with the care most students took in crafting a reply that was both technically correct as well as polite.
The post final
Although I write fairly long comments on my students' exams, I know deep inside that they never read them, and yet I continue to do it, because I know that there is the potential for learning there. This year, I was pleasantly surprised -- for the first time in twenty years, I had a couple of people engage with me after the exam to discuss answers. They weren't looking for points but were instead, trying to make sure they understood what a right answer to a few questions would look like. In one case, a student wrote up about 5-6 paragraphs as an answer to one problem to make sure that the student had a complete grasp on the answer. I was (pleasantly) astonished. While I can't make any causal statements between new pedagogy and this behavior, I could certainly believe that because I am much more closely engaged with students, they felt more comfortable doing this. In any case, I was quite pleased.
The end
And so, one woman's adventure with flipping comes to an end. I'm looking forward to crunching data, and I'll certainly post here if/when I find something interesting to say. In the meantime, here are my parting thoughts.- It's good for an old dog to learn new tricks.
- Flipping lets me spend time with those students for whom the material is challenging.
- Learning takes place by doing, not by listening to me.
- TF engagement is critical.
- It takes a lot of effort to come up with effective in-class work.
- Pre class web forms are AWESOME. They let me engage with students in an entirely different way and to gather lots of interesting data.
- CS161 is even more time intensive than I thought.
- It would be useful to help students learn what it really means to design something.
- Flipping is a great equalizer when students enter with different experience levels or exposure to different topics.
- Fully integrated and coordinated materials take real effort but pay off tremendously.
Tuesday, April 30, 2013
Flip N-1
The Pain/Gain Scale
One of our former PhD students is now on the faculty at Swarthmore and he suggested to me the pain/gain scale as a way to determine if the degree of difficulty in assignments was perceived to be worth it. This seemed like a fine idea, so I polled students on the two major assignments completed so far, asking them to rate on a scale of 1-5 the amount of pain the assignment caused and the amount of gain they derived. The results are fascinating.Here is the histogram of the responses:
At first I thought that Assignment 3 was really nice correlated, but
then I calculated the ratio for each student and graphed that -- it
tells a different story:
While most people found the ratio about 1:1, there were some who clearly suffered and others who had a blast. I have not done further analysis, correlating these ratings with how well groups did, but I found it interesting nonetheless.
The Partially Flipped Class
The astute reader will have noticed that while upholding my deal that I will not make students prepare for class while they are also implementing processes, building a VM system, or making a file system recoverable, it's been difficult to maintain the energy and enthusiasm of the flipped classroom. As I entered the homestretch, I decided to try an experiment with what I am calling the partially flipped classroom. I take my lecture notes and I break them into two parts -- a short intro (6-8 slides) and the rest (10-20 slides on technical content). I present the intro and then I dispatch the class for 15-20 minutes deriving from first principles some of the challenging aspects of the particular topic and approaches for addressing those aspects. For example, on Tuesday I was teaching virtualization. Now that they had built major pieces of an operating system and become intimate with other parts of the OS, I let them figure out what the big problems were in building a virtual machine-based environment. The groups each came up with excellent lists, which we then pooled. When this was all done, we'd covered much of the material in the rest of the slide deck. So, I was able to breeze through the rest of the deck relatively quickly and I knew that they had a much more complete picture of the topics, because they had largely derived them themselves. It felt very good. I repeated the process on Thursday with security. I introduce the three A's of security: Authentication, Authorization, and Access Control and then let them play around with identifying what can go wrong if you fail to address each aspect properly and then proposing approaches to solving it. This week, we'll be able to breeze through my slides on the topic relatively quickly, because most of the ideas came up in discussion. That will leave us time to talk about a few real world exploits.I think this is the approach I want to try for all the material that I had been teaching "the old way." I'm pretty sure that with some careful thought I can make it work and we may be able to maintain a higher level of energy and engagement during the roughest parts of the semester. We shall see!
The Flipped Exam
Thanks to the wonders of Facebook and my friend Jill Levien, I was introduced to the idea of a Flipped Exam. Now, I don't teach game theory and I'm not entirely sure how this would work for my class, but I found it intriguing, so I posted it to our class discussion board with the directions "Read/Discuss." And my fabulous students engaged!Student1: We should have such a final exam: the entire class: write as much as possible of an operating system in 24 hours together. editdelete Student2: While I do think there is a place for individual examinations, the current educational system overvalues such tests. My basic understanding of the "real world" is that most projects of any real significance are done in teams, be it research, commercial products, etc. Sure, there have been cases of lone wolfs that create paradigm shifts or fantastic projects (and this should be encouraged as well!), but teamwork is ultimately a skill that should be encouraged, not subdued. (24 hours might be a bit short to get much functionality done on an OS even with a full class of people, but I certainly wouldn't mind a collaborative written final). Student3: Synchronizing >40 people would be so much fun... Student4: I really like that idea. It will be a fun, collaborative experience, and we'll end up synthesizing the entire semester together into one glorious product. Student5: This idea was the first thing I thought about as well. I think it would be fun, maybe not necessarily writing a whole new operating system, but writing another subsystem for OS/161. TF1: Rob Bowden: Networking!! Student5: Indeed that would be fun. Student6: so punny Student7: You get a synchro bug! And you get a synchro bug! Everybody gets a synchro bug! Student8: I find this much more reflective of how the world actually works and I think it reminds people that the best answer is 99% of the time not produced in solitude. I'll second Student2's final suggestion as well. Student9: If it's a coding assignment, maybe we'd need (want) groups of a smaller size. 45 people can't all be coding at the same time, so it seems as if the exercise would likely devolve into "Okay, you two/three/four code, we'll just watch...and write a DesDoc? And, uh, run tests!" What about five partitions of nine, sorted day-of? You've got 24 hours to do what you can on [subsystem], working with some, all, or none of your tablemates. Coordinate sleep, food, and synchronize your watches...here's an assignment spec, go!Right now I think I'm still going with the individual exam, but I'll definitely have to think about the concept of collaborative exams for the future. I am glad the class (or at least a subset of it) thinks that working together would be fun.
T shirt Designs
Even in the midst of a pretty heavy workload, my students know how to have fun. Check out the T-shirt designs on our home page. I'll let you know which design won after the T-shirts have been delivered.Monday, April 22, 2013
Flipped Out
I sit at home, working from the relative safety of a town six or seven miles way from the mayhem that erupted in the greater Boston area last night. As is old news by now, within roughly 48 hours of the Marathon Bombings, FBI and police had identified two "persons of interest" who became suspects last night after engaging in a robbery, a shooting, a police chase, and explosives. As of right now (8:13 AM on Friday, 4/19), one of the suspects is dead; the other is the target of an enormous manhunt that has much of the greater Boston area on lockdown. All the local universities are closed; my kids' schools are closed; people in six towns are being told to stay home and leave doors locked. I've certainly never experienced anything like this, and I'm guessing the vast majority of my students, friends, and acquaintances haven't either. So, as I did Monday, I turn to blogging.
The end of the class is in sight! Because many groups took late days on the virtual memory assignment, we pushed the design reviews for assignment 4 (make a file system recoverable by adding journaling) a few days. This gave students a much needed bit of breathing room and as a result they came to class Tuesday pretty energetic.
Rather than continuing with new material, I made Tuesday a flipped class. They had a few short pre-class questions to make sure they'd read the assignment and understood what they needed to do. We then spent class time letting them code read to figure out how the existing file system worked, what kinds of things were going to need to log, and how they might recover those log records. On one hand, we could have done a structured code walkthrough, but I'm pretty convinced that letting them do the walkthrough in small teams results in a better understanding than a guided walkthrough. (Although I'm beginning to wonder if we shouldn't offer code walkthrough sections each week ... that might be a really nice supplement. Or perhaps we could make a series of audios to walk through the code ... oh I'm liking that idea a great deal.)
We spent the first chunk of class time letting students merge new code into their trees. My theory is that this would be a relatively quick process and anyone who ran into trouble would have immediate assistance. I'm not sure that part worked very well. Some students got the merge done quickly, but others spent a significant chunk of time on it, even though they didn't need our help.
They were then supposed to pick a couple of operations and walk through them to figure out what they'd need to log and what recovery would look like. Some groups found this useful; other groups felt lost. Even for those who felt lost, my sense is that it was a decent use of time, as the students really need to understand the process of figuring out how to determine what needs to get logged and what needs to be recovered.
This class was followed up by the peer design review. The room sounded very animated during the review and I heard a lot of good discussions. I think the extra days after A3 really paid off here. We need to work with the timing to make the A3 peer design a more useful endeavor. That's going to require some serious rethinking about what we do in-class versus outside of class.
All in all, this week was a classic flipped classroom and everyone seemed much happier. I also took the opportunity to snap photos of my students with their group placards. For those of you who've never been involved in Harvard's CS161, we ask each group to select an animal as their team name. We interpret the term animal quite broadly and we've had everything from unicorns to meowbears in past years.
This year, I had someone who shall remain nameless draw each team's
animals and then we printed them on bright yellow paper, laminated
them, and but them in stands to assign groups to tables.
With permission from the people photographed,
I bring you a subset of the images of 2013 CS161!
The Blobfish
The Anteaters
The Caribouyah
The Centaurs
The Falcons
The Sphinx
The Squids
The Baby Velociraptor (singular)
The Dingos
The Koalas
Monday, April 15, 2013
Anxiously Flipped
I figured that blogging might be a distraction from the day's events, but nonetheless, my suspicion is that this will be short and perhaps scattered.
It was a rough week to be a CS161 student. The virtual memory assignment was due on Friday and it is a particularly devilish assignment -- the number of lines of code to write is not huge, but getting the synchronization correct is just downright hard. This sets the students up for a lot of late night debugging, perhaps an all-nighter, and then perhaps a couple of late days. This has all played out reliably over the past many years.
So, this week's status has much less to do with flipping and much more to do with CS161, virtual memory, and why anyone would put themselves through this. For your reading pleasure, I bring to you a Facebook discussion that took place after I posted a note about how guilty I felt coming in at 8:00 AM and finding my students (surprisingly happily) working away, having been up all night. Let me just say that I never looked that happy after an all-nighter.
My original post: There is nothing quite so guilt-inducing as walking into your office at 8:00 am and seeing your students sitting in the lounge working on your problem set after having been working on it all night. Oh wait -- there is -- seeing multiple groups later that day around 4:00 PM -- all of whom have been up all night and are still working. Many of them now have operating systems that mostly support virtual memory, though!
In the following, I have converted names to initials, to provide some shred of anonymity.
- GK: That last bit, "mostly support virtual memory," reminded me of this and this.
- MO: that's guilt-inducing? I thought that's what you live for!
- RH: And yet, it didn't keep you from giving them the assignment in the first place. Hmm! Actually, IIRC, that was the easy assignment. It was the stupid filesystem one that almost killed me. I wonder if these assignments would still seem as hard today as they did 15+ years ago when I did them (am I really that old?).
- GK: Tell them that in 24 hours Andy Sudduth finished his operating system, finished putting together his HeathKit computer (this was in the early-to-mid 1980s), wrote a full screen editor, and then wrote a paper due the next day in his editor on the computer he built running the OS he wrote. Or, you can tell them what I try to tell myself—start sooner. Editorial note: Yes, Andy Sudduth, Harvard legend and Olympic rower is also a graduate of CS161
- Me: We've tried many, many things over the years to attempt to make the problem sets less time consuming, but at the end of the day, learning to build and debug asynchronous, multi-threaded programs is just plain hard. And we continue to get the feedback, even years later, that the time is well spent. I have an awesome class of 45 who, although exhausted, still seem deeply happy with what they are doing.
- GK: And since I mentioned Andy, rest his soul, he also loved that course.
- PA: I have fond memories of programming such things when using an ascii terminal to an under-powered VAX11/750 running BSD4.2 with it crashing every now and again... your students should be happy their development environment is pretty stable!
- RH: @Margo: I agree. It was a good class. It retains, however, the distinction of being the only 40-hour-a-week class I ever took. Second place was about half that. @GK: If those students are anything like me, not starting soon enough wasn't the problem.
- MO: it's no fun without an all-nighter once in a while
- MO: Of course, I never took the course, just had the enjoyment of watching BF and others suffer through it
- MW: Wow, 45? That's awesome.
- PR: It's a public service. Think of of the trouble those students could be getting into if they weren't locked up building an OS.
- NM: I wonder how many techies at the likes of Google, Facebook, etc. would still get a chill down their spine at the mere mention of "assignment 3"?
- DK: I still fondly remember our all-nighters in CS161 (with CY). Professor Hsu said we built the class's most elegant OS (everything was based on a common semaphore implementation). And it always ran for at *least* 30 seconds before crashing.
- CY: I remember the class being a 9 to 5 affair. PM to AM, as that was the only time M, D, and I had in common. And I fear that the elegant semaphores were woefully slow--the guys that used atomic ops to access the priority queue were twice as fast as our version where the semaphores also managed priorities.
- DK: I think that was also the class where we wondered "what happens if you tell the unix shell while (1) fork() ?" and took down the department compute server for the night.
- CY: I think it was only microvax-6.husc.harvard.edu, out of of a fleet of 10 or so microvaxen, and not the file server (thank goodness). The Magican's Apprentice moment was priceless, though, as we tried and failed to kill the parent process and started getting "no more processes" messages from our other shells. The next morning, the sysadmins were more worried that we were doing some important computation than about our abuse of the machine--they just power cycled it and everything returned to normal.
- CY: Actually, the real thing I should say about Prof. Hsu's CS161/261 class is that, of all the hacking classes I took at Harvard, it was by far the most useful in the rest of my career. I think it was the construction of synchronization primitives in a raw environment and figuring out how to debug code built on top of those primitives--there's nothing like it for combining power and danger, and for whatever reason being able to build such things remains a rare skill.
- DK: I think the "whatever reason" is that we have done such a good job of creating layers that shield students from the "raw environment"---students building a web application in Django have no idea how many layers of complexity are hiding between them and the hardware, and wouldn't know what to do down there. The hard question is whether this is a good thing or bad----given that we have already built the layers, do we want to force every student to really understand them, or is it enough for them to use them?
- And to make my colleague MM happy, I won't cut this one out DK: As for most useful classes, I've got to flag Umesh Vazirani's CS124, at least to the extent that "useful" means "determined where I am now".
And to top it all off -- this week's picture, straight from China: Tao Stein's students at midnight working on their ray tracer.
Monday, April 8, 2013
Missing Flipping
First, I recently learned that one of my students is blogging the course! I was thrilled when she said I could link to her blog. She is far more poetic than I, and I love reading her entries.
Second, I was chatting with a colleague recently who said that she couldn't tell whether I liked flipping or not. I thought I had been making that clear, but just in case, the point isn't coming across. I'm going to rant for a bit on just how excited I am about flipping.
I am completely sold on the flipped classroom. I will talk at great length to anyone who will listen; I recently used flipping as the centerpiece in the session I did for our teaching practicum; I have yet to hear a compelling argument for why you would not want to flip (as long as you could figure out space issues for a large class). I believe that I have covered all the reasons I am so enthusiastic about it, but just to put them all in one place:
- The teaching staff can focus time and energy on those students who need it the most rather than those who are most vocal.
- Pre-class work gives me concrete data on how students are doing -- what they know, what confuses them, what they don't know.
- Because the pre-class work is built into the class structure, I have a mechanism to obtain immediate feedback on most everything. I believe you could do this in a more conventional class as well, but it falls out naturally here.
- The pre-class work also gives me a way of staying connected with each student. I regularly check in on how partnerships are doing. We had one pair on the brink of divorce, but it appears to have been salvaged. I will point out that I have never taught this course without having at least one divorce, and it currently appears that we will have no divorces, even though we have more students than we've had in a decade.
- The physicists demonstrated that peer instruction works, and this confirms it. The students learn a lot from each other.
- Flipping can be the great equalizer -- if students already know half the material, they can skip the audio decks on those topics rather than having to sit through a lecture on them, just to get what is in the second half of that lecture.
- Preparing the coordinated materials makes me think much more deeply about what I'm teaching, how I'm teaching, and why I'm teaching. The end product is therefore better thought out.
- I've discovered that one becomes very sloppy while lecturing. You can walk into a class with lecture notes and wing it. However, I cannot wing recording an audio track. Because I can revise each slide track by itself, I will re-record until I have a very crisp, tight, clear explanation. I think this is a big win.
All that said, flipping, like any other mode of teaching, can be done badly. I've certainly experienced that. I believe that the work to be done outside of class, the pre-class quizzes, and the in-class work must be tightly coordinated. I also believe that the instructional staff needs to be fully engaged during class.
Monday, April 1, 2013
Inadvertent Flip
Peer Design Review
This was our second peer design review and I was struck by two things. First, it seemed that it took longer for the students to read through the design doc of their peer group. I speculated that perhaps the students had started writing more in-depth detailed designs, so I decided to ask about that in the week's web work (which are just short surveys at this point). Turns out that this was not what I was seeing -- about half the class claimed that the A3 design they reviewed was about the same quality as the A2 design they reviewed. And the other half were split as to which was better (there was a small tilt towards the A3 designs being better, but nothing huge).The second thing I noticed was an overall spirit coming out of the discussions. They seemed genuinely happy to be talking about fault handling and page allocation. They also seemed surprised that when they asked the staff certain questions, they got multiple answers, "I did it this way." or "Oh yeah, I didn't do it that way, I did it some other way." I thought this was good -- there are rarely right or wrong answers in design but instead a set of tradeoffs to be considered. And seeing that the staff didn't necessarily agree but could engage in a discussion of tradeoffs and relative merits was valuable in and of itself.
Participation
I have always graded on class participation, believing all the research indicating that people learn better when they are engaged. I know that class participation is difficult for some, and I always let students know that they can earn participation points by sending email before or after class to discuss things that they found either challenging or interesting. At the same time, I tell those students that my goal is that by the end of the semester they do feel comfortable speaking up in class. And nearly every semester, they get there, and it's a wonderful thing for all of us.Anyway, I always have students who never speak, never email, and never say anything about it. I've come to accept that as normal. Now, this year, I figured that participation grades were easy to come by because I consider web work and small group work in class part of participation. So I didn't worry too much about even my quietest students, because as long as they were in class, there was no way for them not to participate. Imagine my surprise when in the very same week, two of my students brought up the topic of participation. I was shocked. I'm convinced that this never would have happened if I'd been teaching this as a conventional lecture, but somehow these students, who were doing everything asked of them, somehow felt that they should be engaging more.
I assured both of them that from a grade standpoint everything was well, but I used it as an opportunity to engage in the greater discussion about how to participate, what I could do to make it easier, why participation was a good thing, etc. I'm curious to see what happens, but I was so (pleasantly) surprised to be able to engage in these discussions that I had to write about it.
By the way ...
Monday, March 18, 2013
Testing and Flipping
As I watched my students taking the midterm I was quite literally awestruck. Before me sat a room full of incredibly talented students, and I get the privilege of being part of their educational experience. The only other time I've been struck with a similar feeling was when I used to teach our (then) 300-person introductory computer science course (it's now about 750 students and growing strong thanks to the amazingly talented David Malan).
But this was different. These students have been working incredibly hard for the past three weeks getting their operating systems to fork and exec processes. A quick look at the data suggests that a typical student spent almost sixty hours working on the assignment over those three weeks. Yup, that's twenty hours a week for one of four courses. Ouch. I guess that explains the claims of students spending 60 hours weeks on this course -- historically there were always students who didn't do anything the first couple of weeks and then tried to do it all the last week. Things seemed a bit better this year with many people making steady progress over the three weeks.
Anyway, back to the exam. Having loaned my laptop to a student who had somehow missed the discussion about taking the test online, I had nothing to do other than watch them. There they were, all intently focused on the exam, typing, thinking, shuffling through slides. It was an open notes, open book, open course materials (but not open Internet) exam. I appreciated that they asked if they could read the course Q&A site during the exam -- I did draw the line there, even though they all certainly seemed to understand that posting questions was not going to be on the agenda. Towards the beginning of the test, there was much shuffling through materials, but as the test progressed, there was less. My theory has always been that I can make tests open book, etc., so long as I write questions that require synthesizing information so that the materials aren't actually that useful. There were a couple of questions where a glance at a set of slides would prove useful, but in most cases, there simply weren't places they could look for an easy answer, unless they already had a pretty good idea about the question.
I was quite curious to see the results. It's not possible to run a real apples to apples comparison beacuse a) this class is twice as big as it was two years ago, b) it's a different exam, and c) it's a different way of teaching. So, any results are open to interpretation. The results are surprising -- the distribution is almost identical to that two years ago. However, there were no truly bad grades -- last time, 2 of 23 students had grades on the midterm that were cause for alarm; this year, there were no grades that worried me (it appears that grades that worry me are those more than 2 standard deviations below the mean).
I'm pretty sure I cannot conclude anything from this. So, we'll have to wait until the assignments are graded. In this case, the assignment is pretty much identical (I saw pretty much because we asked different code reading questions, but ideally I can dig up the grade breakdown and remove those). Time will tell.
Saturday, March 9, 2013
Half Flip
Thursday was our annual midterm review. In the past I've simply gone over an exam in class or broken the class into fairly large groups and had them make up questions, present the questions, and then answer them. This time I let them submit questions as web work before class, promising that I would, in fact, use some good ones on the exam (having now written the exam, I have selected a few -- I did have a moral dilemma -- one student suggested a nice question, but answered it incorrectly -- I really didn't want to use a student's question and have that student get it wrong ...).
In class, I let them work at tables (groups of 3-5) on sets of problems and then we came together to talk about them as a class. Since many of my exam questions have multiple answers, it provided a good opportunity for both table-wide and class-wide discussion. I guess we'll see how it all works out next week.
I have noticed one other difference between this year and previous years. My staff seems to be thinking a lot more about teaching -- what we teach, how we teach it, etc. It's hard to attribute that to any one thing, because there are so many variables, but I do think that there is something about the setup that gives us more opportunity to think critically about what we're doing. Also, the fact that we have some flexibiity in how we present material means that we can be facile at adapting to the students, what's working and what's not. This week's discussion at staff meeting revolved around design -- how do we teach it? Why don't we teach more of it? What can we do now to do a better job? I have such respect for and appreciation of my teaching fellows who engage wholeheartedly in such discussions -- they are close enough to having learned the material that they remember what is hard and they are enough past it to know what skills matter. We're still in the midst of this discussion -- I'll let you know how it works out!
Monday, March 4, 2013
Unflipped!
We've entered that time of the semester when the students are busily creating user level processes in their very own operating systems. It is both exhilarating and exhausting. Due to that exhausting part, this is the part of the course where I promised not to make them do pre-class work -- there would be no videos to view and no web work (other than quick surveys about how much time they are spending). That means that, for the most part, I have had to revert to a more traditional class structure -- I talk and call on people and we try to collectively learn stuff. It feels dull; it feels unsatisfying; it feels as though students are not engaged. I am looking forward to polling them about this class structure so we can decide how to move forward during the next several assignments.
Since we're not flipped, I have little novel pedagogy to talk about nor do I feel the exhilaration I've felt throughout most of the semester (and any of you who have run into me and dared ask about the course know that I can now babble incessantly about how fantastic flipping is). So, today's entry is about two things: how it feels to be lecturing again and thoughts on what I would do if I had 250 students instead of 50.
The Traditional Lecture
It sure does take less time to prepare for class! Once I finish class notes, I have finished class notes. I don't have to think up pre-class activities; I don't have to think up in-class activities; I do spend a bit more time working through the notes, deciding when I can engage the class and which activities can be converted into class exercises.
As has historically been the case, I find myself interacting with about 20% of the class, instead of all of them. This is rather disappointing. I coax, I tease, I cajole, but some people just don't want to speak up in class. I do find it effective to ask for a raise of hands to vote on whether answers are correct or not and I seem to get better participation than I have in the past, and my sense is that even when I'm trying to extract thoughtful answers from people, I have a slighly larger fraction of the class engaged, but it's nowhere near the 100% that are engaged when the students are completing in-class work.
I also have found it effective to try to convert some of the open-ended questions I typically ask into exercises that the students can complete in groups. Even if it only takes a minute or two, it means that more people are engaged. It is this last tidbit that got me thinking about our next topic ...
Flipping 250 Students
A few of my colleagues have indicated that they have been thinking of how to flip significantly larger courses and it poses an interesting question. What would I do if I were teaching CS51, which is a heavy-duty programming class and has roughly 250 students?
Classroom space is the first obstacle. I love teaching in our new state-of-the-art classroom that has reconfigurable, mobile tables, traveling whiteboards, screens on both ends of the classroom, etc. However, it holds only 54 students, and even if you could build a bigger one, I think it would rapidly become sufficiently cavernous that you'd lose any semblance of a class. So, I would constrain my thinking to what would work for a more traditional lecture hall.
Since the students won't be able to cluster around tables, I'd probably try to use some kind of online shared whiteboard-like tool. That way students could bring laptops and work together in groups of 2-4. I would also want some way to do real-time data collection. I don't think you need any special clickers -- I would see if Google Forms work well enough in real time that people could submit answers and I could see the overall statistical picture of how students are answering.
The physical layout is going to make it difficult for the teaching staff to circulate -- you can get to the edges of rows, but getting into the middle would be tricky. I'm not entire sure how to deal with that -- I suppose the staff could join the online discussion, but it's not the same as interacting with the students directly. This would seem to be one of the biggest obstacles -- I just can't see the teaching staff climbing over students to get into the rows; having the students get out of the rows to talk to the staff doesn't sound better. I'm also not completely sure how the noise level is going to play out in a room with that many students. You really want them to be able to talk to one another, but a room of 50 sounds pretty chatty; I'm not sure how loud a room of 250 will sound.
So, those are my thoughts, but I think that the best thing would be to run a experiment. Perhaps if one of my colleagues invited me to guest lecture in his class, we could prep a single class and give it a try!
Thursday, February 21, 2013
Addicted to Flipping (week 4)
Tuesday I had scheduled for peer design reviews. Let me tell you a bit about how the course works, so that this all makes sense. The three meaty assignments of the course ask students to 1) implement user processes (fork, exec, and a handful of other useful system calls, etc), 2) add virtual memory, and 3) make a simple file system recoverable by adding logging to it. The students work in pairs and are left to both design and implement solutions themselves. We ask them to first submit a design document. The teaching fellows review the design documents carefully and then interview each team, giving useful advice, pushing them on how much they have really thought about what they need to do, clarifying any misunderstandings, and otherwise acting a bit like a project manager.
Historically, design documents vary widely in quality. A few are always nicely done, thorough, well thought out, and complete. Many are usually superficial, because the students are still struggling to wrap their heads around what they are supposed to do. TF feedback is invaluable.
So, this year, before handing the designs in to the teaching fellows, we scheduled in-class, peer design reviews. Each team met with one other team and exchanged designs and then spent an hour discussing and questioning each other on the design. The questions and discussions we heard sounded quite good -- people were asking all the right questions and the teams were pushing each other in just the right way.
At the end of the class, one student made a point to say that he thought the class had been incredibly useful. We will be getting the final designs this evening, so I won't know until next week whether this resulted in qualitatively better designs, but I remain optimistic.
But I suppose my real addiction came out Thursday. Since they are now deeply enmeshed in Assignment 2, as part of my contract, I will not ask them to prep before class. In general, I assumed that meant I would return to lecturing, but as I surveyed my notes on various advanced scheduling algorithms, I couldn't quite see me or the class getting excited by a lecture on this stuff. So, I took a risk.
I prepared four presentations on some scheduling algorithms (a Solaris style Multi-level Feedback Queue scheduler, the Linux completely fair scheduler, a BSD-style fairshare scheduler, and a lottery scheduler). Then I created six roles: four engineers, each of which is partial to one of the schedulers, one product manager and one engineering manager. Each role had some "private" information and the roles and information were on pieces of paper stuck in paper cups, 6 to a table. Students formed groups of 6, selected a role from the cup and then spent about an hour in a design discussion. Given their target platform (which only the product manager started out knowing), they had to select a suitable scheduler.
My goal was that rather than having me go through the plusses and minuses of each approach, they would be forced to compare and contrast and figure out when each one was good/bad. Then each team presented their target platform and their selection, with justification. We wandered around and answered questions and also assumed some other roles: CEO who has an opinion about everything, but isn't technical, VP of marketing who fundamentally distrusts engineers, and the head of QA who would be responsible for testing their systems.
Once again, it's difficult to tell, but my sense is that they covered the important points and each has a deeper, more gut-level understanding of the various approaches. Only time will tell!
Saturday, February 16, 2013
Not Flipping Out (week 3)
I'm still a fan.
There were no enormous disasters this week nor were there any sudden breakthroughs (I fully expect that simply due to this new course structure, one of my students will inadvertently do something to win him/herself a Turing award, but not quite this week.)
So, this week I will write about four happy and unintended consequences of this new arrangement.
1. Flipping as an Equalizer
Students enter courses with different levels of expertise and experience. This is always a challenge as some students constantly feel behind and others sometimes feel bored (although rarely do students feel bored while trying to get OS/161 to fork processes). With the flipped arrangement, I can break up the material into distinct topical areas with videos per topical area, and students can pick and choose which ones they need to watch. They can check out the web work and see which questions they know without viewing anything and which ones indicate they may need to view a video or do some more reading. My sense is that this acts as an equalizer. It's not perfect, but it's definitely better than a more traditional lecture where at least one student is lost and another is bored.
2. Maximizing Teaching Utility
I feel like an idiot for not having realized this before, but flipping lets instructors spend time with the students who can benefit most from that time. When I try to run an interactive lecture, I typically end up engaging with the students who are most comfortable and most expert. After all, they are the ones most willing to speak up in class. They are also usually the ones most willing to ask questions. Sure, I can look at the class and try to detect overall class confusion, but do I always catch the student who is really confused? I think not.
In contrast, when I and my teaching staff* wander around the room, we spend time with the students who are struggling or who have questions. We check on groups and cheer on those who not only have fork and exec working, but have implemented job control, backgrounding, programmatic scripting, and twelve other features in the past ten minutes. Yes, we're excited for the students who excel, but we can be more useful in making sure that all the students are succeeding. And by and large, at the end of the session, every student has experienced some form of success. I am pretty convinced that few students walk out of a normal class feeling successful. They may feel like they were in a good lecture; they may have enjoyed it; but I don't think they feel that they were personally successful. In contrast, getting your shell to fork and exec, proving a scheduler optimal, or deriving a good scheduling routine can let you feel like you've actually accomplished something.
3. And then I travel ...
As much as I hate it, I do sometimes have to travel during the semester and miss classes. While I have talented colleagues and wonderful teaching staff who fill in for me on those occassions, I don't like it. This February is the worst -- I have three trips I have to make. Under normal circumstances I should have missed four classes, but I couldn't do that so I jiggered the trips so I will only miss two (which means I missed NetApp University day for the first time, which made me sad).
All that said, being away is not nearly as disconcerting as it is under normal circumstances. First, I can check on the web work so I have first hand knowledge of how everyone is doing. Second, I have already prepared the material I wanted to present, so that isn't very different. Third, although I cannot wander around class, I have four or five teaching staff fully engaged in the class time activity and they are wandering around talking with students and getting firsthand feedback on how things are going.
Normally, when I've missed a class and I ask how things went, I get something like, "I think it went OK." This is a truly honest reply, because someone giving a guest lecture has very little feedback on how things actually went. This time, however, I got, "Very well. I guess main things I noted:
- People seemed good conceptually; questions I got were insightful.
- People had a lot of trouble with the proof, so I eventually told them to move on to the design problem and come back to the proof if they weren't done.
- I had time for one group to present their scheduler idea and it was essentially good (a min-heap with priority proportional to sleep time).
- I had time to show the bounded priority queue idea and explain that the distribution of jobs is usually thought of as multinomial. (The group that came up with the min-heap scheduler included a math student who was trying to show it was somehow optimal for a power law job distribution, which was pretty neat).
- I think some people may have been working on A1 in class, but I'm not sure that's bad."
I have a much better sense of what was covered, how students are doing, etc. Yes, it took more work before I left, but it was entirely worthwhile.
4. My Teaching Fellows seem more engaged and happier
* At Harvard, teaching assistants are called Teaching Fellows. This is a moniker that arose when most such people were graduate students and the stipends they received were part of their fellowship. Technically, when undergraduates assume these roles, they are teaching assistants and not teaching fellows, but drawing such distinctions would be silly, so the culture is that we call all such people Teaching Fellows, or more colloquially, TFs. Apologies for not clarifying that earlier.
In a conventional course, the TFs are pretty disengaged from lecture. I usually
want them to attend (mostly to fix things I botch up), but since they've all
taken the course before, they spend much of class tuned out and doing their
own thing (defeating the purpose of my having them here in the first place).
However, in the current setup, the TFs are fully engaged. They are wandering
around talking to students, fixing things, and actually
And, at least to me, they say that the class time is fun. This too strikes me as a good thing.
Thursday, February 7, 2013
Flipped Again (Week 2)
Part One: Time Intensity
Regardless of how I teach, I try to prepare thoroughly. When I lecture traditionally, I hand out notes, so students have the outline of what we're covering, but I leave a lot of blank space that I try to fill in collaboratively with the class. However, I always have the main points jotted down on my notes, so I make sure to cover the things I want to cover. After class, when I discover errors or things that didn't go well, I either fix the notes then and there or leave myself notes for next time. In either case, my notes and lectures are constantly evolving and changing.
So, how does that kind of preparation translate into my current situation? First, taking last year's notes, updating them, and then preparing the audio track requires at least a factor of two more time, perhaps even more now that I'm thinking deeply about it. Applying the same fixes I would have applied were I teaching conventionally takes about the same amount of time it would have. But then I start recording audio tracks. In my case, I'm using discrete audio files per slide, attaching them to the powerpoint, and then converting that powerpoint deck to a movie. The logistics of all this are tedious and take time, but that's not interesting (e.g., the PPTX->movie conversion really only works well using software that runs only on Windows; it produces enormous files, which I then translate using a different tool; these large files get moved around, etc).
The more interesting is what goes into preparing the audio bits. Since students will be listening to these on their own, they don't have the opportunity to interrupt and ask clarifying questions. We do have an interactive message board, and the students are taking good advantage of it, and the TFs and I are working hard to be responsive, but it's not the same as saying, "Excuse me, I didn't understand that." OK, few students ever actually say that in class, but when you can see them, you can frequently figure out that they are thinking that. In lieu of that, I think very carefully about what I want to say, how I want to say it, and then I record. I'd say right now about one in three times I am happy with my first take. One in three times I need a second take. One in three times I do many, many takes. So far, the videos have mostly come in around 30 minutes, so we're talking a couple of hours when you add in recording with incorporating the audio track, setting options, fixing things, etc.
But once I've completed the audio track, then I can really begin. Next I think about what I want them to take away from the video and I translate that into pre-class work. I think I am posing somewhat more challenging pre-class questions than one normally associates with this task, because I use those results to help me identify what issues are really confusing people. (I spend an hour or two in the morning before lecture reviewing answers, analyzing responses to the various questions, discussing them with my staff, and figuring out whether I want to a) simply post answers, b) go over problems in class, or c) do nothing.) In any case, I spend somewhere between 30 and 60 minutes thinking about and writing pre-class work.
Then I begin working on the in-class component. In our case, this involves writing code. Writing code always (always, always) takes longer than you expect. Testing is key. However, since my pedagogical thinking evolves as I am working through the assignment, sometimes I don't always use the practices. This brings me to item two on today's list.
When Disaster Happens
We had no fewer than four different types of disasters happen all in the span of a single 90 minute class (of which 30 minutes was consumed discussing pre-class work). There are many things to learn from this experience, so I'll walk you through what happened in potentially gory detail.
1. Pre-class work
It turns out that unless students were clairvoyant and could read my mind, there was much ambiguity in the questions as I wrote them. Both TF feedback and the analyzed data supported that claim. So I took the first 30 minutes of class to go through the questions and discuss the different interpretations and how those interpretations led to different answers. My assessment is that while it didn't feel good to be me, the end result was actually pedagogically constructive. We ended up talking about a number of important concepts and assumptions as a group, and I think we raised issues that, in past years, never surface, are never addressed, and manifest in difficult-to-debug code later in the semester. It will be difficult to do any kind of hard empirical analysis of this (although if people have ideas, I'm all ears), but my sense is that we had a deeper discussion about synchronization than typically happens.
2. Preparing for in-class work
After that, we turned to the class work. I had prepared the exercises as a separate branch, with the intention that they could play with it in class, and it would not in any way interfere with their own work, and afterwards they could either keep it or toss it without any effect. Great idea in theory. In practice, due to a combination of miscommunication, my own experience with the particular source code control system we're using (there were good reasons to change, but it's not one I've used extensively), and pretesting the process on a tree that was not identical to what the student's had, we ended up having them execute a command that, well, um, kind of messed up their tree.
3. Dealing with the in-class disaster, or "Another Disaster in the Making"
When then insued was sheer insanity. Over the course of the next 15 minutes, we frantically proposed multiple different ways of fixing people's trees. The TFs, students, everyone had an opinion, and we kept posting them. This was just dumb. We should have stayed calm, figured it out, and then dealt with it. But no, we freaked out -- all of us!
Finally, I went around the room and checked on every person to make sure that they had a working tree. That seemed like a good idea. Hoewver, then the fun really began.
4. The Crowning Touch
As they started working on the problems, they discovered truly, horrifically idiotic bugs in the code. How did that happen? I really tested things. I really did. Honest.
Ah yes, I'd done a couple of those last minute, "Hey -- this will make things easier to debug!" fixes and of course did not rerun the tests. That will teach me. Class ended with me feeling like I'd wasted a lot of truly valuable time and the students probably wondering what on earth they'd signed up for.
Transparency
So what do you do after a class like that? Well, I tried really hard to do better on Thursday. Although I did forget to have people put their names on the pre-class work, I didn't make any monumental errors. I did however, address the situation head one. This is what appeared at the beginning of Thursday's work:
-
First, let me start out with an apology -- we did not use your time
efficiently on Tuesday, and that's a violation of the "contract" I
made. I hope you all believe it certainly wasn't my intention, but
that doesn't excuse it. I must, and will, do better. There are a
number of ways in which things went badly, and rather than focusing
on what they all were, I'd like to focus on how to avoid them.
Thanks to Carl, we have a nice simple mechanism in place to help
us prevent git shenanigans. Please refer to the Piazza posting.
All that said, one takeaway is that at some point in the semester,
if you find yourself in the midst of a git disaster (e.g., you and
your partner cannot figure out how to resolve conflicts, get each
other's changes in exactly the place and way you want them, etc),
don't panic. There are ways out. Stay calm, learn to read git commit
logs so you can figure out where things have gone astray and patiently
try to fix them (we did not demonstrate the "patiently" part yesterday
-- we frantically tried N fixes in a course of N-1 minutes trying
to muscle through it; not a good strategy). So, as promised, today's
exercise is simply to finish what we started Tuesday, ideally with
significantly less drama!
I also tried to use the experience as a teaching moment, even if it was at my expense, so I also wrote:
-
There was another bone-headed error that got in here -- I will
explain how this happened, just as a warning about "last minute
changes." I had completed the problem, implemented a solution,
tested it, and was happy. I then removed the synchronization to
commit. "But wait!", I thought, "It's too difficult to debug, because
there was no way to determine which TF is handling which student's
question. The printout simply isn't helpful enough. I'll add a tf
array so I can print out a TF number and that will be easier to
debug." So I tossed it in the TF array at the last minute, and
(here's the boneheaded part) did not rerun all my tests. So, take
this as, "Don't do that!" No matter how minor the change you think
you're making is, you have to rerun your tests. Especially if you're
tired, that "simple" fix is undoubtedly wrong in at least three
ways. Mea culpa. This was the second patch (semprob2.patch ) sent
via email. Apply that patch too.
The Conclusion
So after all that, how did Thursday go? Remarkably well. The students seemed to accept my direct apology and commitment to do better. Perhaps some of them will chime in on their perspective (perhaps not; the one who are really upset will wait until the semester is over, and I understand that). And, by the end of class, all the students were animatedly and happily chatting about whale-mating. It was good to hear!