Long ago (1988) I moved to Berkeley and started sending a monthly "newsletter" to my Boston friends. When I returned to Boston (1993), I continued the tradition for about five more years (or until I had kids). Looking back, I realize that I was actually blogging. Each newsletter contained anywhere from a few to several blog posts. Having been silent for the past decade or so, I've decided to resume these activities. Don't expect anything profound -- I tend to focus on what I find entertaining or amusing and perhaps sometimes informative. We shall see!

Saturday, April 29, 2017

Flipping at Scale: FAQ

Last week I ran a small faculty workshop around the topic of flipping at scale. We only had time to scratch the surface on so many issues (that my colleagues diligently raised during our hands-on exercise), so I promised to write down the questions and produce a FAQ. However, my goal is not to suggest that I have answers to all the questions, but rather to share my experience in the hope that others will share their experience and this will become a living online discussion.

If you want to read more about my take on flipping, here is the index to my weekly blogging of my first experience trying this.
Questions about flipping in general
What do I do in a flipped classroom if there is no textbook or we have one, but it's difficult to understand?
How do I accommodate the wide range in my students' backgrounds?
Doesn't an instructor's enthusiasm come across best in lecture?
How do you avoid students "hiding" in the group work?
How do you provide feedback to students -- in real time and coherently?
How do you detect/correct student misconceptions?
How do you design exercises so that you guide them to discover what you want them to discover?
What kind of pre-class work is most useful?
What do I do about non-intuitive content that students are unlikely to figure out on their own?
How do you avoid cliques in the small group work?
How do you ensure that students have the necessary basis so they can successfully tackle the in-class exercises?

Questions about scalability of flipping
Doesn't the room get really noisy when you flip a large class?
How do you get students to buy in to flipping?
How do you figure out the right TF:student ratio?
How do you help students not be distracted by their devices during flipped sessions?
How do you flip a large class in a 60 minute slot?
How do you (the instructor) get to every table?


Questions about flipping in general

What do I do in a flipped classroom if there is no textbook or we have one, but it's difficult to understand?

I am, perhaps, not the best person to answer this question as I rarely teach from a textbook. I typically select a book to use as a reference, but I usually invent my own material and perspective on the subject. I don't claim this is good and I can't even tell you where I got the idea that this was reasonable, but it's how I do it.

So, in the context of flipping, I tend to prepare my own pre-class materials -- typically videos (prepared using Microsoft Mix, a Powerpoint Plug-in, that I adore). I will also usually include readings from a book that I've selected because it explains many things well, rather than because I follow its structure that closely.

If I present material in class, I'll make any slides available (but I'm careful to explain that the slides are prepared for my benefit to help me lecture; they are not designed as replacements for lecture, so if there are lists to be filled in or open ended questions, I do not feel compelled to complete them.

How do I accommodate the wide range in my students' backgrounds?

This is actually one area where flipping is, I think, ideal. When I think there are areas where students may be missing background, I'll prepare supplementary pre-class material (i.e., videos). I'll include a couple of questions on the pre-class surveys about this material and suggest that if students can answer the questions without previewing the material, then they can skip it; if they can't answer the questions, then the material is available to them.

Doesn't an instructor's enthusiasm come across best in lecture?

To some extent I agree that enthusiasm is best conveyed in lecture. However, I hope I don't offend anyone by this, but that really turns lecture into marketing. That's not inherently bad, but I don't believe that we need to devote two times ninety minutes times fourteen weeks to marketing.

It is possible to capture some of that enthusiasm in videos, but I think you can also convey that enthusiasm when you set the stage for the in class work or wrap up at the end of class. Some ways I do this include:

  • Going over the pre-class work, highlighting particularly interesting aspects of the questions, common answers that are incorrect in interesting ways, or anything else that is engaging.
  • Setting the stage for the in-class work -- "Today we're going to do an exercise that is based on a bug that caused the entire East Coast web to stop working yesterday."
  • This one I do a lot: I structure an exercise so that at the end of it, the class has (re)discovered something quite fundamental that was a huge breakthrough in the field. Then when we discuss the exercise at the end of class, I point out how they made progress on something that was once upon a time a huge open problem. This seems to work quite well.
  • When students ask questions during the in-class exercises, I'll take notes and after the exercise, I'll discuss the questions that come up frequently or any particularly interesting ones. I'll also engage with the students and point out how things they've done relate to relevant real-world problems.
I find that the genuine love we all have for our material can across in pretty much any interaction we have with the students -- 1:1, in small groups, in front of the class, and even in a video or audio recording.

How do you avoid students "hiding" in the group work?

On one hand, I think this is a real challenge; on the other, students hide in lectures all the time. I think you have to be comfortable engaging a quiet student in a question about the material. Some students are simply inherently quieter; they may still be absorbing material and be mentally engaged, but you can't tell. Engaging and letting them know that you care if they are learning goes a long way. Teaching your TFs to do this also requires lots of encouragement.

How do you provide feedback to students -- in real time and coherently?

This was, by far, the most frequently asked question.

We have tried a number of different approaches that get at the heart of this question. In smaller classes I frequently bring the class together and let each table present parts of their results/solutions. With 15 tables, each table can contribute something in a reasonably short period of time. With an even smaller number of students I've done something I call a "wandering design review" where we walk to each table as a group and have each table present (I like having the students up and walking about).

In larger classes, this becomes trickier. Posting some kind of solution to the in-class exercises is pretty much essential. We've done this two ways -- one is simply providing written solutions. The other (which was particularly popular among our extension students) was to produce video walk throughs that have the instructor talk through his/her actively solving the problem.

The precise details here are going to vary based on the kind of problems. My high order thought is that you have to provide some kind of solution that students can learn/study from. Even if you do something in class, you have to have something more long-lived that they can refer back to.

How do you detect/correct student misconceptions?

This is mostly what should be happening during the in-class exercises, when staff are wandering around the room. I'm sure there is some confusion that persists, but if students do what we ask (i.e., work collaboratively with others), then we hope that either their peers help them or a staff member helps them. We find that students do not raise their hands when they are stuck, however, so it's really really really important to intervene and explicitly check in and ask how things are going at each table. Some of the things I'll ask are, "Progress or stuck?" "How far have you gotten?" "Anything preventing you from making forward progress?" It shocked me how many times I'd approach a table and discover that students were stuck, but weren't raising their hands (contrast this to office hours, where students will ask a question every time they encounter any obstacle -- I don't understand it).

How do you design exercises so that you guide them to discover what you want them to discover?

This is both the most fun and the biggest challenge. I typically work a bit backwards from what I want them to figure out/discover and then also work forwards from how I used to present the ideas/concepts when I used to lecture. Having lots of small steps is part of the solution. Having checkpoints where you ask the students to step back and answer some conceptual questions also helps. In a couple of cases I've suggested that students who feel more comfortable should skip the step by step instructions and dive it. This lets me guide the students who are struggling the most in a step by step fashion to exactly where I want them to get.

What kind of pre-class work is most useful?

Wow -- this is a huge question. I'm inclined to cop out and say, "It totally depends on the field." Let me try to not cop out completely. I focus on using pre-class work for first presentation of material. This can be done either via a video or reading, although my impression is that students are more inclined to watch videos; they've become used to the idea that reading is redundant with what will take place in lecture. So, I think that the idea of pre-class work is that it gives students the opportunity to become familiar with material before we ask them to use it in completing exercises.

What do I do about non-intuitive content that students are unlikely to figure out on their own?

I'm going to fall back on, "This is what in-class work is about." That is, just because you explain it in lecture, doesn't mean that the students "get it." If you present the material (e.g., via a video) and then ask them to use the material in a step by step manner, you can intervene when they get stuck. This is precisely the beauty of a flipped approach -- it gives you a chance to get to each student at exactly the point s/he gets stuck, instead of having to drag the entire class through every possible point where a student will get stuck.

How do you avoid cliques in the small group work?

This is really a fundamental question: Do you let students arrange their own groups or do you assign students to groups? Personally, I've done almost exclusively student-selected groups. The only exception is in my operating systems class when we do peer design reviews. We discovered that if the teams of students were not at roughly the same level, then at least one group and sometimes both did not have a positive experience. In that case, I roughly divide the course in thirds and assign pairings within the thirds. In other courses, I've seen two other models: one of our courses randomly pairs students together; in another, students are carefully paired, but I don't know the algorithm. The bigger problem I find than cliques is students who are isolated and don't know anyone and then work on their own. I try to engage those students, but I don't think I'm great at it. I think a combination of the student-selected, randomly-assigned, and carefully-assigned might be worth trying.

How do you ensure that students have the necessary basis so they can successfully tackle the in-class exercises?

I think this is the role of the pre-class work. Naturally, we have expectations about student's preparation and I find flipping is actually particularly effective at addressing different backgrounds. I like to provide either my own background materials to help students without the proper background come up to speed. For example, in my OS course I want students familiar with synchronization primitives and how to solve synchronization problems. Some students have this background; others don't. I use materials I prepared for a different course as background material for this topic. Some students view it, others don't.

In some areas, you need not even prepare your own materials -- you can direct students to other existing materials. You need students familiar with solving simultaneous equations? Point them at your favorite Khan Academy video.

Short story is that if you have students used to completing pre-class work, and you make supplementary materials available that way, then this is a total win!


Questions about scalability of flipping

Doesn't the room get really noisy when you flip a large class?

This is a great question. The noise does not seem to be a problem for most students. However, there are a few students for whom the environment triggers sensory issues. I completely missed the boat on this the first year we flipped CS61, but after one student explained this to me, I made a habit of reserving a small room adjacent to the big room, so that we could let a few groups with such students work in a less stimulating setting. Unfortunately, I never found a good way to advertise this in a way that didn't result in students taking me up on it to feel uncomfortable. This is still a work in progress.

How do you get students to buy in to flipping?

Transparency, transparency, transparency.
  1. I explain why I flip ("Research suggests that you will learn more.").
  2. I explain how the pieces of the class fit together: Videos present concepts, exercises let you experiment with those concepts and prepare you for tackling the problem sets, problem sets let you develop mastery, exams test that mastery.
  3. I tell them that they can always ask, "Why are we doing this?" and if I don't have an answer, that's my problem and I will fix it.
  4. Slide 3 contains my "contract" with the students.
  5. Then I try to live up to my contract -- it's not always easy!

How do you figure out the right TF:student ratio?

Honestly: experimentation. A TF can handle about 3 tables; sometimes it varies; I like assigning each TF to 6 tables, but then have every table covered by at least two different TFs.

How do you help students not be distracted by their devices during flipped sessions?

I'm not sure this is any different from the same issue during lecture. Well, in flipping it is both better and worse. It's better in that you're wandering around and you can see what they are doing. It's worse, because everyone has their devices out, so you don't know who's doing what until you approach a table. The bigger problem we encounter is students who want to ask questions about the homework during flip time. We try very hard not to answer these questions, but sometimes they are desperate, and it's hard to hold firm.

How do you flip a large class in a 60 minute slot?

I would probably flip the entire class. Provide pre-class work. Introduce the exercise for 5 minutes; work on the exercise for 45-50 minutes and then bring the class together to wrap up. This might require more post-class prep to write up solutions or answers to Frequently Asked Questions.

How do you (the instructor) get to every table?

I try to wander the room in an orderly fashion, but will subvert the order to attend to a raised hand (however, you may recall above I said that hardly anyone ever raises hands, so the orderly fashion pretty much works). A lot of the students will be happily working, so tables will take different amounts of time. When I visit a table, I do some combination of:
  • Check in: How's it going?
  • Learn names: OK -- let's see if I can learn all of your names...
  • Answer a question
  • Pose a question that tests whether they've gotten key concepts
  • Peer over their shoulder while they are working
  • Ask them how their semester is going in general (this is often quite revealing)

Thursday, April 20, 2017

Happy Birthday Harry

As some of you may recall, I have an odd fascination with Dr. Seuss. I've posted a few before. So, in that spirit, on the occasion of my advisor, mentor, and colleague, Harry Lewis's 70th birthday, I give you:

 Harry Hatches CS.

Sighed Martin, the Spartan Dean of D-A-S
I'm stymied and I'm bored; what is this C-S?
Is it math? Engineering? Neither or both?
It's something quite different I swear under oath
Let me create some new courses said Harry
Theory of computation won't be scary

It's mathy and logical just right for me
I'll make some new courses and teach them you see
Then Harry, the young guy, started creation
He called the course, theory of computation
Then AS eleven started migration

To the course known today as CS fifty
You didn't know that? It's really quite nifty.
Algorithms next, called AM one nineteen
Now it's one twenty four, you know what I mean

Before a CS concentration was made
In many areas the students played
Some did AM, while others studied Eng Sci
But eventually the right thing did fly
Computer science became a field of study
Intrepidly led by Harry our buddy

A concentration must have a head tutor
The chief advisor and main troubleshooter
Harry quite naturally stepped into this role
Dedicated to students in heart and soul

And as CS grew, Harry knew what to do
After careful analytical review,

Come Henry and Stuart, Margo and Michael
Salil and David continue the cycle
Then Sasha joined to make the number seven
Only four more and we'll be at eleven!

With computer science now growing and blazing
Harry did something even more amazing
The Dean of Harvard College, Harry was named
Still teaching his course, he became quite acclaimed

House randomization came as no surprise
His report of the college did so advise
Random numbers play a key role in CS
So should they too for our students' address

Harry decreased` the size of blocking groups too
From sixteen to eight, I'm afraid that it's true
But during his Dean's tenure, there was no doubt
His devotion was to students throughout
Perhaps lesser known is how much all admired
Dean Lewis whose efforts were always inspired

After eight years of Deaning in faithfulness
He returned to his home in good old CS
The job of head tutor became the nutty
Director of undergraduate study
Assuming this title in style of no peer
Advising more students than days in a year
He leads by example; he shows us the way
To care for our students in the proper way

Perhaps his service to Fair Harvard was through?
Oh no, oh no, there was still more he could do
With Cherry departing and no Dean starting
Harry did what he could before departing
He became the Interim Dean of our school
Fairness and respect to all was his ground rule

He's had so many jobs, so many titles,
We could go on and on with our recitals
Instead, we'll mention just a few titles more
Because they allude to what might be in store

He's a father, grandfather, father-in-law
To us he's much more, given what he oversaw
He's a teacher, advisor, mentor and friend
None better on your side could we recommend

As the clock on two years begins winding down
Here is how to avoid our collective frown
Go forth and have fun teaching CS classics
With your wit, charm, and happy dynamics
Back to Montana for the best fall around
But come back to Cambridge where colleagues are found
We wish you the best and Happy Birthday too
With our love, respect, and gratitude. Thank you!