How to Present your Research: Part 4: Presenting Results

We've talked a lot about story-telling so far, and I bet you're all wondering, "OK story telling is fine, but I have some actual research results I want to show!  Let's get to that." And indeed, we will. But, guess what -- each result is a mini story.

A result story has three basic parts: 1) Why am I showing these results? 2) How did I obtain these results? 3) Here are the results.  Let's dive into each one.

Why am I showing this to you?

Each experiment you perform or result you show should be either answering a research question or showing evidence to support/refute an hypothesis. I think these two are really the same thing, so for the rest of this post, we'll call them research questions (RQ for short).  Note that it has become a lot more common in recent publications (even in systems and ML) to explicitly state your research questions. I encourage you all to do this before you write any experiment code -- it's great research and mental discipline.

Before you dive into the details of the experiment, explain what RQ you are answering. This could be no more than a sentence, "We want to know if the overhead of OurGreatIdea is acceptable, so we conducted N experiments to determine if we were able to do SomethingAmazing without introducing more than X% overhead."  Wait -- notice how I snuck that goal in the end? I think that 99% of the research out there asks the question, "How much overhead does something introduce?" And then, regardless of what that overhead is, we declare victory!  That's not exactly science. I urge you to think (before you run any experiment), "How much overhead is acceptable?" Once you do that, then you can decide if your overhead is great, good, acceptable, almost there, or terrible.

But, I digress. Sometimes, you simply have to state the RQ. Sometimes, you might want to go into a bit more detail. Some of the greatest insight comes from ablation studies; these answer the question, "Which of the N things I did account for the great results we get?" Let's say that you built a system that introduced three optimizations. People will wonder both A) How much does each thing matter? B) Do the results compound? In the case of such an ablation study, you might want a slide to review the various things that  you did and then explain that you're going to introduce each one by itself or remove each one and leave the others in. Depending on your system, one of those is likely to make more sense. If you can do both, go for it!

A cousin of the ablation study answers the question, "How much time do we spend in each part of our BigComplexSystem?" Usually, you can just explain that with a single sentence; sometimes you'll want an architectural picture to explain it.

Regardless of the complexity of the experiment, please explicitly tell them what question the result you are about to present is going to answer.

How did I do this?

Explain the experiment!  It sounds easy enough, but you'd be amazed how many times I am shown a graph with 0 explanation of: A) the data used, B) the workload run, C) any details about whether I am seeing averages of many runs, a single run, the best of many runs, etc. So, describe your experiment.

In a paper, you will want to give precise details about the machine on which you're running. In a presentation, I think it's fine to categorize the platform, e.g., "We ran this on my laptop." or "We ran this on a huge server." or "We ran this on a typical server with a Big Honkin' GPU."  You can go into more specifics, but I think just putting the detail on the slide is more helpful than saying the particular model of processor you are using (no one can wrap their heads around that verbally; OK maybe someone can, but I do not think most people can).  Typically people want to know how much main memory you had and if you are running a modern machine. If you are doing storage work, then of course, they want to know something about the storage system too.  If you are doing ML, they may care what GPU you are using. Think carefully about what your audience wants to know and the best way to present it (on the slide, but say nothing; on the slide and highligh key features; on the slide and read it).

Describe the workload!  Tell the audience whether this is a standard benchmark in your field or one you made up (and if you made it up, you are going to have to justify why). Is this a microbenchmark or a macrobenchmark?  Is there input data? Where did that data come from? What exactly are you going to show? The average of N runs? One run? One cherry-picked run? A number you made up?  Sometimes, you might even say, "Here is what we expect a good system to do on this benchmark."  Then you can compare that strawman to your actual results. Alternately you might way, "To the best of our knowledge, the best system for this is X and here is how it does on this benchmark." Again, then you can compare your result.

Your goal is that when you finish describing your experiment, the audience is excited to see your results.

My Results

Tell me what I'm looking at! Again, this seems obvious, but I have sat through too many talks where a speaker excitedly tells me how great these results are before I have any clue what I'm looking at. So, breathe ...

Most results (in systems) are either tables or graphs/charts (I'll use the term graph here to describe a visual depiction of data). In either case, describe what the audience is looking at. For example:

Graphs

This graph has throughput on the Y axis and the number of cores on the X axis (if you are using log scale, say that explicitly here, even if it's in the axis labels which it should be). So, big bars (higher numbers) are better. [Please, always add this last part!]  Sometimes I just show the structure of the graph with no data as I explain this.  That allows the audience to understand what they are going to see before they get distracted with the actual data. The error bars show standard deviations, the middle 50th percentile, whatever -- tell me!

Tables

Most of the same rules apply here. Each row represents something and the columns correspond to something else. Tables of numbers are relatively difficult to comprehend, so I encourage highlighting the things you want the audience to take away from the tables -- embolden the best in each row/column; color code good/bad/mediocr results. Basically, help the audience take away the message you want them to take away from your results.

Takeaways

Draw the listener's attention to the things you want them to get from the figure. Do not assume that your results are so obvious that they will get exactly what you want them to! Inevitably, they will draw a wrong conclusion or one different from the one you wanted them to.  Walk them through the results.  I like to compare and contrast with what we might expect. Some examples:

We expected that we would scale well until we were running one thread on each socket, but we found that we actually scaled well until we were running one thread on each core. As we'll see in the next graph, our cache footprint was small enough that the data fit nicely in the per-core caches. [This is a great technique to explain WHY you get the results you do -- show a macrobenchmark result and then follow it up with a microbenchmark result that explains the results you got in the macrobenchmark.]

We had hoped that participants using our tool would complete the task more quickly. While they did produce better solutions (point to the part of the picture that shows that), we noticed that they actually took longer. [When you have a surprising or unexpected result, this usually indicates something interesting, even if it didn't match your intuition. In this case, perhaps it's something like, "Our qualitative survey results suggest that people found our tool so much more enjoyable to use that they were willing to spend more time using it to produce high quality solutions; in contrast, old tool was so painful to use that participants stopped as soon as they produced anything close to correct."]

In my speaker notes, I almost always have a numbered list of 3 +/- 1 key points that I want the audience to take away from any result I present.  I do not write the details of those points in the notes, because I want the timing of my explanation to match the ability of the audience to comprehend what I'm saying. If one writes out the exact explanation, it is almost always the case that that explanation is presented too quickly for the audience to comprehend. If the speaker has to think about it, then their delivery almost always better matches the audience's receptive timing. I think this point is crucial!  When we are nervous, we almost always start to speak more quickly and often become quieter towards the end of our sentences. You want to do everything possible to avoid this, especially when presenting results.

Key Takeaways:

1. Don't forget any of the three parts: question I'm answering, how I am answering it, what I found.
2. Explain the format of the results you are presenting.
3. Use microbenchmarks to explain the results of macrobenchmarks.
4. Highlight surprising results.

How to Present your Research: Part 3: Status Updates

Normally, we think of presentations in terms of formal talks. However, I am willing to bet that we all spend far more time giving information status updates (which are, in fact, presentations) than we do giving formal talks, yet I do not believe I've ever seen any guidance on how to do that effectively.  How strange!

Well, here goes ...

Surprise: I'm going to argue that every status update is, wait for it, a fairy tale! (This is going to be a recurring theme.) Before diving into the details of that fairy tale, let's talk about different categories of status updates.

The 1:1 meeting with your advisor/supervisor/team lead. At some point, you'll find yourself in a 1:1 with someone we'll call "your boss" and you'll want to update them on what you've been doing. The key thing to help organize your thoughts and presentation for this meeting is to ask yourself, "In how much depth does this person understand what I'm doing?" In some cases, the boss will know exactly what you are doing and could be doing the task themself.  In other cases, the boss has a big picture idea of the project, but is not knee deep in the details. Obviously, you're going to have to present your work differently depending on the situation you are in.

The project meeting. This is typically a small meeting with the people directly working on the same project you are working on. In this case, you can assume that the others in the meeting know the project-level fairy tale, so you are going to focus on the smaller fairy tale describing what you are doing.

The large group meeting. This meeting includes people working on projects different from the one on which you are working. Even if you see these folks every week, you probably cannot assume that they remember from week to week exactly what you are doing, so you need to plan accordingly.

Regardless of which of these meetings you are having, you are still going to present your status in the same structure: 1) some context setting, 2) a discussion of the challenge you were addressing this week, 3) where you are in addressing that challenge, and 4) what comes next. (Sound familiar?) It doesn't matter if you're winging this entirely verbally, giving a formal presentation with slides, or communicating via interpretive dance, you pretty much must have all four parts.

So let's focus on some of the finer points in how to think about this.

The 1:1 Meeting

You can pretty much assume that your boss knows the overall project you're working on. If they aren't knee deep in your work, I suggest starting by reminding them of your overall task, e.g., "I'm working on our server performance," and then helping them recall where you were at the end of your last meeting, e.g., "Recall that when last we met (or at the last project update), I had been working on <something> and had gotten to <some particular point>." If they are knee-deep in your project, you can probably skip that. But even if they don't need that context, because, for example, you just met yesterday, I still recommend starting out with what you were working on during the time since the last meeting. For example, "My focus the past day/week/month has been on improving the throughput of our server." If this has been a long-term undertaking, then perhaps there has been progress reported at previous meetings, "And so far, we have 1) added multi-threading, and 2) partitioned the data structures to avoid too much synchronization overhead."

At this point, you've brought your boss up to date on the context in which this period's work happened. This is really important. All too often, it's easy to forget that while every teeny tiny detail is crystal clear in your mind, that is not the case for your boss. If you met two days ago, they may very well be on the same page as you, but a short reminder won't hurt.  If it's been longer, your boss is interacting with way too many people, they have been gone, they have been distracted, or (as is the case with me) they simply have a sieve-like brain, getting them focused on your details will let you make the most of a a meeting.

Now you can shift into "the problem." What specific thing were you trying to accomplish? It may have been an easy week: you had a straight forward implementation challenge; it may have been a difficult week: you were tackling a completely wide open, open-ended task. In either case, the problem/solution part of your meeting is where you can engage others in helping you, so think in advance of any areas in which you might want assistance. In explaining what you were working on during this period, be as specific as possible. You might use data you gathered to describe the problem in more detail. For example, "It was unclear where the next bottleneck was, so I spent some time profiling the server." This is beginning to blur the line between problem and solution, but I think that's actually just fine in this context.

So, the problem is really what you started out the day/week/month intending to do and the solution part consists of all the things you did to address that problem. Again, be specific, show data. If you spent the time reading papers, organize your reading into a structure, so you can say something more meaningful than, "I read a bunch of papers."  Instead, start with why you were reading papers (problem), "I did not feel that I had sufficient background in different ways to improve server performance, so I looked for papers describing similar tuning efforts. I looked in the following conferences for papers, and I found N papers that seemed appropriate."  That is what your problem phase looks like -- in your solution phase, report what you learned. "After reading them, I realize that performance tuning breaks down into three basic classes .... the first two don't apply here, so I skimmed those papers for techniques, but focused primarily on the third class, which was closest to our work. I identified the following three things that might be good next steps."

I picked this example, because I find that many times, students do need to spend time reading, but there is not necessarily a clear goal or perspective from which the reading happens. The more you can focus your reading on trying to fill a specific gap, learn something specific, or build background in a particular area, the more value you will get out of that reading. However, you can apply that same process to your weekly work.

Once you've presented what you've actually been working on you should find yourself in one of two positions. You might know exactly what needs to happen next and you lay that out as your final "Next Up" (happily ever after).  Or, you could use some help.  Then you invite discussion and brainstorming around your problem. Continuing our example, "The profiling didn't seem to indicate any one area where all our time is going, so I'm not really sure what to try next. I was hoping we might brainstorm ideas."

So now you have turned your "status update" with your "boss" into an engaged discussion. I intentionally started with this one, as it's likely to be the most detailed (and perhaps the most stress-inducing for some); all the others are simply subsets of the 1:1 update, so let's move on to them.

The Project Group Meeting

Your assumption here is that everyone knows the project you are working on, but may not know all the intricate details of the specific thing that you are doing. So, your context starts pretty much at sentence number two of the 1:1 context setting part: "Recall that when last we met (or at the last project update), I had been working on <something> and had gotten to <some particular point>." Assuming that these meetings are happening relatively frequently, that's probably all the context you need.  You can move right into problem/solution.

The problem/solution part of your status can be pretty much identical to what you might do in the 1:1 meeting. And it could lead to the, "Here is what I have to do next," or you might say, "I could use some help brainstorming next steps. Perhaps after all the status updates, I could get a few minutes from the team to do this?" If your project meetings are jam-packed, then ask who might be willing to meet at a different time to do this. I fear that students often feel that they must solve every single problem all by themselves, and that's not how research (or development) works.  It's a team sport. It is absolutely OK to ask for help; it just means that when others ask for help, you should try to step up and help. (It's OK to say, "You know I don't have much experience in this area, but I'm happy to do my best to help!") 

The Large Group Meeting

For this one, think of Systopia Standup. Before going into the specifics, I want to take a moment and explain why we have a large-group standup. When I joined UBC, after the students got to know me a bit better, the single most frequent complaint I heard was, "We have no intellectual cohesion. No one knows what anyone else is working on." TLDR: Our large-group standup was invented as a solution to this problem.  But let's ask the question that underlies that complaint: Why is it important to know what others are working on?  For some, the answer may be obvious, but for others it might not be, so I want to be explicit about this.

Why have large group status updates where hardly anyone really understands what I'm doing?

1. The people in the lab form the beginning of your professional social network. You may have absolutely nothing in common with them outside of the lab, but simply by virtue of being members of the same lab, you have a common experience. These people will show up over and over again throughout your professional career, and you never know when you might suddenly find yourself wanting their expertise, assistance, or advice. You don't have to become best friends, but you should take advantage of this opportunity to forge professional relationships with your lab mates. (You never know, 25 years later, you might find that you are their colleague, e.g., Will Evans and I were at UC Berkeley at the same time, and although we were not in the same research group [as I'm sure should be apparent], we knew each other well enough that when I intereviewed here, I felt that I had a colleague here that I knew].
2. Great research rarely happens in silos. Elegant solutions to real problems frequently require expertise in many areas or at least different perspectives from within the same area. If you have some understanding of what your peers are working on, you can figure out who to reach out to when you encounter an obstacle. Help need not only come from your own project participants or only from students with your advisor. Here are some examples of ways in which my students have gained explicit benefit from knowing what's happening around the lab: Ali and I were stumped on a proof for a paper on weighted GOSDT. One of us mentioned it in standup and the next thing you know, Mathias and Ali had come up with a lovely probabilistic bound on the problem we were trying to solve. Zainab is getting direct end-user feedback on her system. It's not a formal user study; it's not even a pilot. It's obvious to us that Paul is the right person to ask about this. (And I know that Nichole has also had similar experiences.)
3. Our field is HUGE. You can't possibly know everything happening in it or even all the techniques people in different areas use. Large group status updates give you a glimpse of what's happening other areas. These are an incredible learning opportunity.
4. Some day you are going to either A) be on the job market or B) be talking to someone who is looking to hire someone. The best thing that labmates can do for one another is promote each other. If you are in category A, would you not love it if your peers introduced you to their former managers who might be looking for someone just like you? Think how impressed your colleague in situation B will be when you say, "Oh, one of my labmates would be perfect for this." If you don't know what others in your lab are doing (and they don't know what you are doing), none of this can happen.
5. And speaking of that job market. I am 100% convinced that one of the reasons I was as successful as I was on the new prof job market (oh so long ago) is that I could actually talk to people in fields other than my own. I knew enough about what folks in my lab were doing and some of the big ideas others in the department were working on that I could ask questions of everyone, from theoretician to computer architect. You want to be that candidate, and you can't be that candidate by staying firmly in a silo and ignoring all that is happening around you.

End Digression

OK, so let's say you now understand why large group status updates are useful. How do you do a good one? Tell a very short fairytale (a nice short story is way better than and endless list of tiny things you accomplished).

Context: Recall that I work on the Awesome Server Project.

Problem: This week I was trying to figure out how to improve throughput on the server. We are handling X requests/per second and the state of the art systems seem to be able to handle 2X requests/second.  We don't necessarily have to match that performance, since we do This Awesome Thing. But a 100% performance penalty will never be considered worthwhile, so we need to get performance to within at least 10% of state of the art.

Solution: This week I tried removing every other line of code. Good news: Performance was great. Bad news: The server no longer works.

Up Next: I think I'm going to be a bit more strategic in what I remove; I'm going to do some dead code elimination to see if I can improve the iCache hit rate, since I took some measurements and I'm getting truly awful results for that.

It's just another fairy-tale, but one focused on your activities for the week, not your overall project, not your dissertation, just the thing that has been keeping you up at night this week.

Wrapping Up

Here is a tiny table to remind you how to think about the four parts of the fairytale in different settings.

	The 1:1	Project Update	Group Update
Context	Remind, Recall, Focus	Recall	Recall (Big Picture)
Problem	Specific (maybe with data)	Specific	Importance
Solution	Specific (include data)	Specific (include data)	Overview
Next Up	Brainstorm or Plan	Specific Plan	High-level Plan

How to Present your Research: Part 2: Visual Materials

How to Present your Research: Part 2: Visual Materials

(I wrote a related post a long time ago about use of visual materials in teaching. It is related to what's discussed here, but is focused specifically on presentations in the classroom. At the same time, I would argue that whenever you are presenting your work, you are, in fact, teaching.)

About a millisecond after you are asked to give a talk, I'd be willing to bet your attention turns to slide preparation. While I too, almost always, use slides when I give a talk, I do want to point out that there are fields where this it not common. There are fields where talks are people (literally) reading their written work out loud and fields where people give talks without any kind of visual aids.  If you're reading this, I am guessing that you might find this anywhere from surprising to shocking.

Once you have gotten over the shock, however, perhaps we should ask ourselves why we use visual aids. I'm completely serious -- why do we feel the need to have slides to accompany our words?  There are a few reasons:

1. To help illustrate complex concepts.
2. To remind yourself what you want to talk about.
3. Because it's expected.
4. To leave behind an artifact that summarizes your work.

I hope you won't be shocked if I opine that 1 and 2 are good reasons, and that 3&4 are not good reasons. (The artifact you leave behind is, in fact, your paper!  Or, today, perhaps you've been asked to produce a short, e.g., 2-minute, video overview.  This is a much better advertisement for your work than a slide deck.)

If you believe that you produce slides purely for #1, let me ask the following: if you are merely trying to illustrate complex concepts, why do you have slides for every part of your talk (including the introduction and thank you)?  Surely those are not complex concepts?  In other areas, you might see talks that are only occasionally punctuated by images or visuals, which really do illustrate a concept -- they show a picture of a novel experimental apparatus or some results or some particularly complicated concept.  However, there are not necessarily slides to accompany every single part of the talk. In our field, that is not the norm.

So, I ask again, what is the purpose of the slides?

I believe that most of the time, whether we admit it or not, slides are mostly to help us remember what we want to say or talk about as well as to remember how we decided to talk about it. Once you embrace this concept, you buy yourself a great deal of freedom in what you place on your slides. Your slides no longer need to parrot your talk. Instead, they can enhance and embellish your talk. Almost ten years ago, I decided to switch from a fairly conventional slide style where words were primary and images secondary to a style with as few words as possible. Two things happened: First, it started taking me MUCH longer to prepare slides for a talk, because it took forever for me to find exactly the right image to convey what I was trying to convey. Second, I began writing much more extensive speaker notes, frequently, because I like strategic animations, and I want to make sure I use them at precisely the right point in the voiceover. (Sadly, I almost never look at my notes while giving a talk; I ad lib almost completely. Thus, I need to go over my slides enough that I know where I'm supposed to click and hope I get it right. This usually works well the first time I give a talk, because everything is fresh in my brain; when I pull out a deck a year later and "tweak" it and then give the talk, I don't do nearly as well.)

Enough rambling about me, what advice do I have about preparing visual materials to accompany a talk?

1. Do keep in mind that the purpose of the visual materials is to accompany the talk, not replace it.

<flame>

One of my greatest frustrations as a teacher is when students believe that the slides are a replacement for lecture. I do not intend them to be. I tell students this. Yet, time and time again, students who cannot be bothered to attend class complain that the slides sometimes are incomplete. That is explicitly by design so that in class we can discuss and probe things!

</flame>

2. If the slides merely repeat what you are going to say, why should people attend your talk?

3. Think hard about how can you use a visual aid to:

A) Burn an image in your audience's brain that will immediately conjure up you and your work (i.e., make your work memorable).

B) Make something abstract concrete

C) Simplify a complex concept

D) Connect multiple concepts 

4. Use slide real estate to communicate with the audience; use notes to communicate with yourself (although it's good of the slides themselves trigger you to remember most of what you want to talk about).

5. Remember that you want the audience, for the most part, listening to you, not frantically trying to read paragraphs of prose on slides.    

How to present your (Systems) Research: Oral Presentations (Part 1)

It has been a long time since I wrote anything for my blog. I have a bunch of half written things, but let me see if I can actually write a complete post!  Well, not a complete post - how about the "first in a series" post?  I'm running a workshop this summer and it's more importanbt that I get little pieces of this out than I finish it all at once, so welcome to:

Part 1: Introduction and The Fairy Tale

It took me a long time as a professor to realize that I spent a great deal of my time working with students teaching them to communicate. Most of our students enter graduate school with excellent technical skills, and most have learned how to teach themselves additional technical skills. However, they have not necessarily been taught A) how to find a good research question worth pursuing or B) how to communicate what they have done effectively. Today's post is about the latter.

I've already written about writing (i.e., how to write a dissertation), so let's talk about how to present orally. There are two different dimensions that we can use to describe presentations: by type and by content. We'll do both. Let's start with the different types of presentation that largely differ in length, which leads to a corresponding difference in the level of detail that can be presented.

1. The Elevator Pitch: this is what you can tell someone in an elevator ride when they ask, "So, what are you working on?" It is also the 1-slide presentation you might give as part of an introduction. In Margo-speak, this is the fairy tale (an idea that will be familiar to anyone who has taken Harvard CS261 or UBC CPSC 508 or to anyone with whom I've written a paper); I'll get to that in just a bit.
2. The Status Update: This is typically presented to someone who knows what you are doing and your goal is to add to their understanding of the project by sharing not simply what you did recently, but what you've learned, how that has advanced the project, and how that leads to next steps. In other words, how did the things you've done in the current time period move the project forward, and what obstacles have you encountered.
3. The Work-In-Progress Talk (WIP): These are perhaps the most difficult. They are approximately 5-minute talks designed to summarize what you are working on now. Many conferences have explicit work-in-progress sessions. However, for years, the majority of NeurIPs presentations were 5-minute Spotlight talks, which were supposed to tell the whole story of a research project. For the most part, you best you can do here is convince someone to read your paper; it's difficult to relay too much technical content in five minutes when you first have to lay out the problem space.
4. The Conference Presentation: This is usually about a 20-minute talk (at least in my field). You get to describe the problem, present your solution, and demonstrate how well your solution works.
5. The Full Talk: This is typically a 45-60 minute talk (although even if you are told 60, I recommend 45 leaving plenty of time for questions). In many ways, this is the easiest talk to give, because you have the luxury of time, and you can tell the whole story.

Let's now disect a talk into its constituent elements (not all of which can be crammed into every talk).

1. The story: I will refer to this as the fairy tale, and I'll talk about that below.
2. An outline: Depending on the length of the talk, you often need to provide some structure, so the listener knows what's coming and how you are going to tell your story.
3. Motivation: What is the problem you are solving and why is it important? You might think it's important, but that doesn't mean your audience will care about it. Your job is to convince the audience that they really care. If you fail to convince them of that, it's going to be quite difficult to keep them engaged, even with the best work.
4. Your work: So, what exactly did you do?
5. The evaluation: How well did your solution work? Ideally you show both the strengths and weaknesses of your approach (although the latter is far too frequently swept under the rug)
6. Related work: How does your work fit into the landscape of existing work?
7. Conclusions: A good conclusion not only repeats what you've said already, but draws some larger takeaways and implications of your work.

If you have read my blog post on writing a dissertation, these parts should all appear familiar.

Now we can examine which elements go into which talks.

 

	Elevator Pitch	Status Update	WIP	Conference	Full Talk
Fairy Tale	✅	✅	✅	✅	✅
Outline	❌	❌	❌	✅	✅
Motivation	✅	✅	✅	✅	✅
Your Work	❌	✅	✅	✅	✅
Evaluation	❌	❌	✅	✅	✅
Related Work	❌	❌	❌	✅	✅
Conclusions	❌	✅	✅	✅	✅

The astute reader will notice that the fairy tale appears in all talks, so let's start there. Humans love narratives. They like stories. They do not like being spoken at -- they want to be engaged and drawn into a compelling narrative. Therefore, if you want to truly engage your audience, you need to tell them a story. I claim that the fairy tale is the ultimate story and is particularly well matched to presenting research.

The Fairy Tale

A fairy tale follows a predictable pattern. It begins with, "Once upon a time." This is context setting. For research, what is the relevant state of the world (technology) from which your work begins? For example, "Computation is moving into the cloud," for work on cloud computing or data center operation. "Eighty-five percent of the world's population own cell phones," for work on mobility or phone apps. "DRAM accounts for 30% of the cost of most data centers today," for work on far memory. "Data centers alone account for 2% of all electricity consumption in the world, and this number is predicted to grow to 8% by 2030." for work on energy efficiency. Notice that this also serves as a teeny bit of motivation - it tells the listener both how to place your work in context but also why your work matters.  Contrast that opening with, "We present a better algorithm for doing X."

Soon after the context setting, a fairy tale introduces us to the villain to be vanquished. This corresponds to the problem we are trying to solve. Try to be precise -- poor performance is a weak villain; making people wait or wasting energy or using resources inefficiently so things are expensive are more compelling than, "We make something X% faster."

And for every good villain, we need a hero -- that's your solution!  How does your solution address the problem you just introduced?  In your initial fairy tale, you need not go into great detail here, just enough to give the listener a flavor of what you've done.

Finally, fairy tales end with a, "happily ever after." What good will come from your work? How does your work make the world better? How does it change that context in which your work took place? Why should the listener care about the work you've done.

So, the minimum story of your work is simply four sentences: the "once upon a time," the villain, the hero, and the "happily ever after."  It's really that simple -- you can write an abstract for pretty much any paper in 4-8 sentences if you can identify these four key points.  Try it!

For a well written paper, you can read the abstract and extract exactly these four elements. For a badly written paper, you will have to wade through at least the entire introduction (and maybe more) before you can extract these four ideas.

So, the key take away for this post is: master the elevator pitch by having a crisp, clear, and easy to understand statement of the four parts of the fairy tale.

Running a Lab using Discord

Graduate school is a challenging and sometimes isolating experience in the best of times; during COVID, I worry that it's particularly so. As a professor responsible for both my own students and the greater systems student community, I worried a lot about the impact that COVID was going to have on our students. 

From March to May, like half of the rest of the world, we adopted Zoom for group and project meetings, with each group having its own meeting. Unfortunately, since these were all on my account, that didn't help too much, because if I'm in a meeting with Project A, the Project B folks can't use the Zoom room for their project. This was definitely suboptimal.

We also used Slack, but I have something like ten different slack channels and I found that keeping up with it (and just remembering to look at it) was a challenge. 

At the end of April I was co-chairing a conference (Eurosys) and after consulting the coolest people I know (my kids), I decided to try using Discord to simulate a hallway track. We introduced it late and not a ton of folks used it, but the few who did convinced me that there was enormous potential, so shortly after, I switched the lab to Discord (since the other faculty member was on sabbatical, I felt like I could make the decision somewhat unilaterally).

We've been using it now for about three months, and I'm pretty happy with it. I did a survey after a month or two and also got (anonymous) student feedback. So I'll try to both report on why I like it and try to summarize the student's response.

How we set up the server: 

Since this was an experiment, I kept things relatively locked down. Faculty have admin powers and everyone else can participate fully, but not really administer the server. (It wasn't that I didn't trust people, I just wanted to run a bit of a more controlled experiment.) 

We created a schedule channel where we could keep track of weekly meetings.

We then created pairs of text and voice channels for each project group.

As we started to arrange group social events we created a text-base social-event channel and a voice-based Watercooler, which gets used for grad students to hold Office Hours for the undergrads (an initiative completely undertaken by the students).

Some conversation on the general channel resulted in our creating a channel on technology, law, and government regulation.

We created channels on which to have follow up discussion from our weekly reading and writing groups. But mostly, we tried to keep the number of channels down to a dull roar. (I'm guessing that when new faculty start and return from sabbatical, we might introduce some hierarchy around the projects, just to keep the list of channels more compact.)

My reactions

1. It's an integrated "place" where I can engage in both asynchronous text-based conversations and synchronous meetings/conversations. I think it would be nice if a voice channels could have text components, so we didn't have to pair two separate channels (i.e., I'd prefer something that looked more like Skype with its message channels). But the integration was important for me -- I could keep one "thing" open and watch that thing.

2. There is visibility into what the different projects are doing -- if I see that my Project A folks are meeting a lot, that could either be a great thing or it could be a sign that something is up -- it lets me ask.

3. There is also social visibility -- if you see people hanging out at the Watercooler, you can join in, just as you might do in real life (and if people want to have a private conversation, you can do that via direct messaging).

4. I really don't like the fact that in order to construct a group direct message conversation, I have to be friends with the people; it's not sufficient that we're just on the same server. This is a pain and is probably my #1 feature request.

5. I go back and forth about whether I like the fact that direct conversations are globally scoped -- that is, if I have a conversation with Arju, that conversation is not part of any particular server (organization), it's just a conversation with them. In contrast, on Slack, I can have a conversation with Arju as a TA for my course or as a member of my lab and those are entirely separate. Initially I preferred the Slack model, but I may be coming round.

6. A few folks seem to have more difficulty with video calls on Discord than they do on Zoom. I don't quite understand why -- I find that all seven thousand of the different communication platforms I use sometimes get confused about who owns my microphone and speakers and which one I'm using, but it doesn't seem to be Discord specific.

7. Changing between Discord channels is WAY better than moving among Zoom breakout rooms (I'm pondering use of Discord in class).

8. High order bit: to me, our lab server genuinely feels like a place and community to me. It feels like it does a moderately good job of creating an atmosphere for our lab. The combination of Discord and our overall lab culture has allowed 'visiting students' to feel welcome and part of the community and to develop a sense of our culture. This may be possible with other platforms as well, but it seems that Discord has worked particularly well for that.

General student feedback

Slightly over 70% of the students reported that Discord was either fabulous or good (split evenly).

17% were neutral

13% were negative (split evenly between "don't particularly like it" and "really don't like it").

Creating a positive lab culture

Although I attended my very first NeurIPS this year, I missed the keynote by the incredible Celeste Kidd. Fortunately for me, she shared her closing remarks, which I encourage everyone to read. She says important things far more eloquently than I and raises issues that frequently go unspoken.

Inspired by her words, I want to talk about the proactive, constructive part of the story that we as professors, advisors, and/or managers can do to create a healthy culture in our labs and offices. Much of what I share here I have learned from my amazing colleague, Radhika Nagpal. She too has deep insight and material on creating a positive lab culture.

Let's start with the obvious. Lab culture is important; it's what we live and breathe every day. When I moved to UBC, I decided to explicitly write down what I hoped to achieve in terms of lab culture. My website says,

Lab Culture People do their best work when they are both challenged and happy. Research provides plenty of intellectual challenge; our lab culture is designed to encourage happiness.

We are open and collaborative We value diversity in all dimensions, welcoming students and collaborators of all genders, orientations, races, and religions. We collaborate within the lab and with colleagues outside the lab. Today, we have projects with colleagues from theory and engineering, statistics and business, Harvard and Duke, the UK and New Zealand, colleges and high schools.

We are positive Feedback is critical to the research enterprise. However, feedback can take many forms – we welcome constructive criticism. We criticize ideas, not people. We criticize with respect. We are open to others’ perspectives. We do not always agree, but when we disagree, we do so collegially and respectfully.

We are supportive Each of us shares in the mission to enable every member of our lab to achieve their greatest potential, realizing that each person’s definition of success is highly personal. We support each other in realizing our individual images of success. (Radhika Nagpal’s awesome talk articulates this eloquently.) We do not tolerate personal attacks and discrimination of any kind.

The part in italics comes directly from Radhika.

Here I'd like to talk about concrete steps I've taken to try to create that culture.

Day 1 -- My first face to face meeting with a student

For the past five years or so (it took me a long while to figure this out), I open my first conversation with students with the following.

Welcome!  We are so glad you are here. You were admitted, because we believe that you have what it takes to be fabulously successful in our program. That said, graduate school can be hard and there will be times when things aren't going well. It may be your research; it may be something personal; you may be unwell. No matter what it is, do not hide. You are welcome to come in here any time and say, "I didn't get anything done this week."  Just do not hide. Come to class; come to lab; come to our meetings. Together, we will get through this.

I like to think that this is the first step in establishing culture. It says that the students are not alone and that my job is to help them get through this big, scary ordeal they are starting. It is my starting point at building a trusting relationship.

Lab 1 -- our first lab meeting of the year

We began the year by welcoming back all our continuing students, welcoming our new students, and welcoming our visiting students. We then played the game where each person wrote down, on a sticky note, something about themselves that they thought others would not know and that would make them unique. I then read each one and the group tried to match the statements to people. This turned out to be a lot of fun. We were terrible at guessing!  You have to make up rules about people telling the truth (i.e., if I say, “This must be you Juanita!” and it is, you have to fess up).  We tabled the harder ones and came back to them.  I think I would probably use some physical sorting during this to keep people moving and to separate those “still in play.”

Next we spent the thirty or forty-five minutes talking about what an inclusive environment looks like and what behaviors make an environment not feel inclusive.

I shared memories of my first days and weeks at UBC and how students and faculty alike went out of their way to make me feel welcome and included. I also shared stories of a conference I'd attended recently where I was invited to one of their VIP events and felt incredibly unwelcome and most definitely not included. Other students shared their experiences.

We have  a diverse international group and better gender balance than many computer science and especially systems groups, so we heard many perspectives. People were thoughtful and respectful.

Lab 2 -- one week later

I had given people homework to take any two Implicit Association Tests. I also shared my favorite quote on the topic, "Being biased doesn't make you a bad person; it makes you human. Being unaware of your biases and/or being unwilling to work at mitigating them is what makes you a bad person."

I admitted that I always test biased against women in science. I asked how many people found their results uncomfortable. Then I talked about the specific things I do to try to compensate for my biases, e.g., blind grading, the conversations I have with myself before meeting a new student.

Then we played a two-minute video from here.

Next we played The Tag Game. This did not work as well as I had hoped. I labeled the tags with numbers of stickers of different sizes, shapes, and colors and then asked people to form groups. They were all kind of lazy and just formed groups as a function of where they were standing instead of looking at the name tags at all (but maybe this was a sign that my group didn't need this game?).

The basic idea is that people focus on finding people with similar shapes and/or colors on their nametag and when you see the patterns, you can start talking about how you were drawn to finding others like yourself and what this means in other contexts.  So, even though the game didn't work, we had the conversation anyway!

Then I  talked about different kinds of bias and let the conversation just go from there — fabulous interaction! Some of the biases that came up:

• Gender bias
• Beauty Bias
• Affinity Bias
• Conformity bias
• Halo bias
• Horn bias

Lab 3 -- one week later

I came in with a stack of sticky notes on which I'd written one-sentence scenarios, all of which were based on genuine interactions that I'd had with students. These scenarios ranged from health problems to family problems to stress reactions -- some were pretty intense. And I warned people that some of the topics might make them uncomfortable.

The students paired up and each student took a sticky note. Each student got a chance to both be a student dealing with the issue on the sticky and a student to whom someone was coming for advice. Here were the directions:

Each person is going to grab a sticky and then with a partner, you'll take turns being a student who is experiencing some difficulty or a friend that is being asked for "help". You'll get about 5 minutes in that role and then you'll switch; we'll do this with about 3 different partners.

Every one of these scenarios is based on a real conversation I've had with students (or former students).

As the listener: Your first task is to figure out what the person wants:

• to vent
• advice
• help
• something else

Many of these are problems you cannot "fix" -- this is sometimes frustrating and uncomfortable as an engineer. Figure out what your partner wants/needs and what you can do (sometimes it's just getting them to talk with someone else). As the person in difficulty, think about:

• How do you ask for help?
• How scary is it to ask for help?
• What do you want (see above)

After each person had had a few different partners, we had a group discussion. It went really well. Students acknowledged just how hard it was. A colleague said that they'd found themselves using words that they weren't used to using in conversations with students -- FEELING words. Some people were confronted with situations to which they couldn't relate, e.g., I'm male and my partner was dealing with a pregnancy. Even so, we talked about how to be a supportive colleague even when you can't empathize.

I expect that we'll do a similar sort of orientation/culture-building exercises this year, but we will talk more explicitly about racism. We'll talk a lot about mental health -- COVID-19 has changed our lives in myriad ways, and I worry more than ever about students feeling isolated. I moved my lab to Discord about a month ago, and it seems to have been a good move. I spend a lot of time on Discord, Skype, Zoom and make sure that every one of my students (almost) has some structured time to talk with me every day -- it might be short; it might be in the context of our reading group, but for the students who want it, they can have daily human interaction. At the same time, many of these meetings are optional, so the introverts don't need to feel overwhelmed.

The Long Kiss Goodbye

It was the beginning of the end -- my daughter's high school graduation, my last Harvard commencement as a faculty member, and it was my 35^th reunion. Unsurprisingly, it was a jumble of emotion.

I was repeatedly asked "how it felt" to march in my last commencement ... but wait -- I get to come back for reunions and march as an alum, so perhaps not my last, but yeah, it was a special one. I was struck by two things in particular: the 25^th reunion class were my students!? How could that possibly be? I found it significantly more shocking that I had 25^th reunion students than that I was celebrating my 35^th reunion. And then there was the national anthem. As I stood listening, I realized that quite possibly, the next national anthem I hear at a Commencement will be O Canada. Sobering. (I spent the afternoon learning the words, in English; the French words will come later.) [My daughter's high school graduation preceded Harvard's and I don't think they played the national anthem, anyway.]

A jumble.

The Harvard Crimson and I have not always been BFFs, but of late, we've had a good relationship. Last year, the commencement issue included a profile of my colleague, former Dean of Harvard College, Harry Lewis. It seemed fitting -- Harry is the heart and soul of computer science at Harvard; he was the Dean of the College; of course they would do a profile piece about him after he'd announced his upcoming retirement (they also did a wonderful piece about him and his wife, Marlyn McGrath Lewis, this year). But it came as a bit of a shock when managing editor, Hannah Natanson, asked if they could do a profile of me for the Commencement issue. I'm sure there is no bias when I say it's the best thing that I've seen the Crimson do. I had a blast working with the reporters, who managed to unearth an impressive array of characters from my past (not all of whom appeared in the final copy, but to whom I am still extraordinarily grateful). I was happy to see that my habit of speaking up was spun in a positive light and that it was referenced by most who wrote; it is, perhaps, the most important thing I've done at Harvard. But most of all,  I am deeply humbled by the piece they produced and the kind words that my friends and colleagues offered. (And while I normally hate watching and listening to myself in video, the video piece, which I missed the first several times, was lovely.)

A jumble.

Our memorial service -- 8:30 AM on Friday morning. We all showed up. In a class of roughly 1600, we've lost 57 classmates. Some were my friends: bandie Larry Millet, fellow Computer Scientists David Brownell, my twin (we share a birthday) and inspirational mathematician, Matthew Bovell. But unsurprisingly, my thoughts turned to Alex, and I found myself silenced and in tears during the final verse of Swing Low Sweet Chariot. Alex should have been graduating. I heard from both his parents in the days leading up to graduation. My heart continues to break for them every time I think of him.

We lost another member of the class of 2018, Luke Tang, during the fall of 2015. While I did not know Luke, I remember that a group of his friends were taking CS61 from me that semester. I still remember where they used to sit in the room and how I wanted to, somehow, be able to ease their pain in the days and weeks after we lost Luke. Until I started writing this blog, I didn't realize the light that came out of this darkness. I've not yet seen the documentary, but I want to. And I want to know why every faculty member hasn't seen this; why it's not discussed; why we continue to keep these stories in the dark.

A jumble.

The SEAS Commencement Reception -- a chance to meet the families of some of my wonderful students. Lots of proud families and many wonderful pictures with my students, my colleagues, their families.

Reunion receptions. Classmates I haven't seen since Freshman year; classmates I've never met; classmates I see regularly. It's all a blur.

A commencement address to the students completing their Master's degrees in Information Technology fields. At the encouragement of the students in my online business analytics program (HBAP) with HBS, I talk about courage, perseverance, and gender equality. I later give that talk to the HBAP cohort at their immersion.

Reunion panels and talks -- the raw, honesty brings us together in a way that four years at Harvard never did. It took thirty-five years of life to create class cohesion so palpable that nearly all of us used the same words afterwards to describe what an amazing reunion it had been.

A jumble.

Then commencement and reunion are over, and it's officially summer.  Israel, Michael Rabin. Weekly lunches with my group. Birthday cakes. Many, many coffee, breakfast, and lunch dates with friends and colleagues. The Harvard club of Rochester, an awkward coming home of sorts, but a fabulous group of people. Marlyn and Harry, the two who have been with me since before I was admitted to  Harvard and until now, celebrate their 50th wedding anniversary. Teagan takes me to see Dirty Dancing. Soccer. A going away party with the chucks. DE Shaw Research; MongoDB World. More lunches with my students. More soccer. Usenix annual tech. DC. DARPA deliverables. Late nights and long weekends leading up to DARPA deliverables. More lunches with students. The Eagles. An outing to Kimball Farms with the Oracle crew. Dinners with friends. Soccer. USWNT in Hartford. The quintessential NYC experience: waiting in line for six hours in a torrential downpour for Hamilton tickets; success! ISAT in Wood's Hole. My students begin departing -- to home, to Africa, to NYC. Students slowly return to campus. My last week; Teagan's first. Heat. Humidity. More heat. Boxes. Drinks with colleagues. More boxes. Pictures. Lots and lots of pictures. Pictures from a life time ago.