WWC2023: Italy versus Argentina

Imagine the following: Kerstin and Margo have just come down from the Sky Tower. It's around mid-day. Someone says, "What do you want to do tonight?" The other says, "I wonder who's playing tonight and if there are tickets?" Well, the answers were 1) Italy v Argentina and 2) Yes!  Not only were there tickets, there were "Cat 3" tickets for a whopping $20 NZ dollars (~14 USD or 18 CAD). So we found ourselves high up enough to have a great view of the field situated just past the Italian goal keeper's shoulder (in the first half). These felt better than many of the Cat 2 tickets (which are a whopping $30 NZD).

We had no idea what we were in for, but it was a clear (albeit cool) evening, we wanted to check out the train for transportation to/from the stadium, and well, what else do you do in Auckland on a Monday night? The Argentinian fans were out in full force.

This turned out to be a spectacular match. Italy seemed the dominant team, but Argentina had a very speedy and skillful left wing (or maybe it was left mid???), Estefania Banini, who made for some dramatically exciting play right in front of us. In the opening minute, Argentina's Mariana Larroquette, appeared to be ready to secure a 1-0 lead, but her rocket went just a tad wide.

Perhaps that was the wakeup call needed for Italy to take control. Both teams move the ball beautifully, but Italy definitely had the edge and a significant majority of possession.  And then at the 15 minute mark, Italy puts one in the net, only to have it called offsides (the VAR makes these calls now unquestionable). Nearly thirty minutes later, history repeats itself: Italy takes the ball down the side, finds net, and ... OFFSIDES!  One of these times, the goal is going to have to not be offsides, right? But at the end of the half, it was 0-0.

That match continued gaining momentum and excitement throughout the 2nd half. By the 80 minute mark, it seemed that the teams were destined to fight to a 0-0 draw. But an 83rd minute sub brought veteran striker Cristina Girelli onto the field. And four minutes later, on counterattack, Girelli blasts the ball into the left side of the goal, with not an offsides call anywhere in site. Italy 1-0!

The remaining 10 minute (7 minutes of stoppage time -- VAR adds a lot of stoppage time to games these days) were a frenzied back and forth up and down the field. There were many heart stopping near misses, but neither team found net and so we ended with a breathtaking 1-0 win for Italy. This was definitely one of the most exciting matches of the tournament (and we have watched nearly every game so far, either online or in person).

WWC2023: USA versus Vietnam

It was a beautiful day in Auckland, and all the Team USA fans headed to the top of Mt. Eden, before heading to Eden Park to see the US open up their 2023 World Cup bid.  Although I didn't take this picture until a few days later (from the top of the Sky City tower in Auckland), here is Mt. Eden (the green mound at the top of the picture).

It is the highest Volcano in Auckland proper. While it's not a huge hike, it does feel like work to get to the top. The grassy bowl is oh-so-tempting, and many years ago, you used to be able to run down in it, but they don't allow that any more. In addition to attracting ALL the US soccer fans, the hike was a favorite among the local dog population who were scampering all of the rest of the park.

The top of the park gives a lovely view of Auckland and its environs. So, before getting to the game, let's play the Auckland quiz!

1. Spot the Volcano parks

2. Can you find the stadium?

OK -- now, the game! We were seated in the corner immediately between the US first-half goal and the US bench. The stadium was packed (although not quite as packed as for the New Zealand opener, which was nice). In the opening minute, Vietnam made it quite clear that they were ready to play an aggressive, physical game -- Trinity Rodman went down and stayed down a tad longer than anyone would have liked, but eventually popped up and looked no worse for the wear.

I am pretty sure that everyone in the stands was expecting that the US was going to 'welcome' Vietnam to their first world cup as they did the team from Thailand in 2019 (13-0). However, Vietname was not going ot have any of that. They had an organized and disciplined team that quiet effectively shut down the US. When they did get the ball, they were quick (as in really quick) on the attack, but the duo of Girma and Ertz were having none of that.

The US pushed and created a couple of exciting opportunities, but the first 15 minutes were clearly frustrating the US.  And then about 15 minutes in, Sophia Smith collected a nice pass in the box and deftly sent the ball past the keeper into the box. 1-0 US!

The fans were hoping that this was the beginning of the floodgates, but Vietnam had other ideas. They continued to thwart the US attack throughout the rest of the first half. The US were playing a somewhat 'workman' like game, and they really needed some creativity to breakdown the Vietnamese defense. In the 43rd minute, Rodman went down again and after VAR consultation, the ref awarded a penalty. Morgan stepped up and you could feel the crowd breathe a sigh of relief, just knowing that she was going to make it 2-0. But, like nearly all the other PKs in this first round action, the keeper stopped it!

And so the game continued into extra minutes. With time running out, Smith sends a shot into the keeper; it's not a rocket, but it someone rolls past a few players, including the keeper. The jubilation was, however, short lived, as Morgan was called offsides. But then, the VAR review happens, the ref looks at the play and the offsides is overruled -- USA 2-0!  And it's now halftime.

The US comes out strong and starts peppering the Vietnamese goal, but the Vietnamese keeper will have none of that -- she is brilliant. Absolutely nothing is getting past her. Finally, at about the 60th minute, Rapinoe and Lavelle come in to replace Morgan and DeMelo (WWC debutante and effective wing midfielder). Things definitely start to look better for the US. Lavelle works her magic in the middle and the game looks a lot more interesting and the US a lot more threatening. Sure enough, about fifteen mibuntes later, it's Smith again, but this time it's with an assist to Horan to calmly sends it to the back of the net. USA 3-0!

The rest of the match sees the US with more shots and more brilliant saves from keeper Tran Thi Kim. I think we'll be hearing a lot more about her throughout this tournament.

My assessment: The play reminds me of the beginning of the 2015 WWC, where the US won games, but did not look particularly inspired doing so (compared to say, Germany, who looked fantastic against the over-matched Moroccans). Julie Ertz is a great replacement for Saurbrun in the back, but we really miss her as the holding mid. We need Lavelle healthy to create some life in our midfield. The Netherlands seemed in a similar bind in their opener against Portugal. So, both teams need a shakeup before their match in two days time!  Until then, signing off from Auckland (and headed to Wellington).

Live from Auckland, New Zealand: It's Women's World Cup 2023!

 It's 10:21 PM in Auckland on July 20, and the opening match of the Women's World Cup is in the books. (For some of the games, dates are going to get confusing given that most of you are on the other side of the world, so we just won't worry too much about dates.)

The opening match was absolutely everything one could want in an opening match: an upset, a win for the host nation, a 1-0 game, many a heart-stopping near-goal in the closing minutes, a team's first WWC win, and 99 minutes of outstanding soccer!

The match was scheduled for 7:00 PM, with the opening ceremony starting at 6:30. We were staying at an AirBNB in the heart of Auckland downtown, so we hopped on a bus and got to the stadium in plenty of time for the opening. We were just a few rows away from Mike and Teresa Olson, fellow WWC afficionados. I am accompanied during the group stage by Berkeley Bruiser Kerstin Pfann and WWC-2019 veteran Chloe Lemmel-Hay.  Unfortunately, changes in travel plans resulted in our having 1 seat separated from the other two. Kerstin and I were together in the second row directly across from the edge of the Norway bench. 

The opening was high energy, fun, incorporating song and dance from the Maori (the indigenous people of New Zealand), representatives of all the competing teams, fire works, and young New Zealand and Australia performers Benee and Mallrat.

The opening match featured 26th ranked host New Zealand against 12th ranked Norway. The crowd was unsurprisingly full of New Zealand supporters, but we all knew who the favorites were. However, it took only a few minutes to see that this was not going to be a match that Norway dominated. The first half showed a physical Norway being dominated by incredibly well-orchestrated team play. 

The first half saw several aggressive plays from Norway sent NZ players tumbling. While the fans were convinced there were fouls being committed, the ref had none of that, and I have to confess that I was pretty sure NZ had been robbed on the plays that happened far away. But when the incidents happened right in front of me, I found myself agreeing with the ref, so I'm going to say it was a well-reffed game.

The two teams fought to a 0-0 half time tie.

And then ... just 3 minutes into the second half, New Zealand put together a well orchestrated (and I'm going to guess, well practiced) sequence: starting with a goal kick that a defender took (rarely see that in world cup soccer), two quick passes up the field, a cross and beautiful show from Hannah Wilkinson, put the Ferns up 1-0. For the next 20 minutes or so, it really was New Zealand's show. Norway was having difficult receiving their own passes, their goal kicks were as likely to be collected by the Ferns as their own players, and the Ferns continued to look threatening.

To be fair, there were a couple of heart stopping near-misses from Norway as well. As time wore down, Norway started asserting themselves and began repeatedly threatening the New Zealand back line. A couple of brilliant moves stole the ball away from them just a few yards in front of the goal.  And then, after the 80th minute, play stopped. The VAR called for a penalty check. Time stood still (or so it seemed, that incessant clock kept ticking). And then ... handball in the box!  The replay showed that it was most definitely a hand ball, but having to decide whether it was in the box or not was nontrivial.

NZ stepped up for the PK and ... it hit the crossbar. This near miss was just what Norway needed. They started pressing and attacking and attacking and attacking. I couldn't really count, but I'd say that at least a third of their shots happend in the nine minutes of stoppage time. They hit the crossbar; they forced a pair of goalie saves. But alas, they could not buy a goal, and when the final whistle blew, New Zealand had won their first world cup match ever!

Every four years, the level of play in the tournament improves, and if tonight's match was any indication, it's going to be fantastic tournament!

How to Present your Research: Part 6: Common Presentation Pitfalls to Avoid

In our communication skills group last week, the student presented (for the first time in this group) work-in-progress talks or a 3-5 minute research summary. I try to give everyone detailed feedback, but there are common things that came up, so I figured I'd try to write them down here. This post could really be called "Tips for Presenters."

1. Timing

The assignment was to prepare and deliver 3-5 minute presentation for a generation audience (i.e., not your project team, but a group of smart computer scientists). I then asked each person how long they thought their talk would go before they gave it and then how long they thought it actually went. I timed all the talks.

Unsurprising, practically every presentation went longer than expected. Similarly, everyone knew they'd gone longer than they expected, but they were not super accurate at realizing how long they went. And this is 100% normal on both cases. Our ability to judge how long it is going to take to present material is not particularly good.  The problem is that while a 3 minute talk might run over by a minute and that doesn't seem bad, proportionally, that means your 20 minute talk just became a 26 or 27 minute talk and that's actually a real problem!

Why do we underestimate? (There is likely science on this, but I'm going to draw on my experience here; it would be interesting to compare my experience and assessment with the literature, but I don't have time to do that now, so we'll leave it for a future post.)

I think there are three main reasons: 1) We think more quickly than we speak, so unless we practice the talk out loud, our timing is off, 2) If we practice without an audience, we have no feedback about whether what we're saying is getting through -- when you have an audience, you are getting feedback and that almost always results in your slowing down, 3) We have an inherent fear of running out of things to say, so we over plan.

Just like we read more quickly than we speak, we also think more quickly.  So, if you practice your talk by going through it in your head, your practice will not match your delivery. The solution to this is simple: practice out loud. I hear you, "Oh that's so cringy."  Yes, it really is. Do it anyway.

Similarly, do a real practice talk in front of real people. You clearly are not going to do this for every status report, but when you are presenting for real, do a practice talk. This will be the best way to see how long your talk is going and to check whether you and your material are in sync and conveying the key things you want to convey. And then ask for and gracefully accept constructive feedback. You then get to be judicious in how you incorporate that feedback, but while it's being given, I encourage you to follow the Gotham Booth approach and simply listen, take notes, and ask clarifying questions, e.g., "Was it the graph you found confusing or how I described it?"

Now, how do you conquer that internal dialog that goes something like this, "You know, there is nothing particularly deep here. You're going to be done in two minutes and everyone will think that what you did is simply no big deal." How did I know that's what you were thinking?  Because we all go through that dialog to differing degrees. My advice is to keep the presentation itself streamlined -- explain things clearly and in ways that connect. When you are tempted to off on a digression about some fascinating detail, make a set of backup slides. I don't know how long powerpoint has had "sections," but now that they do, sections are a great way to organize various sets of backup slides. You have them in case someone asks the question (and won't you look so awesomely prescient when you pull them up!), but you can avoid embedding the digressions and extra detail in the talk.  And, the sheer existence of the backup slides should help qualm that annoying little demon who tries to convince you that you have nothing to say.

2. Matching Delivery Rate to Absorption Rate

For longer than I care to admit, I could not exactly figure out why I was uncomfortable with one of my former students who always wrote out their talks completely in their speaker notes. I too sometimes write out what I'm going to say, but I absolutely never actually read what I write while giving the talk, so why did it bother me so much when this student wrote out what they wanted to say? It took me a couple of practice talks to figure it out.

If you read your talk (and aren't excruciatingly careful), you are able to deliver your talk at a rate that is faster than the rate at which your audience can absorb it.  This sounds obvious in retrospect, but only in retrospect. When you are simply reading the words you want to say, you aren't pausing and stopping to allow your audience to keep up with you. When you do not have a written script (and haven't memorized your talk, which is equally problematic), then you tend to pause at the right places, and since you have to think a bit about what you are going to say next, you give your audience time to internalize what you've just said.

Therefore, I advise writing notes/bullets to yourself in your speaker slides (I find numbered points particularly helpful and try to avoid having more than about 3 points I want to make on any slide; a point can require multiple sentences). Think of these are reminders of the topics to cover, but not the exact words, so that you have to engage in a thinking process (not a reading process) when you speak.

If you really really really don't want to have to select words on the fly (and I completely understand many reasons why you might not), then you should 1) repeat the words, "Slow Down" in your brain at the beginning of absolutely every slide and 2) insert <pause> in the right places in your talk,  where you need to stop and give listeners a chance to catch up. I also insert <click> when I want to sequence through animation on a slide.

Pacing your words to keep your listeners engaged is perhaps that most challenging part of giving a talk, and it requires a lot of practice!

3. Buzzword Bingo (this is really about clarity)

There are few things as frustrating as being in a talk and having the speaker toss out terms and acronyms that you don't understand. (My department is terrible at this and when I first joined, I spent every faculty meeting raising my hand and asking what every acronym meant.)

So, nearly all acronyms should be explaind on first use (you can assume everyone knows what a CPU is; you cannot assume that everyone knows what a BANANA -- yes that's an acronym, but not in CS: Build Absolutely Nothing Anywhere Near Anything). In general, avoid inventing acroynms just to make your life easier (I have had students in the past who would introduce 10-20 acronyms in the introduction just to avoid having to type words out later in the paper; this is not a winning strategy for getting papers accepted). I like to think of writing and reading as a give and take process. Every time, you ask your reader to learn something to read the rest of your paper (i.e., what BANANA means), you are taking. Before you take, you'd better give them something pretty darned special to have them be willing to give you the effor to learn your acronym. After being forced to learn about three new acronyms, I am pretty grumpy reading the rest of the paper.

If you find yourself explaining some term and never using it again; don't use it; just explain what you're doing that one place you're tempted to introduce it.

Assume you are talking to a non-expert and define/explain every term that a non-expert is unlikely to know. Ask yourself, "Did I know what this term meant before working on this project?" If the answer is no, then chance are you need to explain it during your talk.  And, the more time you spend explaining terms, the less time you have to discuss things that are truly interesting.

Take Aways

1. Practice your talk out loud.
2. Give a practice talk (and invite a range of people who can give you feedback from different perspectives).
3. Keep the maintalk streamlined and have backup slides (material).
4. Match your delivery tempo to the absorption rate of your audience. This requires careful and thoughful pauses in your speech.
5. Keep acronyms to a minimum.
6. Define all terms specific to your area that are unlikely to be understood by those not in your field.

How to Present Your Research: Part 5: The 3-5 minute (work-in-progress) talk

Recall that in Part 1 of this series, we laid out a collection of different types of talks and then identified what has to appear in each of those talks. We've been working our way through these and today's topic is the laid out in the third column as the "Work in Progress" (WIP) talk. It may also be a talk that goes along with a poster.

The parts that need to appear in a WIP are: The fairy tale (of course), motivation, your work, evaluation, conclusion. Mostly, I like to think of this as a simple expansion of the fairy tale: to a first approximation you are basically going to have approximately one slide for each part of the fairy tale. In all likelihood, you'll have more than four slides, but you probably won't have 10. So think about it as: 1 "once upon a time", 1-2 introduction of the villain, 1-3 introduction of the hero, and 1-2 for the happily ever after which might include results and/or next steps and will certainly include a reminder of why this work is important.

Lest you think that these talks are simple, throw-away talks that don't really require a lot of effort, I want to highlight two instances of such a short talk with enormous consequences: 1) pre-COVID, the papers that were not quite good enough for Oral Presentations (of which there were maybe 15) were selected as Spotlights (around 50 of these, out of the approximately1500 papers accepted) and were given a 5-minute talk slot. 2) The Bell Labs Prize Phase 2 competition requires that teams summarize their work in a slide deck not to exceed 10 slides. Needless to say, a lot of time/effort goes into these short talks.

The Cover Slide

I can see and hear you now: You are rolling your eyes and thinking (or even saying out loud), "You have got to be kidding me -- she wants to tell us what to put on our cover slide?  And the answer is, "No." I believe that you know what to put on your cover slide (the project Title, your and your collaborators' names) and ideally some logo or graphic or picture. However, I am going to talk about what to say when you begin your talk.

What not to say: Please, please, please, do not simply read your slide to the audience. I have sat through so many talks where the session chair says, "Next we'll hear from person, who is going to talk about title." And then person gets up and says, "I'm person and I'm going to talk about title." And, of course, the entire time, I've been staring at a slide that says, "Title" and has an author list with person's name in bold.

I hope by now you realize that if all you do is read your title and name, you are duplicating information and missing a huge chance to get the audience interested in your work.

If, and only if, A) you have not been introduced, B) the audience does not know who you are, and C) you have co-authors, may you introduce yourself.

Then, rather than read the title to the audience, explain why they should care what you are about to tell them for the next five minutes. This might be the 'once upon a time' of your fairy tale; this might be an explanation of your work that avoids buzz words that might appear in your title; this might be a question, i.e., "Have you ever wondered ..." Whatever it is, give the audience something that they will not get from simply reading your slide.

I'm going to pick four random titles from HotOS (the last event I attended) and offer some cover slide material.

Putting out the hardware dumpster fire -- Some of you probably already heard this, but I'd say something like, "Today's hardware is really complex, and for the most part, when we talk about operating system security, we completely ignore this complexity. As a result, I'm going to claim that we aren't really solving the problem that we think we're solving, and I'm going to offer some suggestions for actually solving the hard problem!"

First, Mothy needs to introduction in our field and second, these few sentences are much more descriptive than the title, which I still believe is entirely wonderful.

Degrading Data to Save the Planet -- We will never have less data than we do today! Every day, we produce far more data than we destroy. More importantly, producing the devices on which we store this data (i.e., flash) produces significant carbon emissions. Let me tell you about our new sustainable storage design.

Doesn't that tell you a bit more about what's to come than the (also great) title did?

Towards (Really) Safe and Fast Confidential IO -- Customers do not trust their cloud providers! Confidential computing is an approach to using cloud resources in a way that does not reveal confidential data to the cloud provider. Unfortunately, today's work in confidential computing mostly ignores information that is revealed through IO APIs. Let's talk about how we can improve on this situation.

Once again, I think this tells you more about what's coming than the (perfectly fine and accurate) title.

Automatic kernel offload using BPF -- I explicitly picked this one, because the title tells you exactly what this paper is about. Even so, I think you can say something here that tells your audience more.

The extended Berkeley packet filter (eBPF) has become the de facto approach for running kernel extensions. But how do you decide what should be offloaded? Offloading the wrong thing can produce terrible performance. Let me tell you about a tool we developed to automatically figure out what you should offload to eBPF.

See -- I bet you did not realize that that's what this paper was about.

So, having now written an almost entire blog post on your cover slide, let's move on.

The Once Upon a Time

Others, less well-versed in fairy-tales, will tell you that this is the motivation. They are partially correct -- the motivation is actually the combinination of the once upon a time and the villain. In a short presentation, the once upon a time may also be short -- you just to have place the work you're going to talk about in the right setting. Is this a paper about data centers? containers? isolation? performance? security? privacy? formal verification?

If you are presenting a short talk, chances are that you are one of many talks in a session and there might be little or no relation between the talk before you and yours. That makes it all the more important to help your audience get in the right mind set for your paper.

Consider the Degrading Data paper above. If the paper before you was on disaggregated memory, the audience is primed for talking about in-memory data; if they paper before you was a storage system paper, they are primed for a storage system talk. You want them primed for your talk.

The Villain

Now that you have your audience primed, you really need to motivate them that your work is important. What is the problem you're solving? Do not assume they agree that the problem you are solving is A) important, B) interesting, C) feasible. You need to convince them that they actually care about your problem.

Sometimes the villain takes the form of a story (i.e., "Imagine that you are ..."). Sometimes it could be a performance problem, "Have you ever done X and then had to sit there and wait until Y?" It could be something the user has never wanted to do, "If you are a networking researcher, packet traces are worth their size in gold!" (So, even if the listener is not a networking researcher, they ought to be able to appreciate that someone who is would find packet traces valuable.)

The more concrete you can make the problem the better. Compare, "Keeping data safe is important," to "Customers lost twelve trillion dollars when this very specific bad thing happened."  That ought to make people sit up and take notice. Even if they are not data geeks or security/privacy experts, the thought of losing twelve trillion dollars is likely to seem important.

The Hero

This is where things get hard. I think that with focus and a bit of thinking, you can really nail the context and the problem, but how do you summarize everything you've done in only 2-3 slides?  You don't! Focus on the big picture and maybe (just maybe) one technical detail.

What do we mean by the big picture? This is not necessarily a detailed block diagram of the 100,000 linse of code that you've written to build some system. Instead, what are the three key ideas that you've had to solve this problem?  How do they fit together?

I've spent a fair bit of time developing short 'research vignettes' that let me describe research in myriad different areas to a general audience. Each of these is really just a short talk.  Here are some examples:

For part of a PLDI keynote, I summarized a student's MSc work (tens of thousands of line of code) in four slides: context, picture that shows what the student produced while my voice over explained why we were trying to do it, picture showing the key decomposition approach that let him solve it, and a slide with one key technical approach used to in solving the problem.

At a recent talk at Duke, I summarized another student's MSc work in 6 slides that used the same primary picture on each slide and then added different pictures on the side to illustrate how he solved the key parts of the problem.

It takes awhile to get good at capturing the essence of the work and explaining things at just the right level of detail. I find it helpful to have a specific audience member in mind. This person is most definitely not someone on your project. They are likely not even working in exactly the same area you are. But they are a computer scientist and they are smart. Think about how you might convince them that they should care about what you're doing and then once you've made them care, figure out how to present your work in a way that helps them "get it" even if they cannot appreciate all the nuance and subtlety.

The Happily Ever After

As mentioned above, the happily ever after must include a reason why it was worthwhile to conduct the work. How have you made the world a better place, even if only a tiny bit? In addition, it can also include, next steps, things you learned, suggestions for future projects, etc. If you can leave the audience wanting to work on your problem, they you have succeeded!

How to Present your Research: Part 4: Presenting Results

We've talked a lot about story-telling so far, and I bet you're all wondering, "OK story telling is fine, but I have some actual research results I want to show!  Let's get to that." And indeed, we will. But, guess what -- each result is a mini story.

A result story has three basic parts: 1) Why am I showing these results? 2) How did I obtain these results? 3) Here are the results.  Let's dive into each one.

Why am I showing this to you?

Each experiment you perform or result you show should be either answering a research question or showing evidence to support/refute an hypothesis. I think these two are really the same thing, so for the rest of this post, we'll call them research questions (RQ for short).  Note that it has become a lot more common in recent publications (even in systems and ML) to explicitly state your research questions. I encourage you all to do this before you write any experiment code -- it's great research and mental discipline.

Before you dive into the details of the experiment, explain what RQ you are answering. This could be no more than a sentence, "We want to know if the overhead of OurGreatIdea is acceptable, so we conducted N experiments to determine if we were able to do SomethingAmazing without introducing more than X% overhead."  Wait -- notice how I snuck that goal in the end? I think that 99% of the research out there asks the question, "How much overhead does something introduce?" And then, regardless of what that overhead is, we declare victory!  That's not exactly science. I urge you to think (before you run any experiment), "How much overhead is acceptable?" Once you do that, then you can decide if your overhead is great, good, acceptable, almost there, or terrible.

But, I digress. Sometimes, you simply have to state the RQ. Sometimes, you might want to go into a bit more detail. Some of the greatest insight comes from ablation studies; these answer the question, "Which of the N things I did account for the great results we get?" Let's say that you built a system that introduced three optimizations. People will wonder both A) How much does each thing matter? B) Do the results compound? In the case of such an ablation study, you might want a slide to review the various things that  you did and then explain that you're going to introduce each one by itself or remove each one and leave the others in. Depending on your system, one of those is likely to make more sense. If you can do both, go for it!

A cousin of the ablation study answers the question, "How much time do we spend in each part of our BigComplexSystem?" Usually, you can just explain that with a single sentence; sometimes you'll want an architectural picture to explain it.

Regardless of the complexity of the experiment, please explicitly tell them what question the result you are about to present is going to answer.

How did I do this?

Explain the experiment!  It sounds easy enough, but you'd be amazed how many times I am shown a graph with 0 explanation of: A) the data used, B) the workload run, C) any details about whether I am seeing averages of many runs, a single run, the best of many runs, etc. So, describe your experiment.

In a paper, you will want to give precise details about the machine on which you're running. In a presentation, I think it's fine to categorize the platform, e.g., "We ran this on my laptop." or "We ran this on a huge server." or "We ran this on a typical server with a Big Honkin' GPU."  You can go into more specifics, but I think just putting the detail on the slide is more helpful than saying the particular model of processor you are using (no one can wrap their heads around that verbally; OK maybe someone can, but I do not think most people can).  Typically people want to know how much main memory you had and if you are running a modern machine. If you are doing storage work, then of course, they want to know something about the storage system too.  If you are doing ML, they may care what GPU you are using. Think carefully about what your audience wants to know and the best way to present it (on the slide, but say nothing; on the slide and highligh key features; on the slide and read it).

Describe the workload!  Tell the audience whether this is a standard benchmark in your field or one you made up (and if you made it up, you are going to have to justify why). Is this a microbenchmark or a macrobenchmark?  Is there input data? Where did that data come from? What exactly are you going to show? The average of N runs? One run? One cherry-picked run? A number you made up?  Sometimes, you might even say, "Here is what we expect a good system to do on this benchmark."  Then you can compare that strawman to your actual results. Alternately you might way, "To the best of our knowledge, the best system for this is X and here is how it does on this benchmark." Again, then you can compare your result.

Your goal is that when you finish describing your experiment, the audience is excited to see your results.

My Results

Tell me what I'm looking at! Again, this seems obvious, but I have sat through too many talks where a speaker excitedly tells me how great these results are before I have any clue what I'm looking at. So, breathe ...

Most results (in systems) are either tables or graphs/charts (I'll use the term graph here to describe a visual depiction of data). In either case, describe what the audience is looking at. For example:

Graphs

This graph has throughput on the Y axis and the number of cores on the X axis (if you are using log scale, say that explicitly here, even if it's in the axis labels which it should be). So, big bars (higher numbers) are better. [Please, always add this last part!]  Sometimes I just show the structure of the graph with no data as I explain this.  That allows the audience to understand what they are going to see before they get distracted with the actual data. The error bars show standard deviations, the middle 50th percentile, whatever -- tell me!

Tables

Most of the same rules apply here. Each row represents something and the columns correspond to something else. Tables of numbers are relatively difficult to comprehend, so I encourage highlighting the things you want the audience to take away from the tables -- embolden the best in each row/column; color code good/bad/mediocr results. Basically, help the audience take away the message you want them to take away from your results.

Takeaways

Draw the listener's attention to the things you want them to get from the figure. Do not assume that your results are so obvious that they will get exactly what you want them to! Inevitably, they will draw a wrong conclusion or one different from the one you wanted them to.  Walk them through the results.  I like to compare and contrast with what we might expect. Some examples:

We expected that we would scale well until we were running one thread on each socket, but we found that we actually scaled well until we were running one thread on each core. As we'll see in the next graph, our cache footprint was small enough that the data fit nicely in the per-core caches. [This is a great technique to explain WHY you get the results you do -- show a macrobenchmark result and then follow it up with a microbenchmark result that explains the results you got in the macrobenchmark.]

We had hoped that participants using our tool would complete the task more quickly. While they did produce better solutions (point to the part of the picture that shows that), we noticed that they actually took longer. [When you have a surprising or unexpected result, this usually indicates something interesting, even if it didn't match your intuition. In this case, perhaps it's something like, "Our qualitative survey results suggest that people found our tool so much more enjoyable to use that they were willing to spend more time using it to produce high quality solutions; in contrast, old tool was so painful to use that participants stopped as soon as they produced anything close to correct."]

In my speaker notes, I almost always have a numbered list of 3 +/- 1 key points that I want the audience to take away from any result I present.  I do not write the details of those points in the notes, because I want the timing of my explanation to match the ability of the audience to comprehend what I'm saying. If one writes out the exact explanation, it is almost always the case that that explanation is presented too quickly for the audience to comprehend. If the speaker has to think about it, then their delivery almost always better matches the audience's receptive timing. I think this point is crucial!  When we are nervous, we almost always start to speak more quickly and often become quieter towards the end of our sentences. You want to do everything possible to avoid this, especially when presenting results.

Key Takeaways:

1. Don't forget any of the three parts: question I'm answering, how I am answering it, what I found.
2. Explain the format of the results you are presenting.
3. Use microbenchmarks to explain the results of macrobenchmarks.
4. Highlight surprising results.

How to Present your Research: Part 3: Status Updates

Normally, we think of presentations in terms of formal talks. However, I am willing to bet that we all spend far more time giving information status updates (which are, in fact, presentations) than we do giving formal talks, yet I do not believe I've ever seen any guidance on how to do that effectively.  How strange!

Well, here goes ...

Surprise: I'm going to argue that every status update is, wait for it, a fairy tale! (This is going to be a recurring theme.) Before diving into the details of that fairy tale, let's talk about different categories of status updates.

The 1:1 meeting with your advisor/supervisor/team lead. At some point, you'll find yourself in a 1:1 with someone we'll call "your boss" and you'll want to update them on what you've been doing. The key thing to help organize your thoughts and presentation for this meeting is to ask yourself, "In how much depth does this person understand what I'm doing?" In some cases, the boss will know exactly what you are doing and could be doing the task themself.  In other cases, the boss has a big picture idea of the project, but is not knee deep in the details. Obviously, you're going to have to present your work differently depending on the situation you are in.

The project meeting. This is typically a small meeting with the people directly working on the same project you are working on. In this case, you can assume that the others in the meeting know the project-level fairy tale, so you are going to focus on the smaller fairy tale describing what you are doing.

The large group meeting. This meeting includes people working on projects different from the one on which you are working. Even if you see these folks every week, you probably cannot assume that they remember from week to week exactly what you are doing, so you need to plan accordingly.

Regardless of which of these meetings you are having, you are still going to present your status in the same structure: 1) some context setting, 2) a discussion of the challenge you were addressing this week, 3) where you are in addressing that challenge, and 4) what comes next. (Sound familiar?) It doesn't matter if you're winging this entirely verbally, giving a formal presentation with slides, or communicating via interpretive dance, you pretty much must have all four parts.

So let's focus on some of the finer points in how to think about this.

The 1:1 Meeting

You can pretty much assume that your boss knows the overall project you're working on. If they aren't knee deep in your work, I suggest starting by reminding them of your overall task, e.g., "I'm working on our server performance," and then helping them recall where you were at the end of your last meeting, e.g., "Recall that when last we met (or at the last project update), I had been working on <something> and had gotten to <some particular point>." If they are knee-deep in your project, you can probably skip that. But even if they don't need that context, because, for example, you just met yesterday, I still recommend starting out with what you were working on during the time since the last meeting. For example, "My focus the past day/week/month has been on improving the throughput of our server." If this has been a long-term undertaking, then perhaps there has been progress reported at previous meetings, "And so far, we have 1) added multi-threading, and 2) partitioned the data structures to avoid too much synchronization overhead."

At this point, you've brought your boss up to date on the context in which this period's work happened. This is really important. All too often, it's easy to forget that while every teeny tiny detail is crystal clear in your mind, that is not the case for your boss. If you met two days ago, they may very well be on the same page as you, but a short reminder won't hurt.  If it's been longer, your boss is interacting with way too many people, they have been gone, they have been distracted, or (as is the case with me) they simply have a sieve-like brain, getting them focused on your details will let you make the most of a a meeting.

Now you can shift into "the problem." What specific thing were you trying to accomplish? It may have been an easy week: you had a straight forward implementation challenge; it may have been a difficult week: you were tackling a completely wide open, open-ended task. In either case, the problem/solution part of your meeting is where you can engage others in helping you, so think in advance of any areas in which you might want assistance. In explaining what you were working on during this period, be as specific as possible. You might use data you gathered to describe the problem in more detail. For example, "It was unclear where the next bottleneck was, so I spent some time profiling the server." This is beginning to blur the line between problem and solution, but I think that's actually just fine in this context.

So, the problem is really what you started out the day/week/month intending to do and the solution part consists of all the things you did to address that problem. Again, be specific, show data. If you spent the time reading papers, organize your reading into a structure, so you can say something more meaningful than, "I read a bunch of papers."  Instead, start with why you were reading papers (problem), "I did not feel that I had sufficient background in different ways to improve server performance, so I looked for papers describing similar tuning efforts. I looked in the following conferences for papers, and I found N papers that seemed appropriate."  That is what your problem phase looks like -- in your solution phase, report what you learned. "After reading them, I realize that performance tuning breaks down into three basic classes .... the first two don't apply here, so I skimmed those papers for techniques, but focused primarily on the third class, which was closest to our work. I identified the following three things that might be good next steps."

I picked this example, because I find that many times, students do need to spend time reading, but there is not necessarily a clear goal or perspective from which the reading happens. The more you can focus your reading on trying to fill a specific gap, learn something specific, or build background in a particular area, the more value you will get out of that reading. However, you can apply that same process to your weekly work.

Once you've presented what you've actually been working on you should find yourself in one of two positions. You might know exactly what needs to happen next and you lay that out as your final "Next Up" (happily ever after).  Or, you could use some help.  Then you invite discussion and brainstorming around your problem. Continuing our example, "The profiling didn't seem to indicate any one area where all our time is going, so I'm not really sure what to try next. I was hoping we might brainstorm ideas."

So now you have turned your "status update" with your "boss" into an engaged discussion. I intentionally started with this one, as it's likely to be the most detailed (and perhaps the most stress-inducing for some); all the others are simply subsets of the 1:1 update, so let's move on to them.

The Project Group Meeting

Your assumption here is that everyone knows the project you are working on, but may not know all the intricate details of the specific thing that you are doing. So, your context starts pretty much at sentence number two of the 1:1 context setting part: "Recall that when last we met (or at the last project update), I had been working on <something> and had gotten to <some particular point>." Assuming that these meetings are happening relatively frequently, that's probably all the context you need.  You can move right into problem/solution.

The problem/solution part of your status can be pretty much identical to what you might do in the 1:1 meeting. And it could lead to the, "Here is what I have to do next," or you might say, "I could use some help brainstorming next steps. Perhaps after all the status updates, I could get a few minutes from the team to do this?" If your project meetings are jam-packed, then ask who might be willing to meet at a different time to do this. I fear that students often feel that they must solve every single problem all by themselves, and that's not how research (or development) works.  It's a team sport. It is absolutely OK to ask for help; it just means that when others ask for help, you should try to step up and help. (It's OK to say, "You know I don't have much experience in this area, but I'm happy to do my best to help!") 

The Large Group Meeting

For this one, think of Systopia Standup. Before going into the specifics, I want to take a moment and explain why we have a large-group standup. When I joined UBC, after the students got to know me a bit better, the single most frequent complaint I heard was, "We have no intellectual cohesion. No one knows what anyone else is working on." TLDR: Our large-group standup was invented as a solution to this problem.  But let's ask the question that underlies that complaint: Why is it important to know what others are working on?  For some, the answer may be obvious, but for others it might not be, so I want to be explicit about this.

Why have large group status updates where hardly anyone really understands what I'm doing?

1. The people in the lab form the beginning of your professional social network. You may have absolutely nothing in common with them outside of the lab, but simply by virtue of being members of the same lab, you have a common experience. These people will show up over and over again throughout your professional career, and you never know when you might suddenly find yourself wanting their expertise, assistance, or advice. You don't have to become best friends, but you should take advantage of this opportunity to forge professional relationships with your lab mates. (You never know, 25 years later, you might find that you are their colleague, e.g., Will Evans and I were at UC Berkeley at the same time, and although we were not in the same research group [as I'm sure should be apparent], we knew each other well enough that when I intereviewed here, I felt that I had a colleague here that I knew].
2. Great research rarely happens in silos. Elegant solutions to real problems frequently require expertise in many areas or at least different perspectives from within the same area. If you have some understanding of what your peers are working on, you can figure out who to reach out to when you encounter an obstacle. Help need not only come from your own project participants or only from students with your advisor. Here are some examples of ways in which my students have gained explicit benefit from knowing what's happening around the lab: Ali and I were stumped on a proof for a paper on weighted GOSDT. One of us mentioned it in standup and the next thing you know, Mathias and Ali had come up with a lovely probabilistic bound on the problem we were trying to solve. Zainab is getting direct end-user feedback on her system. It's not a formal user study; it's not even a pilot. It's obvious to us that Paul is the right person to ask about this. (And I know that Nichole has also had similar experiences.)
3. Our field is HUGE. You can't possibly know everything happening in it or even all the techniques people in different areas use. Large group status updates give you a glimpse of what's happening other areas. These are an incredible learning opportunity.
4. Some day you are going to either A) be on the job market or B) be talking to someone who is looking to hire someone. The best thing that labmates can do for one another is promote each other. If you are in category A, would you not love it if your peers introduced you to their former managers who might be looking for someone just like you? Think how impressed your colleague in situation B will be when you say, "Oh, one of my labmates would be perfect for this." If you don't know what others in your lab are doing (and they don't know what you are doing), none of this can happen.
5. And speaking of that job market. I am 100% convinced that one of the reasons I was as successful as I was on the new prof job market (oh so long ago) is that I could actually talk to people in fields other than my own. I knew enough about what folks in my lab were doing and some of the big ideas others in the department were working on that I could ask questions of everyone, from theoretician to computer architect. You want to be that candidate, and you can't be that candidate by staying firmly in a silo and ignoring all that is happening around you.

End Digression

OK, so let's say you now understand why large group status updates are useful. How do you do a good one? Tell a very short fairytale (a nice short story is way better than and endless list of tiny things you accomplished).

Context: Recall that I work on the Awesome Server Project.

Problem: This week I was trying to figure out how to improve throughput on the server. We are handling X requests/per second and the state of the art systems seem to be able to handle 2X requests/second.  We don't necessarily have to match that performance, since we do This Awesome Thing. But a 100% performance penalty will never be considered worthwhile, so we need to get performance to within at least 10% of state of the art.

Solution: This week I tried removing every other line of code. Good news: Performance was great. Bad news: The server no longer works.

Up Next: I think I'm going to be a bit more strategic in what I remove; I'm going to do some dead code elimination to see if I can improve the iCache hit rate, since I took some measurements and I'm getting truly awful results for that.

It's just another fairy-tale, but one focused on your activities for the week, not your overall project, not your dissertation, just the thing that has been keeping you up at night this week.

Wrapping Up

Here is a tiny table to remind you how to think about the four parts of the fairytale in different settings.

	The 1:1	Project Update	Group Update
Context	Remind, Recall, Focus	Recall	Recall (Big Picture)
Problem	Specific (maybe with data)	Specific	Importance
Solution	Specific (include data)	Specific (include data)	Overview
Next Up	Brainstorm or Plan	Specific Plan	High-level Plan