Highlights From the Sloan Sports Analytics Conference

I spent the last two days at the MIT Sloan Sports Analytics Conference in Boston. This is the second year that I have attended. It was a great two days for sports nerds like me.

The conference has a number of parallel sessions with a who’s who cast of sports executives, media, vendors, and academics. To give you an example, the opening panel “Revenge of the Nerds” included Mark Cuban (owner of the Dallas Mavericks), Daryl Morey (GM of the Houston Rockets), Paraag Marathe (COO of the San Francisco 49ers), Nate Silver (Author of Signal to Noise, etc.), and Michael Lewis (author of MoneyBall, etc.).

While the discussions focussed on sports, many of the insights were applicable to the broader business community as well. Here are some of the themes that I took away from the conference:

• From individual capabilities to system success. A lot of the analytics like XY coordinates in basketball and FIP in baseball look at the capabilities of individual players. But the success of a team often depends on how players perform as a group. This was the basis of some discussion around defensive systems in basketball and the value of positional flexibility in MLB lineups.
• Focus on successful behaviors instead of outcomes. A common theme across sessions was the notion of using analytics to understand capabilities and behaviors, not outcomes. The same way that pitchers are better measured by opponents’ on base percentage than by their win-loss record. In this way, you can better measure players against the things that they contribute (or don’t) to winning the game.
• It’s about good decisions, not right answers. Throughout the conference people reminded the “data geeks” that analytics need to support decision making processes. Stan Van Gundy warns “a lot of you analytics people think the game is a video game and that your players will always react the way that your models say they will.” Sam Hinkie of the Houston Rockets said that analytical models are imperfect representations of the real world, so you can’t assume that you know everything. Analysts need to have empathy for people’s jobs and goals, build trust, and use the data as a foundation for discussions. Alec Schneider of the Cleveland Browns noted that no one relates to scattergrams and that there’s a lot of power in using anecdotes to make the findings of analytics more compelling.
• Medical analytics is the next frontier. It’s clear that all sports are looking into how they can use analytics to predict and prevent injuries as well as to help identify optimal treatments.The best player in the world become useless if he or she is not playing because of an injury. One of the research efforts found that the occurrence of injuries was heavily dependent on age but the length of time for recovery was not. With an understanding of potential injuries, a team can develop an appropriate value for players and build a roster based on those expectations. They can also chose prevention and treatment options that will deliver the best value of that player for the organization. Marc Cuban noted “we’re locking in our medical staff for longer than we lock in our players.” Daryl Morey half joked about the lack of medical analytics when he said “there’s no double-blind test on who works on Yao Ming’s foot.”
• Look for in-game analytics. The data collection and analysis has reached a point where some of the analytical data from a game could be used to drive decisions during the games. While the technology may be ready, the leagues and the coaches aren’t quite there yet.
• Location data will be huge in basketball. Half of the NBA stadiums have cameras that track the XY location of all of the players and the ball in near real-time. Much of the coolest research I saw came from this data, including findings that David Lee is the worst internal defender in the league. This data will help create analysis around defense that is sorely missing
• Human performance is not yet understood. Nate SIlver observed that “we haven’t done a lot of work in analytics to see if a coach is able to get more performance out of players.” Paraag Marathe notes that players’ physical attributes have a narrow margin of difference, which is why the mental aspect is so important. Marc Cuban explained that he sits close to the bench in games and goes to practices so he can observe the interactions of his players and coaches.
• Watch out, metrics matter. This wasn’t a theme, but it was a very insightful observation by Joe Posnanski of NBC Sports who said that the “save” changed baseball. Because people started to track this statistic, we have one inning pitchers who make a lot of money. We created the stat, started measuring it, and then we started to pay people money to achieve it. The number changed the game. It not only changed how players are paid, but also on how managers manage the game.
• Analytics need to augment, not replace sports expertise. Even with the best analytics, scouts and human expertise don’t go away. Sig Mejdal of the Houston Astros says that “We need to treat the scouts respectfully, not like they show on MoneyBall.” Scouts provide important insights along with the analytics. We use them to collect information and to provide hypotheses.
• Use analytics to minimize bias. In several sessions, people discussed the role of analytics in removing cognitive biases. We all have biases that shape how we look at the world. Some common biases include recency (we overly focus on recent experiences), confirmation (we overly focus on things that reinforce our beliefs and discount those that don’t), halo effect (we have positive impressions of a player’s skills because we like him), and horn effect (we have negative impressions of a player’s skills because we don’t like her). Google actually trains some of its leaders on how to minimize those cognitive biases.
• Marc Cuban and Tony Reali rock. It must be pretty cool to be Mark Cuban. He’s uninhibited, funny and has a strong business sense. We actually met in the hallway and shared our common background as standout JCC basketball players in our youth. Tony Reali was a fantastic moderator, very funny and insightful. I didn’t take notes from his session on football analysis, just enjoyed being there. I might also throw Stan Van Gundy into this mix, he was pretty dynamic and entertaining as well.

Highlights From Sessions

Here are some highlights from the sessions that I attended. You will see that the comments from different speakers cover sports-specific insights, more generalized business insights, and just some cool things that I heard. Wherever possible I put the information in the voice of the speakers, although they may not be exact quotes.

If you like this level of detail, then you might want to check out the 2013 research paper finalists. The documents are pretty short and fairly easy to read.

Revenge of the Nerds:

Nate Silver (Author of Signal to Noise, etc.):

• Last time I played sports was baseball in 6th grade, the year they started pitching overhand.
• Fantasy sports is a good way to learn applied statistics.
• Almost all of the ROI in baseball is to draft and develop young players.
• In almost any field you get diminishing returns as you add complexity to the analytics model. But we are getting fresh data, like break in guys curve ball and movement of people on the field. I would use the new data for predictive. But there’s no more low hanging fruit with stupid teams.
• We haven’t done a lot of work yet with analytics to see if a coach is able to get more performance out of players.

Mark Cuban (Owner, Dallas Mavericks):

• You don’t really own the team, the community owns the team.
• The number on job of a GM is not to win a championship, but to keep their jobs. They are over-incented to take big risks. They lose their jobs with mediocrity.
• We are just scratching the surface with XY data. Looking to extend data capture at practices and training. Applying analysis to what can we learn about developing talent.
• Medical analysis is important for injury prevention and also in-game treatment for things like anti-inflammatory and genetic testing. We’re locking in our medical staff for longer than we lock in our players.
• There aren’t any analytics on organizational dynamics, so I need to be there to see what happens in the huddle and on the bench to pick up on those dynamics to see if there are issues I need to address.

Michael Lewis (Author of MoneyBall, etc.):

• Lots of people got angry with me after Money Ball. I found out it was because I was costing people their jobs.

Daryl Morey (GM, Houston Rockets):

• Coaches are treated terribly, they are generally only around for three years, so they naturally look for short-term victories.
• New data like XY data is helping to improve analytics. Location of people and ball in real-time on the court, getting data 30 times/second. That’s where the action is. On the court, we are nowhere in basketball. We’re running the same stuff we’ve been running for 20 years. XY data will help identify plays and areas to exploit.
• We don’t have good data on medical staff. There aren’t any double-blind test on who works on Yao Ming’s foot.

Paraag Marathe (COO, San Francisco 49ers):

• Analytic work is less than 50%. Communicating and getting buy-in is where the difficulty is. Once I realized that, everything changed for me. Making it everyone’s collective idea, not just some little Indian guy walking around with charts.
• Chemistry on the coaching staff is an area we can’t measure too well yet. Often times the styles don’t match.
• A lot of uncharted areas in football. One area is mental aptitude. Physical attributes have a narrow margin, but the mental is so important.

It’s Not You, It’s Me: Break-Ups in Sports:

Brian Burke (Former president and GM, Toronto Maple Leafs):

• If I overpaid for an athlete, it’s not the player’s fault. I’m an idiot.
• If you want to be in sports, you’re gonna get fired and you’re going to have to move. It’s very public. There’s a humiliating aspect to it.
• The worst thing that ever happened on sports is sports radio. And the internet is sports radio on steroids. No one has ever run a title with Moneyball. Numbers are overrated by a long way.

Stan Van Gundy (Former coach, Orlando Magic and Miami Heat):

• Believe me, it was a lot harder on the guy that gets fired than the guy who does the firing.
• The Magic had told Dwight that he could decide the fate of Otis and me. I told management that this is a problem in the locker room and they should do one of those two things. Not that bullshit faith in the coach statement that means you are going to fire me in two weeks. DO one of two things. Extend my contract or fire me.
• I have to find ways to put my players in better spots to be successful.
• We expect players to act like they’re 45 year-old, but they’re 25 year-old kids. When I think about what I was doing in my 20s and people would scrutinize what I was doing, it would have been crazy,

Steve Pagliuca (Co-owner, Boston Celtics):

• You need to trust your GM, your people on the field. 99 times out of 100 we will take the recommendation of the GM. Our role is as strategic advisors and we need to trust the people who are full-time on their jobs.

Bill Polian (Former GM, Indianapolis Colts):

• The more low-key and teachers they are, the longer that coaches last.
• In the NFL, your core (about 12 guys) stays together for 8, maybe 10 years. s. The rest turns over every 4 years or so.
• You need to consistently find way to motivate your players, but understand that it’s a long season, a grind, and need to know when to back off and when to push. Bill Walsh said he could motivate his guys to play 4 games out of 20 in the year.
• We live ain a parallel universe. Inside the game, we look at the longer-term season. You don’t overreact. The media sees it completely differently. They want perfection 82 games or 162 games per year.
• Coaches and GMs constantly preach against distractions. Athletes need single-minded focus. Distractions detract from that focus.

SAP: New Technologies to Drive Fan Engagement and Team Performance:

Paraag Marathe (COO, San Francisco 49ers):

• People suffer from recency bias and confirmation bias and technology helps minimize that.
• We are re-imagining the fan experience in new stadium: We compete with the couch to get someone to go to the game. You can’t match the spirit of the moment in the stadium on your ouch.

Steve Hellmut (EVP of Operations and Technology, NBA):

• Coaches generally only look back 5 to 10 games and come up with rules of thumb like “Keep Paul Pierce off the left elbow.”
• Bobby Knight once said: “Stats accuse and videos indict.”

Baseball Analytics:

Farhan Zaidi (Director of Baseball Operations, Oakland As):

• Billy sometimes calls me the emotional stats guy.
• When you are an analytical team and follow through on the data, there is a lower bound to the wins, maybe 75.
• Analysis is more about skills, not outcome. We’re more fixated on outcomes more than other sports. We need to find the best skilled players, not the ones that put up the best numbers last year.
• We can tell how good a prospect is, but need to add more analysis to understand how good they can be. There are some success-oriented behaviors that we’ve seen anecdotally, but we don’t have the objective data on it.

Ben Jedlovec (VP, Baseball Info Solutions):

• All scouts take the same data and interpret it in different ways. That’s where the differentiation is, in interpreting the data.

Jonah Karl (Staff Writer, Grantland):

• The teams with money build a “stars and scrubs roster.” They pay for the expensive stars and fill in with low-priced players.

Joe Posnanski (Senior Writer, NBC Sports):

• The identification of a save as a metric changed baseball. Because of it, we have one inning pitchers that make a lot of money. We created statistics, started measuring it, and the then we started to pay people money to achieve it. The number changed the game. Not only changed how players are paid, but also on how managers manage the game.

Voros McCracken (Statistical Analyst/Writer):

• Can’t get carried away with the next thing and forget what you learned. Just because everyone know OPS is important doesn’t make it not important.
• Moving towards modeling where you aren’t looking at home runs and walks, but looking more at person as set of capabilities and what those things mean to our team. We’re looking for underlying capabilities that lead to the statistics. A lot of that data is coming from video data analysis.
• At trade deadline almost all of the contending teams make at least one trade, driven by the appearance that they are trying to win games.

Separating True Performance From Randomness:

Daryl Morey (GM, Houston Rockets):

• There’s too much focus on outcome.

Alec Scheiner (President, Cleveland Browns):

• Most of the problems come from focusing on the outcomes. Instead, we look for things that don’t correlate with success.
• There are some things you are sure about and peddle those first. And then you try to communicate things that are about having better odds.
• You need to convince people that your way (with analytics) can hep the person have even more success. You have to pick your spots or you’ll be seen as too aggressive and won’t be asked to the table.

Jeff Ma (CEO, tenXer):

• Separate decision from output. When you see something that doesn’t make sense, try to understand the root causes.
• Humans are good at asking questions and computers are good at answering them.
• Need to look outcomes over the long run. The short run can have a lot of noise.

Phil Birnbaum (Editor, By the Numbers):

• Figure out what to measure that correlates better to talent than outputs like winning.
• One of the things that analytics can do very well is filter out the really stupid decisions.

Nate Silver (Author of Signal to Noise, etc.)

• People have overactive imaginations. We see patterns in the noise all the time in random data.

Beyond Crunching Numbers: How to Have Influence:

Ken Cantanella (Director of Basketball Operations, Detroit Pistons):

• Build small successes and then bring the big ideas. Ask questions, don’t assume I have the answers. Ask questions. Take the time to get to know the people and ask questions.
• Even though we have scatter charts, the power of an anecdote goes a long way.
• Never be sure you are 100% right when you enter the room. No way that someone else can have something to add to the discussion. We’re really trying to do is to make an impact on an organization by transferring their knowledge to someone in the organization that will make use of it. Let those people add to the discussion and tinker with the ideas.

Sig Mejdal (Director of Decision Sciences, Houston Astros):

• I say “I don’t know” enough that makes people uncomfortable. If you are honest with them, then you build trust. They can tell the difference between what I know and what I don’t know. Sometimes says go with your gut, it’s beter than whatever my data will tell you.
• We need to treat the scouts respectfully, not like MoneyBall. They provide important information, it’s not an either/or with the analysis and the human observations. We use the scouts to collect information and to provide hypotheses that we feed into our analytics.

Sam Hinkie (EVP of Basketball Operations, Houston Rockets):

• A model is an imperfect representation of the real world. Let them poke holes until they have confidence in it.
• I share only as much as they are interested in, no more. Don’t push the underlying details.
• Meet people wherever they are. We don’t ask people to print things out, we bring the paper to them when they need it. They each get handed different documents based on what they want to see. We try and fit to their styles, not try and have them fit ours.

Prasad Setty (VP of People Analytics & Compensation, Google):

• Committees across company make decisions about engineering promotions twice per year. We ran our analytical models and found that we could make 30% of decisions with 90% accuracy without having the committees go over t. We thought they’d be excited because we could reduce overhead by 30%. They hated it. I learned that our analytics isn’t about replacing humans, but in helping to reduce bias from human decisions.
• We use cognitive biased training where people lok for and are conscious of biases such as halo effect, horn effect, and recency.
• We found that we were not learning any more useful information for making a hiring decision after four interviews so we cut the number of interviews to four.

Cade Massey (Visiting Assistant Professor, The Wharton School):

• Knowing the right answer may be less than half of the problem.

Alec Schneider (President, Cleveland Browns):

• A lot of people make decisions, so you need to spend a lot of time getting your message across.
• You are always going to be overruled.
• You need to use anecdotes to drive analytics. No one relates to scattergrams.
• If you don’t have trust with decisions makers then it won’t work. They have ot think that you are trying to help them. They’ve been successful without you. You need to build that trust up so the moment when you say you are pretty certain about someothing they are willing to use the information.
• You need to understand the jobs of the people who are going to use the output of the analysis. Think about the person and have a real conversation.
• Everyone does their own form af analytics. Some watch fils and make decisions. We want to take what they do in looking at 50 plays and look at 50,000 instead.

Basketball Analytics:

Stan Van Gundy (Former coach, Orlando Magic and Miami Heat):

• A lot of you analytics people think the game is a video game and that your players will always react the way that your models say they will.
• We didn’t do two-for-ones, even if the stats say to do it. One player runs in and jacks up a dumbass shot, then everyone watches and stands around. Then the beginning of the next quarter some other guy might jack up a dumb shot. Every time you make an exception to how you play the game, you are breaking down your system a little bit.
• Football people have a saying, cluttered minds equal slow feet.
• Location analytics has a problem. It’s missing the point because it looks at individuals and does not look at the system. Data may show that JJ Reddick is a bad defensive player because he’s small and people can shoot over him, but he will not miss a single defensive rotation.
• If it’s okay for a player to miss his rotations because he does other things well, then other people won’t follow the plan and soon you will not have a defensive culture.

R.C. Buford (President, San Antonio Spurs):

• We learned from the Celtics that we didn’t need to focus as much on defensive rebounds to improve our defensive efficiency.
• We don’t show players spreadsheets; we show them the insights in films.
• We have people crunching good numbers, but we need better people communicating the numbers.
• We’re trying to capture more data in medical and technology then every before.

Kevin Pritchard (GM, Indian Pacers):

• I want my coach to think of scenarios before the game. As a GM I don’t want to be surprised, because two minutes after the game the owners are going to ask me.
• I think we are getting better at understanding of defense.
• Spending a lot of effort understanding rest and recovery. It’s great for the rest of the team, as well, when you rest some of your starts because it gives some other guys some more minutes.

Mike Zarren (Assistant GM, Boston Celtics):

• Everyone talks about points per possession which would have been foreign a few years ago.
• People don’t look at the variance too much. Everything is always an average. You need to think about how confident you should be with the information and advice from analytics.
• I wish we were doing more injury analytics.
• You can do everything right and not win, but you can’t do everything wrong and win.

The bottom line: I loved SSAC and am already looking forward to next year.