When Do We Know if a Goalie is Good?
Introduction
About halfway through the 2013-2014 fantasy hockey season, a LWL regular posed the following question to our staff:
At what point into a goalie's career do we know how good he is?
The question highlights a fundamental issue in the game and has wide-ranging implications for fantasy hockey. I knew immediately that this was a question I wanted to spend quality time answering.
The Method
One of the key problems with goaltender evaluation is the small sample size. And by small sample size (at this point in the discussion) I'm not referring to how many shots the goalie has faced at this point in his career - because, in fact, that's what the original question is about after all. By small sample size, I mean how many goalies, for example, have followed the same career trajectory as Sergei Bobrovsky? He posted the following save percentages in his first three seasons: .915, .899, and .932. If I were interested in digging up all NHL goalies who have followed a similar trajectory, how many do you think I would find? No matter the number, it wouldn't be sufficient to build an analysis upon.
To overcome this lack of data, I decided to approach this problem using simulated data. I would programmatically create a league-average goalie (SV% = .9138) and fire a lot of pucks at him. If you consider that a typical starting goalie plays about 60 games and faces about 30 shots per game, then a 10-year career would yield about 18,000 shots against. So, I fired 18,000 shots at my league-average goalie. But if I analyzed only one goalie, I'd be opening myself up to small sample size problems all over again. Instead, I simulated 1000 careers of league-average goalies with each goalie facing 18,000 shots. In a 30-team league, you'd need about 250 seasons of data to match that number of shots.
The Results
Below, I've plotted the results of the 1000 simulated goalie careers. It's worth mentioning again that all of these goalies are average. That is, they are capable of stopping the puck 91.38% of the time.
What you see above is 1000 career trajectories for average NHL goalies. The SV% along the vertical axis is cumulative, i.e., it tells you the career SV% of a goalie based on how many shots he has faced up to that point in his career. For convenience, I've placed tick marks along the horizontal axis to represent a single season (using the assumption that a goalie plays 60 games and faces 30 shots in each game - neither of which are in any way critical to the outcome of the simuations).
So, what is the answer to the original question that prompted this article? I'd focus on the part of the graph where the spread of the data becomes relatively constant. Without overcomplicating things, nobody would call you crazy for suggesting that you probably want at least 3000 shots of data before you start making bold claims about a goalie's talent. Anything shy of that and you might find yourself saying future Vezina winners don't belong in the NHL.
A Few More Conclusions
Consider the 300 shots against part of the graph. Imagine a vertical line running through the data at that mark. The simulation suggests that an average goalie is capable of posting almost any imaginable SV% over the course of 300 shots. An average goalie (over a 10-game stretch) can look like a rock star or a complete dud. Consider Frederik Andersen of the Anaheim Ducks; we don't know his talent level very well because he's only faced 783 shots in the NHL. But a lot of fantasy hockey managers already have a great impression of him. In his first 300 shots faced, he posted a .932 SV%. Does this mean Andersen is a good goalie? An average goalie? We don't know. We do know that an average goalie is certainly capable of posting a .932 over 10 games - and even a .923 after nearly 800 shots could still fall within the realm of possibility for an average goalie. If you think these results scream "UNCERTAINTY" then you are beginning to grasp the point of this article.
Another way to look at the data is the following: find the data point with these coordinates (4410, .903) on the chart. If you look closely, you'll see about 1-2 goalies in that spot? 1-2 out of a thousand is about 0.1% - 0.2%. So, if you have a goalie who posted a .903 over that many shots (4410), you can say with confidence that this goalie is definitely not average (instead, he's below average for sure). Those are Martin Brodeur's numbers over the past four seasons.
There are a lot of interesting takeaways buried in the simulation results. We've also run simulations for above-average goalies and below-average goalies and there is a wealth of fantasy hockey information to be explored. We'll be posting these results and our analysis of them in the 2014-2015 fantasy hockey draft kit. If you have any interesting interpretations or questions regarding the simulation, drop us a note in the comments!
3 Comments.
Gnial
June 23, 2014 2:50 pm
Mike
June 27, 2014 12:36 am
bottlecap
July 09, 2014 5:58 pm
So it looks like once you get to approximately the 3600 shot mark, you can start to be reasonably certain of goalie production (between ~.905 and ~.922 on the chart). However, we run into the same problem that we run into when trying to predict player skill - what if their skill level changes over time? The longer it takes to assemble the appropriate sample size, the less reliable that data is. 3600 shots works out to about 2 full seasons. Are goalies really the same player after 2 years that they were at the start? Perhaps in some cases.
I think the big takeaway from this is something I've been saying for a long time. Over the course of 100 shots (ie. one week of fantasy play) ANYTHING can happen. The goalie can post .80 or 1.000, and any attempts to predict week-to-week performance is a coin flip. So in head-to-head leagues, should you attribute much value to goalies? I don't think so. At the draft it is better to spend your picks targetting more consistent categories like FW and hits, and then cross your fingers that your bad goalies have a flukey season. (I won 3 fantasy years in a row after implementing this approach, and when I changed my plan last year and picked up a bunch of stud goalies, they invariably were a bust and I lost)
Thanks for the article!
@Gnial - while anything can happen over the course of a week in fantasy hockey, the coin Henrik Lundqvist is flipping has a different weight than the coin Mike Smith is flipping.
This sort of analysis clearly demonstrates how many shots are needed before a career sv% stabilizes and can be taken as an accurate measurement of the goalie's skill (by that measure), and also the expected variability given a specific sample size. Pretty cool to see - and why I come to LWL :). It'd also be pretty useful to see if certain goalies are under-valued because they had a "bad" year that falls within the expected distribution for the number of shots faced for a single season. You might also get an idea if coaching or team changes have really impacted a player, or if it's just the usual variability of the sample.
I'd be interested to see how the distribution/standard deviation around the mean change with variable career sv%. What's your plot look like for a .900 goalie, and a .920 goalie? Basically, can you determine how big a sample is needed before being able to predict the expected career sv% mean (correlating the distribution of a small sample size to the cumulative mean)? It may not be useful (perhaps too many shots are required), but it'd be worth looking into. Even a small correlation would be something.
It'd likely be very helpful to track veterans season-to-season to see when they start to have higher variation then you might expect (a sign of decline) or when the distribution starts shifting towards being predictive of a lower career sv%. Maybe just a control chart sort of plot of a moving average (x shots, maybe 1800, maybe more) would be useful in that way (with a line for the career sv%, and a band for the distribution around that mean given the averages sample size).