Survivor Bias

In a speech yesterday Jon Moulton of Alchemy criticised the data used in research recently published by BVCA. As Patrick Hosking in The Times Business Commentary explained – A Blunt Message from Moulton – the figures had shown Survivor Bias. IE Consulting, who produced the research, had claimed to show that, in the last five years, P/E backed companies had increased staff by 8% pa compared to 0.4% pa for FTSE100 companies. Moulton said those figures were ‘very dodgy’ as they didn’t include all the P/E backed companies that had failed. If they had been included, Moulton reckoned “the number should be more like zero”.

Let me tell Jon that similar dodgy practices are one of the reasons why so many tech research houses produce statistics, and forecasts, that are often far too optimistic.

Let me list a few of the more obvious mantraps:

- information is presented in ‘units’ where as markets are measured in ‘revenues’. Anyone looking at the PC charts right now might believe that we are still in a fast growing market. But as anyone can work out, if PC unit volumes are rising at 10% pa but prices falling by 20% per unit…it’s not quite so good!

Determining historic market growth is often calculated by the detailed examination of the performance of the companies operating in that area. So many just look at the results of the Big Boys which distort statistics. At Holway, in the 1990s, our figures were based on the performance of getting on for 2000 companies. But, however you do it, you have to take care of the following:

- companies always headline “continuing revenues” and researchers often use that as their base. So, just as Moulton suggests, Survivor Bias inflates the overall figures. Indeed, although we at Holway always took the companies that ceased trading into account (consistently, by the way, 20%+ of the 1000+ companies we tracked!) I am not aware that any other researcher did that

- the less knowledgeable researchers often use the headline revenue figures – even though these are almost always inflated by acquisitions

- then we come to the ‘constant currency’ debate. Almost all the major researchers are US-owned and (as we know) Americans believe there is only one global currency – the dollar. Remember that $1 = £1 in 1985. So the UK market has doubled in dollar terms on currency flucations alone in the past 20 years.

All the mantraps I’ve listed above inflate growth. Maybe that’s why I/we were always accused of being ‘too gloomy’ whereas, in fact, we were just being ‘more accurate’.