Echoes of the Dot-Com Bust in the Big-Data Gold Rush

The hype around big data is starting to sound like an eerie echo from the dot-com boom.

Without question, big data can radically improve the way many of Australia’s major companies do business.

The same could always be said of the Internet, but that did not stop the dot-com crash, which set back the growth of the digital economy by up to a decade.

The dot-com era came to a painful end because of hype and ill-informed investment by companies with little understanding of the real economics of Internet business.

They were ripe targets for the “shovel sellers”—infrastructure providers who sold the servers, the routers, and the storage, and who were typically paid up front.

Apart from a few dot-com survivors such as Amazon, the greatest beneficiaries of the dot-com boom were these shovel sellers, who know that finding gold is not the only way to make it rich in a gold rush.

The big-data revolution may be repeating the pattern. Relentless sales pitches and conferences push the big-data story, but there is very little in the way of realistic, business-changing application to create competitive advantage.

Big data is less about technology and more about organizational mind-set and a new form of practical business savvy. That is not the way most businesses are seeing it, and vast sums are being spent on hardware and software that may never be used.

From our work with companies in Australia and internationally, we see five obstacles. First, most companies do not have a sound grasp of data they have had for years. They don’t make use of the “small” data directly provided by their customers and staff and rely instead on entrenched wisdom, anecdote, and assumption. A basic curiosity about how their organization and industry really works is essential for data analytics to make a difference.

Second, where companies have scaled up their analytic ambition, many are hitting roadblocks when it comes to acting on the insights. Successful Internet companies such as Google, Facebook, Amazon, and Netflix generate every move from these insights.

But in the mainstream, many businesses still run on the gut instinct of the chief executive, who is not comfortable with such a powerful and unpredictable challenge to his or her wisdom. Embracing big data requires a fundamental change in orientation for most companies.

Third, big data done well can yield an accelerating stream of highly specific new growth opportunities. Businesses are often not ready for this, even if the chief executive is onside. They are set up to execute a small number of big-picture strategies over months and even years. Big data usually yields many smaller ideas requiring a highly efficient execution machine. Insights must be put into practice before competitors copy them or customer demand wanes.

The fourth issue concerns the hype around big data and the pressure on boards and chief executives to “do something.”

The need for action is commonly delegated to CIOs, who pull the levers over which they have control: they build data warehouses, initiate data cleansing and architecture improvement, and deal with data governance. But the underpinnings of big data are not in these areas. It can thrive on highly cost-effective cloud services and diverse and often “ownerless” data sources. Terabytes of data can be stored for as little as $100 on a pay-per-use basis.

Tackling big data with traditional enterprise IT approaches is expensive and inevitably leads to dashed expectations.

The fifth major barrier arises when companies that have overcome the above obstacles try to scale up their isolated successes. When you have so much data and potential in front of you, where do you start?

This is the bottleneck in the big-data revolution—a global shortage of skilled data navigators who can act as a bridge between the thousands of questions that businesses are asking (or should ask) on the one hand, and the potential of analysis to answer them on the other. Big data might tell your marketing manager an above-average number of cinema goers on Tuesday who saw the latest X-Men ate a Mars bar for breakfast, but how does that help you make money?

The Googles of this world prove the power of big data, but it requires much more than ingenious algorithms, technology spending, and a passing interest. Chief executives and their boards need to understand the business-changing implications of analytics in their businesses and only then commence spending.

This commentary was originally published in The Australian.