The Data Warehouse is Dead — Long Live the Data Warehouse!
September 14, 2018
Author: Jon Bock
After a number of stories about his illness and death had appeared in the press, Mark Twain famously wrote in the New York Journal in 1897 that “The report of my death was an exaggeration.”
If there’s a technology that’s had its share of obituaries written about it, it would be data warehousing. Data warehousing first came into the market almost 30 years ago. As a result of that long tenure, the death of the data warehouse has been claimed on more than one occasion. Most recently, there have been swirls of discussion around whether “big data” was the final nail in the coffin for data warehousing—the speculation felt that the volume, variety, and velocity of big data would prove too much for data warehouses, while Hadoop and similar big data platforms would take over the role of the data warehouse.
Fact or fiction? Snowflake sponsored a recent survey by Dimensional Research that looked for some answers. Dimensional Research gathered results from 319 data professionals who completed a survey about data warehousing, big data, and Hadoop. Their answers provided some interesting insights:
The data warehouse is far from dead. When asked “Is the level of investment in your company’s data warehouse changing?” 96% of data professionals indicated that they are not decreasing their investment in data warehousing. In fact, 70% indicated that they are actually increasing their investment.
Hadoop is not a replacement for the data warehouse. In response to the question “Do you consider Hadoop as a possible replacement for existing data warehouses?” only 4% of respondents said that Hadoop will completely replace their data warehouse. 32% saw some overlap between Hadoop and their data warehouse, but the largest share (64%) saw Hadoop and their data warehouse as complimentary.
Why is that the case? The primary concerns cited about Hadoop were lack of expertise, complexity, and lack of a compelling business need.
But the data warehouse isn’t perfect. Only 3% of data professionals said that they don’t have any challenges with data warehousing. The rest mentioned challenges that included scalability, cost, ease of use, performance, expertise, and time to deploy.
The moral of the story: it’s not that data warehousing is dead, it’s that the technology needs to evolve. Data warehousing has proven its value over multiple decades, but needs to address key concerns in order to continue to maintain its role as the key repository for enterprise data reporting and analytics.
Want to read more? The full survey report is available for download on our website.