Business Intelligence transformed the way we do business, but there is still unleashed potential within it. We are analyzing only a part of our data, not all of it. The insights we get are limited and in consequence, so are our decisions. This is mostly because of Dark Data and how hard it is to analyze. And also because of human bias that doesn’t exactly allows us to see outside the box, so to speak. Now, imagine if you could get insights on all of your data. Your use of BI would be much more meaningful and your decisions would be insight-driven. In order to get there, we must take a couple of things into account.
Dark Data = approximately 90% of your data
Let’s begin with the answer to one of the hottest questions out there in the IT industry: What is Dark Data? According to consulting expert Gartner, the definition for Dark Data is “information assets that organizations collect, process and store in the course of their regular business activity, but generally fail to use for other purposes”. In simple terms: data we have and own but don’t always know how to access and use. It can come from emails, credit card and shopper information, call center records, pictures, etc. Unfortunately, 90% of the data in organizations consists of Dark Data. To add to the drama, this 90% can be very costly and risky. But if we could actually get useful business insights from our Dark Data, the cost of storage and processing would be worth it.
The Streetlight Effect
There’s a lot of valuable information floating around in Dark Data that ends up being neglected. This can be due to the Streetlight Effect. This is basic human psychology: we look for what we are searching in the places we know, in our comfort zone. It is an observational bias that we all tend to fall into. So the questions we ask and the data we analyze is that which is familiar and we know is relevant. But we don’t get insights on all the data.
The Streetlight Effect is what is stopping us from actualizing the potential of all this data that we already have. It is preventing us from making more informed decisions. We keep analyzing questions to the same data over and over again and always getting similar results. We need answers, not just to the questions we ask, but rather to the questions we haven’t asked yet because we are not aware that we should ask them.
What happens in the BI market today?
Most solutions tell you to ask questions to get answers. You will get those answers from the 10% of data you know how to analyze. But there are revolutionary BI tools, like Panorama Software’s Necto, that tell you to get answers to your questions in that 10% of data…and get answers to questions you didn’t even think of asking on the other 90% automatically, thus avoiding the horrible combination of Dark Data + The Streetlight Effect. When you get automated insights on all your data, you can be inspired to have a specific question and generate an insight yourself or you can explore more automated insights.
What can happen in an insights-driven market?
Some examples are:
Telematics Insurance for Cars
The insurance company puts a telematics box, also known as a black box, into your car where you can’t see it. It measures several aspects of how, where, and when you drive. For example, it measures the time of the day when you drive, the speed on different types of roads, if you accelerate or brake sharply, etc. This information is then analyzed by the insurance company. It is used to provide you with a price customized insurance coverage according to your driving habits.
Mid Size Retailer
A mid-sized retailer had frequent product returns, inaccurate deliveries and queried invoices. With the appropriate insight they found out that the problem was caused because there were discrepancies in the customers’ addresses in the customer database. The company made corrected addresses, which in turn reduces product re-delivery costs. Customers paid invoices on time, which increased cash flow. With all this changes, the retailer was able to improve their customer service.
Cereal Manufacturer
A cereal manufacturer analyzed its contact center data. They found that people were constantly asking if product A was gluten-free. It wasn’t, but not because of its ingredients, but because it was made on the same production line as a product that contains gluten. The manufacturer just changed product A to a different production line and opened the product to a whole new market.
Financial Institution
A financial institution stored data in many different data sources. It was impossible to analyze all the data together for identification of new revenue opportunities. When they unified all the data in one data base and analyzed it they found an insight: they could group households together, making unified marketing offers to both husband and wife. And by analyzing further, they could offer other options like savings accounts for the children, even personalizing the offer to college aged children as “student savings account”.
Bakery
An industrial bakery realized through data analysis that they had a lot of chocolate filling leftovers from their star products. They also had a relatively high percentage of hot dog buns breaking in half during the production process; the company could not sell split buns for quality reasons. So they took the broken hot dog buns and filled them with the chocolate leftovers from other products. It started as a product to optimize waste and soon became their most sold product, generating pure revenue.
We can get automated insights, avoiding our own observational bias and thus tapping into Dark Data’s potential. Insights make our decisions much more accurate and can bring new business opportunities; opening our minds to new markets, new products and helping us develop better business strategies.
Latest posts by Deborah Hurtado (see all)
- Releasing BI’s Potential: From Dark Data To Insights - September 1, 2016