Can notebooks be successors of traditional BI tools?

business intelligence dashboard

The term, Business Intelligence was firstly mentioned in the 19th century, especially in 1865. Richard Millar Devens used the term in his book, Commercial and Business Anecdotes. With this expression he described how a banker gained more profit by being first in recieving the hottest information about the environment, especially the battles and political issues. He used that information for optimising the investments. This is considerd to be the first notable case of applying business intelligence.

 

 

Excerpt from Richard Millar Devens' book, Commercial and Business Anecdotes

After that we took a long path to today's BI. The official definiton of BI is that it is a "set of techniques and tools for the acquisition and transformation of raw data into meaningful and useful information for business analysis purposes", as Wikipedia describes.

Recent statistics about taking advantage of BI: five times more likely a company that uses analytics well is expected to make decisions faster than the competition. (Bain & Company)

 

Survey of Bain & Company

How does Business Intelligence perform today?

In my perspective Bi, especially data discovery is simply the connection, the communication channel between the data guys and the business decision makers, the top managers of an enterprise. As business is going to be faster and faster thanks to new technologies and the more and more automation, business methods also has to speed up inside the enterprises. Big Data technologies have opened the door for analysing data that we weren't be able to do before.

These impacts should shorten the development time of a new dashborad, and they should give the possibility to get the response of an ad-hoc question by the end-of the day. But do they? it won't matter that our queries runs much faster until the results won't be delivered faster.

Aberdeen Group measured BI processes within the best-in-class performing companies. Results showed that averagely 6.1 days it takes to get reports built with traditional BI tools. "We live in a generation of information immediacy, people have been conditioned to get what they want when they want it." Forrester states that 60 % Percentage of business and technology decision-makers report time concerns with creating or updating dashboards. And also a big amount of leaders see a lack of alignment between IT and business. Thirty-five percent of survey participants have experienced a lack of a cohesive strategy between IT and business silos.

Usage of Notebook-like tools

Notebooks, like Jupyter and Apache Zeppelin have become the best friends of data scientists recently. The biggest advantage of them instead of using traditional tools, like Tableau or Spotfire (which, I have to mention, are great tools) that they can connect to multiple datasources with multiple interfaes: even programatically and not only SQL.

Using a notebook gives a multifunctional tool which can handle all the aspects of a data project very quickly: all the procedures when you contact with data can be done, including:

  • ETL
  • Pipeline building
  • Visualization
  • Statistical analysis, machine learning

You can also easily share your visualizations or present the data from a notebook. If you can create everything from one tool, the whole procedure can be faster, because it needs a smaller skillset. It means that one person can do it instead of two or three. If there is a situation, and the managers need emergency reports, notebooks could be the fastest way to create and present them.

Traditional BI tool​

Notebook​

Interact with data​

SQL​

programmatically & SQL​

Data cleansing​

limited​

✓​

Visualization​

easy and great​

satisfying​

Data quality it needs​

prepared data​

can eat anything​

Statistical analysis​

limited​

full access to several librarys​

Skillset and tools​

multiple tool neeeded for the whole process​

same skillset but less tool (one)​

Sharability​

easy and can be customized​

in its infancy​

 Comparing notebook-like complex data processing tools versus traditional BI tools

Consequences and predictions for the future

We could easily lose one of the biggest the advantage of big data-related technologies in the tangled web of enterprise processes, the possibility to dive into a vast amount of data really fast and create fast analytics, when the managers don't have to wait days. In the near future, only enterprises that are the quickest at producing answers for their quistions can be the vicotrs of their market. What is really important is how fast a company can make good decisions.

With traditional tools it is not possible to handle everything from one place. You need multiple tools for processing your data, applying statistical analysis and visualization. There is a bigger change that those will need multiple people as well. Data science is changing. People in data industry is changing. As everything is speeding up, not only the enterprises, but we also have to keep up.

Perhaps in the future data engineers, analysts and scientists won't be such separated roles. Perhaps there will be data "guys" who will be able to handle modelling, visualization and ETL at the same time. Separating our tools obstructs us from being fast. We have to extend our skillset, and not only tha data guys, also, the managers have to keep up with tech era.

 

Business Intelligence Growth & Value: 7 Telling Stats / BetterBuys

What will the future bring? Will we separate our tools and our roles as well, or will we prefer complex notebook-like tools and complex roles? If you are working in the data era, you also have a big responsibility on deciding this. So let's build the future together.