The problems of information diversity and the benefits of integration
The “single version of the truth” has become such an over-used term and also one that is heavily punted by those who can only talk about it however they often have very little understanding of the complexities in, for example, building a Data Warehouse to support this statement. (One of the key functions of a DW has always been to fulfil the function of “single-version-of-the-truth”….remember). It does remain an elusive ideal though in most organisations those are serious about their information assets. I am really beginning to wonder if there is actually such a thing as the single version of the truth? One could get philosophical about this topic but I will steer clear from that and rather focus on the real issues, challenges and trends of the current information rich business world.
Where we are and where is it going….
What value does a consistent and integrated base of information have for the business?
- Servicing both tactical and strategic business intelligence information requests – providing consistent answers, fast.
- A consolidated base of information can serve many purposes – provided its design and structure is flexible in other words ready to accommodate changing information requirements gracefully. In such cases, this base of information hardly operates in a vacuum, it interacts with various systems, which feeds data into it and also provides data to many outbound business processes.
- An automated data warehouse can automatically deliver reports, alerts and messages to those who require it in the format they requested.
- All of the above make it possible for the decision maker to have information available to support their actions and re-actions to rapidly changing market conditions and customer demands. In order to grow or retain business and to deliver improved services and products.
It started with data warehousing:
The data warehouse is supposed to make it possible, at a push of a button, to gain reassurance and confirmation that our organisation is operating like a well-tuned engine. Resting assured that an alarm would sound well in advance of any potential problems based on the fact that we have recorded the rules to trigger an early warning. Yet, although we have this performance management framework at our fingertips – there is still an outcry from business that information management and analysis/analytics are becoming more difficult. Additional sources of information are piled on and keep on expanding. Data volumes and different types of data are adding to the complexity of trying to contain the information wave.
The original intent of the data warehouse (or any consolidate base of information for that matter) of facilitating all information requests in a fast, effective and easy way, are not fully realised.
So what went wrong….
The Business Intelligence Satisfaction Gap
The more successful an IT department is at executing their first data warehouse, the more likely their users are to experience a “BI satisfaction gap” (some may want to call it a “dissatisfaction” gap). During the time when users are engaged in the design and prototyping of a warehouse, they receive a lot of attention. However as IT prepares the warehouse for production and starts dealing with all the overhead issues such as performance tuning, automation, change management and so forth – this often happens just as power-users are really starting to push the limits of the logical data model.
The result is a disconnect between user expectations and IT resources.
One way that savvy data warehouse managers learn to deal with this is by running overlapping design/build projects. Unfortunately, this is an expensive approach requiring, hardware, software and development staff. The typical warehouse may go 4, 5 or even 6 months between the implementation of major design changes. Even if we overlap development and cut this in half – two months is a long time to wait for a report.
In other words the traditional method of modelling and design is taking to long.
EII and future trends
Emerging: Enterprise Information Integration (EII)
The business imperative is for faster and easier analysis and presentation of information from multiple heterogeneous data stores.
Advances in techniques and technologies made Enterprise Information Integration (EII) possible. EII compliments and extends the existing BI infrastructure by accessing information from multiple data marts, enterprise data warehouses and operational data stores – even external data. This approach differ from traditional data marts and data warehouses – it gets created on demand rather than as part of an automated, scheduled process. One can think of EII as an ad-hoc form of data integration and analysis that can be used to rapidly prototype new analytic solutions.
From a BI perspective, EII technology is the glue that sits between the end-user and the sources of data they need to meet their request. Like an ETL tool, EII extracts and transforms data. What’s different is that the EII tool presents data to the end-user reporting tool or analytic application (and can load data if required). Think of EII as ETP – Extract, Transform and Present.
From a functional perspective, an EII environment has to do the following things:
- Connect to any source of structured data, regardless of origin
- Transform and merge information into a real-time view
- Allow the view to be manipulated using set-based logic
- Allow new data (virtual columns) to be added to the view
- Allow the view to be called by query and reporting tools, spreadsheets and office automation applications
- Support rapid BI prototyping and deployment
- Help transition views into persistent facts and dimensions in the data warehouse (ETL)
Examples:
With Enterprise Information Integration we demonstrate how traditional BI can be integrated into operational business processes, how to facilitate empowering people in operations and how to facilitate business activity monitoring (BAM) to help improve overall business responsiveness. In addition EII can support critical compliance and corporate governance requirements more effectively.
Through EII, both operational and strategic requests can be addressed much sooner than with any other traditional approach – for example supporting a marketing program of a mobile telco, that promotes additional contracts based on call usage and considering age of household members (teenager sending loads of SMS vs business person doing a lot of talking), quick analytics, short reaction times and customer satisfaction is very important and could be rapidly implemented through effective use of enterprise information integration.
Another example – generating replenishment orders to a supplier for an out of stock item. Generally this would require significant development and involve a number of technologies – but effective use of EII and the Data Warehouse could reduce the turn-around time and costs of reacting to these types of information requests, dramatically.
Meta Group says:
“META Trend: In 2003/04, as data warehouse initiatives continue to unify, enterprises will gravitate toward data integration solutions with the broadest array of data source adapters (e.g., for databases, commercial business applications, message queues, Web services, information service providers). Fuelled by a need for infrastructure/development/operations cost savings and real-time access to heterogeneous data sources, the market for data federating technology (e.g., enterprise information integration [EII]) will approach the demand for batch-oriented extract/transform/load (ETL) by 2005/06. Through 2007/08, all classes of data integration solutions (e.g., ETL, EII, EAI) will evolve to accommodate unstructured content sources. “
Current Approach:
Although EII does perhaps lean a little towards process integration the real focus is still data centric. Operational applications are implemented around distinct business processes such as sales, marketing, financial and manufacturing. In other words there are various data entry points.
The current way to address this situation is to integrate the data, typically by building a data warehouse or some form of a consolidated base of information, or as previously discussed through EII. In essence the problem is addressed at the end of the process, which often makes it costly and very time consuming.
The next wave and future trend is real-time BI. In Part 2, I will take a look at this and discuss how real-time BI fits in with the Real-Time Enterprise.