All too often, a great IT concept gets knocked for not producing the benefits and results it initially promised. Projects are started with great expectations of improving business processes and decision-making, as well as revenues and profitability, only to end in tatters after swallowing large sums of money and time with no results visible. This scenario has become common in the business intelligence market, where companies have been known to spend millions with no measurable benefits.
"Getting involved parties to stop talking about technology and to talk about the business needs is critical."
Designing and developing a business intelligence solution, underpinned by a data warehouse, is a complex process that can easily cause more problems than it solves if not done correctly.
If we compare successful implementations to failed business intelligence projects, the success differentiators take us back to the basics of business intelligence.
Firstly, business users must define exactly what the organisations goals and objective are and what it wants to measure to achieve those objectives. Secondly, these goals and measures must be clearly communicated to the implementation team, which should have to expertise to turn business requests into an easy-to-use, information supplying system.
The data warehousing team must do a thorough analysis of what is wanted in what time from what resources before starting the development process. The key to success is to get users involved from the start and to limit the initial scope of the project to ensure measurable milestones. Securing executive buy-in is also critical. Given the expense of even the smallest data mart in both man-hour and purchased resources, the project should be championed from the highest executive level.
Once secured, the next step is to use the executives to determine what the company's business objectives are and how the data warehouse will help to achieve them. If management cannot name the company's objectives, the project becomes a no-go. Business intelligence and data warehousing are tools for reaching business objectives, not creating them.
Defining milestones along the way
Once the broad parameters of the application are identified (deciding that an application should help reduce customer losses, for example), the "how to" must be addressed. In order to define best what the application should do, developers should get feedback from representatives across the enterprise. While vital, executive opinions are only a fraction of the business picture, or so end users say. Equally important is an intimate understanding of the needs and work habits of the users who will be using the system. Getting involved parties to stop talking about technology and to talk about the business needs is critical and will result in the development team getting the best idea of what is required.
The business user must come to a consensus of the top five business goals the application must accomplish first. This way, the developers can proceed with the project confident that they have full support and know what people need. The participants also become advocates of the project from the start, feeling that they contributed to it, and then gain a better understanding of the needs of other departments.
This simplifies the task of identifying and prioritising the various phases and timeframes of the project – taking stock of what skills are available in the company and what supplies are needed, and defining budget and timeline.
Identify source systems
Once the measures of the application have been defined, the next step is to identify where the data will come from and if the data even exists at all. For instance, if the company has decided it wants to quantify customer profitability. The sales managers and other members of the brainstorming team have determined that profitability includes length of time a customer has been patronising the company, how many calls to customer service the customer has made, and how much money the customer spends annually.
The warehouse developer tries to identify the source systems for this information and discovers that some of it is not being captured in any system. In such a case, the project has reached a dead-end before it has started.
Along with identifying which data to use and where the data is stored, the organisation must also decide how much data to use. One of the biggest challenges in developing a data warehouse is determining the value of the data you keep. Wrestling with just the sheer quantity of data kills your productivity. The successful warehouse will be the result of a balancing act – volumes vs. value of the data.
Decide on an architecture
With the application defined and the data sources confirmed, the next decision is between a fully-fledged warehouse or a data mart. The generally accepted practice is to develop marts first and then tie them together to make a warehouse. This allows developers to build applications incrementally, leveraging the knowledge, experience, and, in some cases, spare hardware capacity in existing systems.
Another decision is whether or not to use a pre-packaged "quick-start" solution or to completely custom-develop the warehouse from scratch. While the price for quick-start solutions may be tempting, this type of programme can be misleading for a number of reasons:
- They are typically targeted only at marts. If a broader mart or warehouse solution is intended, the architecture may require so much fine-tuning that developers must essentially build from scratch anyway.
- The low price tag may come with a very small user or CPU license.
The areas which typically require the most time, energy and effort in a warehouse are the data extraction and cleansing processes, areas in which quick-start programs typically do little or nothing to help.
Once these basic decisions have been made and the appropriate staff members recruited to support the data warehouse process, the task of handling data – from original source to the final warehouse and business intelligence functionality – needs to be defined.
The next Survival Strategies column will deal with the process of turning data into useful information.