Susan Andre is MD of Sagent SA, the e-analytics company. With extensive experience as a business consultant and business intelligence and data warehousing practitioner, she comments on the increasing storage challenges of the digital economy.
Whereas the information age has spawned boundless volumes of data, the knowledge management movement has developed the appreciation of how important it is for information to be current, accessible and useful. The concurrent revolution in software technologies has resulted in ubiquitous data generation and enabled masses of data to be stored and distributed.
There is no doubt that harnessing the intelligence lying latent in our data is becoming a competitive necessity. Databases of the future will contain all types of artefacts, including models, source code, binary objects and documents, all in enormous volumes. Harnessing these boundless resources through creative storage management, enables the agile and innovative organisation, providing the competitive edge.
The storage of data for intelligent retrieval means developing cohesive models that fulfil specialised information requests – for example multi-dimensional and hierarchical processing requires specialised databases, while data warehouses perform optimally when stored in special format. This design is defined by user needs and technical requirements, with the desired end being efficient storage and fast retrieval.
Types of data, such as images and sounds are now a norm. The rapid adoption of Web design techniques has made these commonplace and these sorts of objects will be demanded more in the future within operational and business intelligence applications. Attention to the storage mechanisms that allow the linking of these and other objects to appropriate references is essential if they are not to be lost to intelligent access and use.
Storage management has emerged as the key issue in today's exploding data environment. Manual methods of control mean demanding and immediate decision-making about what should be stored and where and when this should happen. Retrieval is further complicated with slow request procedures and the management of archived material is a full-time project on its own.
Multi-terabyte sites are not uncommon, forcing IS managers to look beyond the traditional methods of storing and securing critical corporate information. Advances in storage hardware and software have made solutions such as hierarchical storage management (HSM), Storage Area Networks (SAN), archiving and disk grooming functions that are designed to optimise the investment in storage hardware, while offering the sort of security necessary to protect valuable corporate information.
Tape libraries are now equipped with tape drives capable of storing dozens of gigabytes of data on a single cartridge. Advances in drive technology, robotics and software have so greatly improved access to information in these devices that tape libraries are increasingly being used to store active user data instead of only the traditional backup and archival applications.
Developing storage solutions that integrate seamlessly, work within existing database and platform infrastructures, offer scalability for every size company and handle diverse objects is undeniably important for the optimal use and benefit of enterprise data.