Confused about EDW, DW, Data Lakes, Hadoop, HTAP, Hive, LLAP, Spark SQL… (…you can complete the list ! )? Are you wondering … To ETL or not, to ELT or not, or to do no ETL/ELT at all?
A plethora of terminologies, acronyms and technologies are currently in the mix. For the un-inaugurated and the professionals alike it has become a minefield, open to misinterpretation AND misrepresentation!
Authorities on the subject of Data Analytics are beginning to agree: there is no one size fits all.
Safe-routes are chosen
Amidst several new entrants to the market and during the incubation (transition period to modern data warehousing), everyone in the know (including those who think they know!) is taking the approach of combining only certain portions of both old and new methods. Proposing all of Business’ data problems will be solved. What can go wrong? It is a safe route and makes sense while testing the waters.
Myths, Fables and Falsehoods
Unbeknown to the Business User, acquiring more technologies simply raise the levels of the already complex Analytical / Business Intelligence / Data Warehouse / Data Integration landscape. We are enticed into falsehoods of: ease of use, data will be prepared in days rather than weeks and self-service solutions. Hurray, no need for an IT department!
Myths such as “Detailed data is not available”, “It takes too long to load data”, “Too much effort to incorporate new data” and so forth is being advocated as reasons to divorce anything traditional and to dive headlong into new lakes in the cloud. Ask pertinently how data is categorized, defined, cleansed, time-lined only then to be advised of having to incorporate existing strategies to address these traditional and still valid challenges.
Users helping themselves or growing a “help-yourself-from-deep-seas-of-data” culture, has long been a fable. Observe what happens when Business Users are dunked into this environment. It is the equivalent of setting off in a rowing boat without any navigation system, trying to cross a stormy sea towards an unknown destination. A dangerous endeavor, destined to fail with disastrous consequences.
Pragmatism and Flexibility
To embrace new techniques and technologies necessitates a deep understanding of the information requirements which are aligned with the Business’s short, medium and long term strategies. Before dashing for the dump-all-data-into-one-lake idea and throwing the baby out with the bath water, consult the road-map that is currently directing your Analytical Information traffic. If it does not exist it is unlikely that new software will do much in navigating the best route to answer the ever elusive business questions.
Choosing a solution well suited for your organisation should be driven by the motto of “Pragmatism and Flexibility” and the mantra must always be “Service the Business in the most effective way”.
Even though we may believe that data modelling and/or ETL has fallen by the way side, truly successful Analytics projects will continue to demand interaction and co-operation between Business and IT.
Understanding which data elements are in support of business strategies and collecting appropriate data while making use of tried and tested methods alongside implementing new technologies, demands thorough preparation, planning and implementation guidelines.
Although there is no one size fits all, fitting your business into a right-sized outfit maybe easier than perceived. Steadfast direction remains the order of the day.
To learn how to complement and extend the lifespan of your EDW, DW and to discuss the roles of Data Lakes, Hadoop, HTAP, Hive, LLAP, Spark SQL and whether or not to ETL / ELT …do attend the ever popular updated technology independent course Concepts, Design and Modelling for Extended Data Warehousing that teaches sound principles to be applied no matter what technology is used. Register HERE