Software supermarkets or specialized vendors?

[tweetmeme source=”jeric40” http:// optimizeyourdataquality.wordpress.com]

Some customers are worried about the complexity of the data quality challenges. Understandable, since they can hear about a lot of projects that has failed. There are several reasons for this. An important factor is using right tool. If you want to get a nut to loosen from a bolt, you might use several tools.

Some use pliers, almost a sure failure.  Others use an adjustable wrench.  This might work, but if it is not tight enough, there is a big possibility that the nut will be rounded and suddenly will become very hard to get off. It will create a lot of extra work, and the bolt cannot be reused.  If you use the correct wrench, the nut will come off, and it can be reused.

Let me transfer this to the Data Quality world.  Way back I was approached by a prospect.  They had a MDM project had hit the wall.  They had spent over a year so far.  They had used all the products from the Microsoft Enterprise Supermarket, and built some elaborate business rules together with external consultants.  In this way they had found about 23% of duplicates.  Their problem was that they could see there were more, but they could not catch them.  They sent the data over, and a couple of days later we could send the results back, with an additional 27% of duplicates on the cleansed data.  This shows me the power of specialized tools.

This is the result from Google when I searched for Data Quality Tools

There are some great software supermarkets out there.  These often offer excellent and good products.  Customers often want as few vendors as possible, and one stop shopping if possible.  You will get special competence on this vendor’s product, and can be cost efficient.  One challenge with this strategy is that you might miss out on a special product that can be critical for your success. An example: Gartner estimates that 50% of CRM/ERP installations fail, due to poor data quality and integration.

Data might be the most important asset in your company; don’t you want the best available product to handle this precious asset?  If I had the king over for dinner, and I knew that it was a 50% chance of failure if I used the meat from the supermarket,  I would definitely go to the best butcher in town.  90% of the purchase would be in the Supermarket and the lat 10 from specialized vendor.  One thing does not exclude the other.

These challenges is not only in the Data Quality field, but is as much present in the e-Commerce world.

“Standardizing and normalizing disparate customer data is akin to having root canal surgery at the dentist.”

I found another interesting article on SearchDataManagement.com about why Data Quality is elusive at most organizations.

Most interesting points according to Gartner:

• Half of the companies have actually deployed data quality tools or started Data Quality initiatives.
• Of those who use data quality tools, less than one third has deployed these enterprise-wide.

The reason is that the information is siloed in different databases collected from different sources, with no tools to connect them together.  Or as Leslie Ament, Managing Partner, Hypatia Research says:

“Many larger retailers have upwards of 10 different databases with different schema for collecting customer data,” Ament said. “Standardizing and normalizing this information is akin to having root canal surgery at the dentist.”

I like analogies like that, hopefully we can make customers address the challenges, even though it hurts.  The good thing is that I think we can see more business is becoming increasingly aware of these challenges.

Read the whole article here.

Is Data Quality as Sexy as Intestinal Medicine?

This weekend I was to a dinner with the Queens Heart Surgeon.  We talked about the hierarchy of surgeons, where he is at the top, Brain Surgeons is at 2nd and Intestinal Surgeons are at the bottom.  He is at the top because he literally holds life and death in his hands, whereas Intestinal Surgeons just has a shitty job.  Even though it’s unpleasant for the patients, they can learn to live with it, and it’s seldom lethal. 

After being to some Conferences and Meetings about BI and MDM, I ask myself:  Isn’t this how it is presumed to be in the IT world?  The big BI and DW vendors and integrators struts their charts and reports and show how important they are to the businesses and imposes as Heart Surgeons.  They claim that Data Quality is something you can take on the fly and want to project it as Intestinal Medicine.

Shouldn’t it be the other way around?  Gartner predicts that 50% of DW and CRM projects will fail, and the single most important reason is poor Data Quality.   It seems like it is the Data Quality Operators that are holding the life or death of these projects in their hands!

So from now on Data Quality Operators are compared to Heart Surgeons, and OK, I’ll give it to the big BI and DW vendors and integrators; they can be Brain Surgeons, a good 2nd in the hierarchy. 🙂