Ten Mistakes To Avoid In Data Quality Management

In TDWI’s Best of Business Intelligence, Volume 5, they have a chapter on ten mistakes to avoid in Data Quality Management written by Arkady Maydanchik.  I will try to summarize the ten points. I strongly advise you to download the report

1.  Inadequate staffing of Data Quality teams

“The Data Quality teams must include both IT specialists and business users.”

2.  Hoping that Data will get better by itself

“One fundamental misconception is that data quality will improve by itself as a result of general IT advancements”

3.  Lack of Data Quality Assessment

“Typically, organizations underestimate of overestimate the quality of their data, and they rarely understand the impact of data quality on business projects. Assessment is the cornerstone of any Data Quality management program. It helps describe the stat of the data and advances understanding of how well the data supports various processes. Assessment also helps the business how much the data problems are costing it”

On a personal note.  We have performed many tests on customer and production databases to help businesses asses the temperature on their databases.

4.  Narrow Focus

“Data Quality has continually detoriated in the areas of HR, finance, product orders and sales, loans and accounts, patients and students, and myriad of other categories.

The main reason we fail adequately manage to quality in these categories is that this data’s structure is far more complex and does not allow for  a “one size fits all” solution

5.  Bad Metadata

“The greatest challenge in Data Quality management is that the actual content and structure of teh data are rarely understood”

6.  Ignoring Data Quality during Data conversions

“The quality of the data after conversion is directly proportional to the amount of time spent to analyze and profile the data and  uncover the true data content. Unfortunately, the common practice is to convert first and deal with the quality later”

7.  Winner-Loser Approach in Data Consolidation

“The correct approach to data consolidation is to view it in a similar light as data cleansing. We select one of the data sources as the primary data source for each data element, design a comprehensive set of tests comparing the data against other sources, and then use the additional data for “data cleansing”. Once the data in the primary data source is correct, we convert it to the target database”

8.  Ineadquate monitoring of Data interfaces

“The solution to inferface monitoring is to design programs operating between the source and target databases”

9.  Forgetting about Data decay

“In the age of numerous interfaces across systems, we have come to rely on changes made in one database to be propogated to all other databases. However, there are times when it does not happen. For instance, interfaces may ignore retroactive data corrections, or IT personnel may make changes using backdoor update query – which of course does not trigger any transactions in the downstream systems.

10. Poor organization of Data Quality Metadata

“Data Quality assessment projects routinely generate innumerable unstructured error reports with no effective way of summarizing and analyzing the findings. As a result, the value of the Data Quality initiatives is greatly diminished. In the worst cases, the projects are abondoned.

This is just a summary. I strongly urge you to download the full report.  You can also mail
me at jei@omikron.net, and I will send it to you.