Combining Similar Data from Multiple Data Sources

Organizations moving their data and applications to the Azure cloud often start with their most business-critical data — what we refer to as master data.

Master data can be considered the “nouns” of your business and can include customers, products, suppliers, locations and more — essentially any core data assets that organizations use across the enterprise.

But companies starting their cloud-migration journey often find they have similar data stored across several disparate source systems and databases.

For example, a large financial institution may have a partial record for a single customer in their Dynamics 365 CRM that contains transaction information and recent marketing campaigns that customers have interacted with — and another record for that same customer in their financial-reporting system with pricing terms, credit scoring and accounting information.

Or a manufacturer may have business data stored across several ERPs due to a half-dozen recent acquisitions and have inconsistent customer IDs, payment terms and transaction history among them.

It is clear how these data, or IT-related, issues can quickly grow into critical business failures.

Perhaps a customer approved for a $10,000 line of credit can successfully purchase $50,000 worth of goods if she orders from different subsidiaries. Or maybe a checking account customer who has opted out of marketing emails will still receive them from the credit card department because they track their CAN-SPAM and GDPR opt-outs in separate systems.

These are just a few examples of how duplicate, inconsistent and incomplete data can inhibit the value of an investment in Azure, causing leaders to doubt the effectiveness of their IT initiatives and potentially reduce investment in future projects.

Working with Low-quality, Inconsistently Formatted Data

Beyond having duplicate records of similar data types, another symptom of data quality issues in Azure is the inability to compare and use data side-by-side because of inconsistent formatting or completeness.

Symptoms of low-quality data include a failure to meet data-quality rules — or perhaps a lack of data-validation rules at all.

Data is of high quality when:

1) It is fit for the intended purpose and use.                      

2) The data correctly represents the real-world construct that the data describes.

But these rules may contradict one another. For example, a customer record in an ERP system may be sufficient for issuing and receiving payment but otherwise lacks critical information required to provide timely customer service.

This highlights the importance of the customer master record, the comprehensive record of a customer with all the relevant attributes, hierarchical information and lineage required to provide a seamless shopping experience, provide responsive customer service and receive timely payments.

Low-quality data users often complain about constant “data clean-up,” or having to manually match, merge, correct and compare records from multiple systems. Whether manually manipulating data in spreadsheets or consolidating records into a reporting or business intelligence (BI) tool like Power BI, this fails to correct the issue at its core as it is incredibly difficult — or impossible — to maintain over time.

Data Consumers Question Analytics, Reports and Recommendations

The unfortunate result of poor-quality data throughout the organization is that IT and business users alike continually question the analytics, reports and recommendations derived from enterprise data.

Incremental revenue may not be attributed to right product or suite of products. Leaders cannot identify poorly performing sales regions. The marketing team does not know who has opted out of email communications or who has requested a copy of their data under GDPR.

These issues often compound and find their way to business operations and customer interactions.

The supply chain team cannot automate procurement or delivery processes without reliable location information. The marketing team overspends, and annoy theirs prospects, by sending the same material more than once to the same person — with the name and address spelled a bit differently.

End users are unable to run simple reports about the state of the business without continuous data clean-up, much less take advantage of predictive analytics or opportunities identified by the BI team.

This can create a toxic data culture and stymie innovation. When no one can agree on a single source of the truth, collaboration diminishes and organizations suffer.