Key Takeaways
A company has achieved data integrity when data from source systems is usable and reliable across the organization.
Maintaining data integrity through technology and policies can reduce the company’s risk for compliance issues confusion or loss of revenue.
Data integrity issues have many causes including human error, data breaches, use of multiple analysis applications and poor technology implementation.
Financial data integrity issues can be avoided with the implementation of technology like Profisee MDM, which lets data practitioners deduplicate, standardize and enrich data to create up-to-date, unique, accurate and trustworthy data that can be used across the organization.
Experienced data experts know that data integrity issues in finance can lead to regulatory, customer experience and operational issues. These problems come at an immense cost to a company’s bottom line, yet many organizations don’t have a clear plan to prevent data integrity problems and improve overall data quality. This guide provides actionable advice to building processes and systems that prevent data integrity issues and promote usable and reliable data across the organization.
Accelerate Microsoft Fabric Success
What Is Data Integrity?
A company achieves data integrity when its data is usable and reliable across the organization. This requires strategies and processes that set benchmarks for usable data and builds data integrity controls across the organization to improve data accuracy, completeness, consistency, reliability and traceability.
In finance, ensuring data integrity can prevent unnecessary fines or delays due to regulatory compliance issues related to customer or financial data. Financial data integrity can also affect business operations by creating cohesive analytics and forecasting across the organization and improving collaboration around data.
Why Is Maintaining Data Integrity Important?
Data integrity maintenance ensures that new data coming into the system adheres to the quality standards set by the organization and that all existing data is updated maintained or deleted in line with standards. These actions ensure:
- Data accuracy and consistency are maintained
- Data is easily accessible for use by the organization
- Data is available and auditable by regulatory agencies
- Analytics are reliable and useful across the organization
- Consistent and reliable financial reporting
Maintaining data integrity requires clear guidelines and consistent follow-through from the entire company to prevent potentially costly data integrity issues.
What Are Data Integrity Issues?
Data integrity issues occur when data is corrupted, breached or missing due to a variety of factors like human error, software outages, security breaches or organizational inconsistencies that cause problems with the transfer or storage of data.
What Causes Data Integrity Issues?
Data integrity issues can be caused by multiple factors that impact the ingestion, transfer and storage of data. Some data integrity issues are caused by humans, including:
- Failure to enter data correctly in software sources
- Mistakes configuring security or database software
- Overwriting new data with old data sets upon update
- Social engineering data breaches
Other data integrity issues can be caused by malware, the use of multiple analytics software solutions, poor auditing, insufficient security software and more.
7 Common Database Integrity Issues in Finance
While the causes of data integrity issues may feel endless, there are seven major database integrity issues that businesses in the financial sector should be aware of. We’ve listed these below along with an explanation of the risks associated with each issue.
1. Lack of Data Integration Across Financial Systems
When different parts of the organization use software with limited integration capabilities or completely siloed software, data can become trapped in different parts of the company and won’t contribute to the larger analytical data pool. Database integrity issues result when, for example, a sales team uses a separate CRM from the manufacturing arm of the company’s ERP and the two software systems don’t align due to limited integration capabilities.
According to Informatica 78% of IT and data teams have trouble with software orchestration that affect data quality. That means getting integrations right or optimizing financial services with master data management software can give the company an immediate advantage just by maintaining database integrity.
2. Multiple Analytics and Reporting Tools
When different parts of an organization use separate analysis and reporting tools — even when used on the same data sets — it can cause varied results. Different data processing systems and logic may result in unintended data integrity errors. Consistently posing and enforcing data integrity controls over which analytics systems are used internally can reduce the impact of analysis and reporting errors by ensuring everyone is reading reports built in a similar fashion.
Common Sources of Financial Data Discrepancies
| Tool | Issue | Affected Data | Severity |
|---|---|---|---|
| Analytics and reporting | Mismatch between new and legacy systems | Data formatting | Medium to high |
| Analytics and reporting | Different formatting from source systems | Any data skipped or dropped due to formatting issues | Medium to high |
| Analytics | Mismatch between data processing systems | All | High |
| Reporting | Mismatch between visualizations and granularity | Highly granular or specific data | Medium |
3. Manual Data Entry and Human Error
Manual data entry and human mistakes can introduce errors or discrepancies to data. While manual data entry may be the most common source of data integrity errors (due to typos and missed fields), human error can also cause catastrophic damage when systems are programmed incorrectly. For example, a data professional may incorrectly assign an older database version to be copied over the most updated version, losing data updates at a broad scale
High-risk workflows and typical vulnerability points to pay attention to include:
- Manual data entry at source systems
- User interfaces where formatting options are limited
- Data replication and backup
- Data processing
- Security vulnerabilities
- Report generation
Awareness and tight data governance can reduce the risk of manual entry errors while automated workflows and continual data quality management can improve overall data quality. Many finance companies enforce data governance with MDM to improve data integrity by ensuring data is standardized, deduplicated and unique.
4. Legacy Systems and Outdated Infrastructure
Legacy systems can cause database integrity issues, as they may not integrate or share data formatting and languages with other, more modern systems. These tools may inadvertently become siloed and cause a data gap that negatively affects the overall data integrity of the organization.
Outdated infrastructure — including hardware and software that has reached end of life or is otherwise not maintained sufficiently — can pose security risks, like data leaks and loss. Additionally, both outdated and legacy systems can slow or complicate data migration to new systems.
5. Poor Auditing and Incomplete Datasets
Similar to the problems caused by outdated systems, audit gaps and data retrieval or erasure problems could prove costly to the company. Data integrity compliance issues make gathering and reporting data to regulatory bodies like the SEC or the EU (in the case of GDPR) difficult or impossible because they reduce the traceability of customer or financial data
6. Cyber Attacks and Data Breaches
Financial institutions and departments are at high risk of cyberattacks because of the prevalence of personal identifying information (PII), like social security numbers, addresses, bank account numbers and financial records. Cyberattacks like malware man-in-the-middle attacks and code injection can manipulate financial information to perpetrate fraud, theft and even log manipulation. Expensive and embarrassing in the short term, these data integrity risks can also have long-term consequences when they undermine the reliability of organizational data.
7. Insufficient Third-Party Oversight
Partnerships are built on trust, but companies don’t get to dictate what data and security systems partners use that may affect data integrity. Even companies with the best intentions can get caught up in security risks or supply chain attacks. A policy of continued and clear communication can reduce data integrity risks.
Financial Data Risks from Third Parties and How to Monitor
| Third-Party Data Risk | Data Affected | How to Monitor |
|---|---|---|
| Data breach | PII internal customer or financial data confidential information | Discuss security controls and training with partners. Limit accessible data. Monitor for announcements and alerts for breaches. |
| Supply chain attack | Internal systems financial data confidential or proprietary information | Conduct systems survey to understand the software landscape and vulnerabilities |
| Service outages | Updated service purchase and customer information | Flag unusual behavior from systems including irregular downtimes. |
| Compliance risks | Regulated data audit logs | Regular auditing of data, monitor for anomalies and outliers that signal data integrity compliance or risks. |
| Privacy risks | Regulated and personal data (PII) | Maintain regular contact about privacy controls and security and monitor privacy alerts. |
Impact of Data Integrity Failures in Finance
Unfortunately, the impacts of data integrity problems can reach far beyond the company, impacting individuals, partner companies and customers. By training the company on how data integrity risks echo throughout the organization and with partners, finance organizations can reinforce the importance of secure and reliable data.
1. Inaccurate Financial Reporting and Analysis
Inaccurate financial reports or analyses are inconvenient and even embarrassing when released internally. They may also result in fines, restatement and lost trust when released publicly. According to Gartner, poor data quality results in a $12.9M average annual loss a huge — and hugely preventable — financial burden for companies.
2. Regulatory Penalties and Compliance Issues
Regulatory issues go beyond reputation, however, causing companies to pay fines and accept penalties for failing to maintain data integrity compliance. These penalties are expensive, and the decreased trust from highly-publicized repercussions can have lasting consequences.
For example:
- Canadian regulators fined TD Bank $6.7 million for failing to comply with anti-money laundering regulations
- Citibank was fined $136 million by US regulators for data integrity issues lasting years
Companies that streamline KYC and AML compliance can avoid many of these penalties by implementing one technology that surfaces relevant data.
3. Financial Losses and Business Disruption
In the case of publicly traded financial institutions, data integrity errors that cause incorrect forecasts may directly affect shareholder trust and reduce the strength of the company in the stock market. Operational costs for corrections audits and system replacements can further damage the company’s reputation and capabilities, leading to reduced revenue lost time and stalled innovation.
4. Reputational Damage
A company’s reputation with investors and stakeholders can quickly erode with the downstream effects of data problems. Data integrity risks from third parties such as malware or ransomware take time and effort to recover from, diverting operational resources from growth and innovation to recovery efforts. Investors and shareholders may lose confidence in the company’s ability to recover, fear that data integrity is at further risk or even reduce investments of time or money in the company’s future.
5. Supply Chain and Operational Risks
Just as poor data quality can affect a company’s reputation with investors, it can also significantly affect the working relationship with partners and internal stakeholders. Consider:
- Vendor relationships can be damaged due to inconsistent purchasing and communication.
- Fulfillment problems can arise due to data integrity issues disrupting relationships with downstream partners.
- Employees can become disillusioned and frustrated with database integrity issues that impede their workday.
Core Components of Strong Financial Data Integrity
Strong financial data integrity comes from a combination of data integrity best practices and data integrity controls that build routines and processes throughout the organization that keep new and existing data organized and reliable. Practicing these five components will help you build and maintain robust financial data integrity.
1. Data Validation and Verification Techniques
Start with a strong foundation of core validation steps that define and build databases of quality and checked data.
- Define requirements: Outline formats and circumstances required to define data as valid.
- Data collection: Consolidate all data and attributes that are needed from any relevant sources.
- Data cleansing: Correct mistakes, deduplicate and check formatting across all data.
- Choose, write and implement validation techniques: Look at the compiled and corrected data and decide on the most appropriate rules and logic to validate data. Write and implement the rules within your data sets.
- Correct errors: When errors are found, they should be corrected. The reasons for the error and proper formats should be documented and disseminated among the team as teaching opportunities.
- Verify results: Double check that the rules and checks have provided the correct results and make adjustments as needed.
- Document, monitor and improve: Write down the processes and procedures for validation and make those available to data stewards and stakeholders. Complete periodic checks to maintain data quality, and tweak rules as necessary to improve data integrity.
Following validation, perform cross-system reconciliation by populating source systems with the validated data. This will reinforce the validation rules that need to be followed in the source systems and reduce the need for wholesale cleansing in the future.
In addition, be mindful of referential integrity by ensuring that all secondary data attributes tie back to the primary data. For example, a customer address should refer back to the unique customer identifier instead of existing without a customer ID.
2. Access and Authorization Controls
Implement role-based access control (RBAC) and least-privilege approaches across the finance organization. In addition to good cybersecurity practices, these policies help limit the type and data any single employee can access to only that which is important to their role.
RBAC defines access based on the employee’s role, allowing access to secure PII or financial information relevant to an employee’s role or job function. Least-privilege approaches are data integrity controls that give every employee throughout the organization access to the minimum amount of information necessary to perform the task at hand, requiring oversight or permissions to access data of higher sensitivity.
Sample Financial User Roles and Access Privileges
| Role | Access Privileges | Example of Data Available |
|---|---|---|
| Customer Service | Can view data necessary to serve customer needs | Name, address and account numbers. View-only access. |
| Accounts Payable | Can view transaction and account data necessary to pay invoices | Vendor name and address. Invoice totals and details. Can initiate transfers. |
| Administrative Roles | Complete view and edit access for the system | All PII transfer account and balance information. Highest access. |
| Data Professional | Can view, edit and maintain APIs, reports and other functional parts of the system but not PII | API keys, database samples, reports and analysis tools. |
3. Encryption Masking and Secure Data Handling
Implement end-to-end encryption throughout the company’s systems to reduce the impact of malware, ransomware or man-in-the-middle cyber attacks and lower overall data integrity risk. This means introducing encryption throughout source and target systems as well using encryption for data in transit between systems. Encryption effectively makes any data leaked to someone without the key useless, protecting data integrity even if it is breached.
Additionally, implement secure data handling policies throughout the organization to help reduce data integrity errors. Secure data handling policies may include:
- Barring employees from emailing or transferring all or part of customer data information to outside systems
- Requiring secure logins for all systems and implementing multifactor authentication (MFA) for logins
- Implementing encryption and RBAC
- Communicating and enforcing third-party security standards
4. Comprehensive Audit Trails and Logging
Building comprehensive audit trails and logging practices within the company help not only with compliance and traceability, but they also improve procedure transfers and training. Many companies implement the ALCOA++ model for data compliance. This model was originally developed for pharmaceutical companies, but it can be transferred to many industries looking to implement data integrity compliance practices, including finance. ALCOA++ stands for:
- Attributable
- Legible
- Contemporaneous
- Original
- Accurate
- Complete
- Consistent
- Enduring
- Traceable
- Available
- Secure
Ensuring that practices are in place to maintain the ALCOA++ level of data reliability will allow for compliance and risk management through audit logs and traceability that improves regulatory compliance.
5. Regular Data Backups and Disaster Recovery Planning
Some of the most devastating database integrity issues are caused by human error that overwrites or deletes data or physical downtime due to disasters. Whether it’s a flood that takes out on-premises databases or an intern who accidentally deletes a database, these incidents can lead to loss of valuable time and revenue. Regular multi-site backups and version control in addition to clearly documented disaster recovery procedures can prevent or largely mitigate mistakes or downtime.
How to Resolve Data Integrity Issues
Even with the best practices and processes in place, data integrity issues will arise. Mitigating and documenting the issues quickly can restore data integrity and confidence within the company. Build an internal response playbook for diagnosing, correcting and preventing recurrence of integrity problems based on these steps in concert with the proactive controls listed above.
1. Identify and Classify the Integrity Issue
Use data quality dashboards, compliance alerts and analysis of reconciliation errors to identify and classify integrity issues throughout the organization. Monitoring and automated alerts can be used to quickly identify problems, helping financial teams restore data integrity within urgent contexts, like reporting deadlines and audits.
2. Trace Root Causes Across Systems and Processes
Once the team has identified and classified the integrity issues at hand, work backward to understand the root causes within the systems or processes. For example, was there an exception in the RBAC that allowed an employee to make unauthorized changes? Other causes might include ETL mapping errors, system corruption or broken API connections and integrations.
3. Correct Errors and Restore Trustworthy Data
Once the root cause has been identified, the team will need to correct errors. Often, investigation of root causes will reveal the extent of the errors and the team will need to isolate the areas for remediation. For extreme cases, rollback to backup versions of the database may be necessary.
In other cases, running data cleansing on the affected systems and re-reconciling data across source systems may be enough. At this point, consider whether the data integrity remediation techniques will fix the integrity issues permanently or whether validation rule changes or process improvements are needed to prevent future issues.
4. Avoid Recurrence with Preventive Measures
If more permanent measures need to be implemented, go back to the validation rules, RBAC and logging practices the team implemented and update them accordingly. This may require another round of data validation and cleansing to double check the integrity of the data sets. In the case of RBAC and logging, communication with stakeholders and employees should take place to ensure everyone understands and follows the new data integrity remediation procedures.
7 Data Integrity Best Practices and Strategies for Risk Prevention
Building and maintaining data integrity is a company-wide affair. These strategies cover governance, process, people and technology for data risk prevention in finance.
1. Implement Master Data Management
Master data management (MDM) is the backbone for maintaining a single source of truth among software across the organization. MDM centralizes data from finance, ERP, CRM, compliance and other departmental systems that manage financial data. Some MDM solutions like Profisee enforce governance rules from your data governance tool, in addition to master data management for financial services, which ensures centralized access to data integrity controls while providing reliable, quality data.
2. Establish Data Governance Frameworks
Use data governance to ensure data quality by implementing frameworks of people, policies and technology that use data integrity best practices and promote behaviors that result in usable and reliable data. Define roles and responsibilities for data integrity best practices and data governance strategy, including:
- Data owners: Employees who are ultimately responsible for the quality and use of data across the organization. These employees are the accountable decision makers for data processes and data governance frameworks. Job titles include: CFO, CTO, IT manager and Data manager.
- Data stewards: Employees with the day-to-day responsibility of maintaining the quality integrity and security of data. Job titles include: financial analyst, comptroller and accountant.
- Data custodians: Employees that manage the security and integrity of systems that store and maintain data. Job titles include: security administrator, cybersecurity professional.
3. Train Employees on Data Integrity and Compliance
Build data integrity compliance into the day-to-day operations of everyone in the company and emphasize anti-fraud and safe-handling training for all financial staff. Practices like dual authentication and regular audits keep employees accountable to data integrity. Technology like AI fraud detection and transaction monitoring can passively monitor employee and customer data to ensure compliance.
4. Automate Monitoring and Analytics for Financial Data
Automated monitoring through Security Information and Event Management (SIEM) and reconciliation analysis can provide real-time detection of fraud or anomalous behavior in financial data. SIEM is a process of collecting and analyzing log and event data for outlier detection across an entire organization which can alert the team of third-party bad actor intrusion or unauthorized internal behaviors. Reconciliation analytics can flag outlier behaviors which may indicate fraud security risks or data integrity risks so the team can respond in real-time without actively monitoring every transaction.
5. Maintain Continuous Compliance with Regulatory Standards
Schedule review timelines for regulatory compliance including SOX, GDPR and CCPA compliance checks. Consider building a data integrity compliance visual timeline that details the data required for each regulatory body, due dates, which employees are responsible for gathering requirements and any potential complications or risks.
6. Assess and Manage Vendor Data Integrity Risks
Communicate with vendors to assess and manage data integrity risk before it intrudes on internal data. Follow due diligence guidelines for understanding a partner’s data integrity before connecting systems, and implement contractual controls and audits to enforce compliance. If broaching the subject for the first time, communicate the value of these controls for both parties and give examples of how data integrity can improve operations for all involved.
7. Enforce Change Control and Incident Response Procedures
Policies without enforcement become no more than suggestions. Document all processes and require change logs to track any changes so administrators can quickly roll back changes if needed. For highly sensitive processes involving PII and other secure information, implement processes that require authorization for any changes meaning another employee or leader must approve any changes before implementation.
When incidents inevitably happen, follow these steps to respond:
- Investigate the root cause to identify the issue
- Investigate the scope to understand what data was affected
- Contain the affected data to prevent further harm to systems
- Remediate the affected data
- Update policies and procedures to prevent similar incidents in the future
- Document the incident remediation and policy changes
Once the incident is contained and corrected, publish an incident response memo to all involved parties and stakeholders to explain the issue, its cause, the remediation that took place and any changes to policy that resulted.
Overcome Financial Data Integrity Challenges with Profisee
Profisee’s master data management platform helps finance companies and departments maintain database integrity with automated features that make data cleansing easy and enterprise data governance simple to implement. MDM and data governance work together to maintain data integrity by default, ensuring reliable and usable data populates back to source systems and enabling compliance through clear logs and change histories. Data integrity doesn’t have to be difficult. Profisee can help you combine integrity best practices with governance to build usable golden record data that moves your company forward.
Request a demo to see how Profisee can help you with maintaining database integrity.
Frequently Asked Questions
What are the four types of data integrity?
The four types of data integrity are
- Entity integrity: Each record is unique and identifiable, usually with a primary key like a unique identifying number.
- Referential integrity: Each record’s integrity is maintained across tables and refers back to the primary key.
- Domain integrity: All data within a domain exists in the same valid type or format. For example, customer names should be listed as [firstname lastname] instead of [lastname firstname]
- User-defined integrity: Rules defined by users to enforce business rules. These are unique to each business.
What are the five principles of data integrity?
The five principles of data integrity make up the ALCOA acronym and are:
- Attributable
- Legible
- Contemporaneous
- Original
- Accurate
Many organizations use the ALCOA+ or ALCOA++ principles, which add these requirements:
- Complete
- Consistent
- Enduring
- Traceable
- Available
- Secure
What is a data integrity issue example?
An example of a data integrity issue is when, in the course of daily work, a database administrator overwrites an up-to-date database table with a backup table from several weeks before. If it goes unnoticed, this action could affect accounting orders or customer service requirements.
Other examples of data integrity issues include:
- Unauthorized formatting changes that cause discrepancies between software
- Mismatched rounding requirements between accounting systems
- Siloed data within a single system not included in the central database
What is the cost of poor data integrity?
According to 2020 figures from Gartner, data integrity issues cost US businesses an average of $12.9 million annually.
Data integrity vs. data quality: Are they the same?
No. Data integrity is how accurate and consistent data is throughout its lifecycle, including within source systems, centralized data repositories, analysis tools and upon deletion. Data quality defines how usable data is in any situation. While a company may have defined practices for data integrity, if those practices are not updated upon the introduction of a new system, that data may not be usable in the new system and therefore would not be considered quality.
Data integrity vs. data security: What are the differences?
Data integrity speaks to how accurate and consistent data is maintained throughout its lifecycle. Data security refers to the systems and procedures that ensure data is secured from unauthorized use access and modification. Data security can include practices such as RBAC, multi-factor authentication and change control procedures as well as technologies like SIEM, malware detection and AI-controlled threat detection.
How do data integrity failures expose companies to regulatory risks?
Data integrity failures expose companies to regulatory risks, including penalties and work stoppages when:
- Audit logs are incomplete
- Data breaches go undetected or unreported
- Customer information cannot be deleted
- Anti-money laundering regulations are not upheld
There are many other reasons that a lack of data integrity compliance can expose companies to regulatory compliance risks due to uncontrolled or incomplete data.
How do you maintain data integrity?
Companies can maintain data integrity with a combination of policies and technology. Policies for data validation data security and data governance ensure that data from source systems is usable and reliable. Technologies like MDM, SIEM and data monitoring systems can ensure that data integrity continues and alerts data stakeholders to changes that affect integrity.
Share
