What data quality tools do you use to improve performance and maturity in your organization?

Finding the right tool in this space is actually more difficult than it should be. The big players all offer their own assessment capabilities as part of their enterprise tools. This is handy for enterprise players who want an organisation-wide solution, but not practical when you are at the start of a data quality assessment. After all, you are not ready yet to spend that amount of money before you know the state of your data and your requirements. This often drives people to either do profiling in tools like Excel or via hand-rolled queries to get the initial assessment, which can be time-consuming and difficult to scale. Understanding where we are is the first step in determining the direction in which we need to go.

Assessing data quality is a critical step in ensuring the reliability and accuracy of analytics. Identify Issues: Tools like data profiling and auditing are invaluable for spotting missing values, duplicates, and inconsistencies. It's about knowing your data's strengths and weaknesses. Prioritize Problems: Understanding the impact and severity of data issues helps in prioritizing fixes. This ensures efficient use of resources. Enhance Data Integrity: Data cleansing and validation are not just about fixing problems; they're about maintaining ongoing data integrity.

Platform is just going to solve one leg of the problem, the bigger question to be answered is how we going to resolve and build processes once we identify that bad data or missing value. There are numerous tools which can tell you about missing values, but you need to careful in selection when its comes to integration layer and alerting supports. Can the tool create ticket in Jira/ServiceNow etc? Is there a process to intimate the business users?

I would categorize this into Data observability as it refers to the ability to gain insights and understand the behavior, quality, and performance of data as it flows through a system or process. It encompasses the monitoring, tracking, and analyzing of data in real-time to ensure its reliability, accuracy, and compliance with desired standards. Data observability involves capturing and analyzing different aspects of data, including its structure, content, lineage, transformation, and dependencies. Data monitoring is a practice in which data is constantly checked for pre-defined or ML-calculated data metrics against acceptable thresholds to alert on issues.

Indeed, the importance of data quality monitoring tools in maintaining the health of your data ecosystem is essential Track Changes & Trends: Tools like data dashboards and scorecards are crucial for observing data quality over time, ensuring that any degradation is quickly identified and addressed. Measure Key Indicators: Setting and measuring indicators like accuracy, completeness, and timeliness helps in maintaining high data standards. Proactive Alerts: The ability of these tools to detect and alert on anomalies and errors is invaluable for preemptive action, rather than reactive fixes.

Tools selection tips: 1. Integration layer - across MDS 2. Alert - Does it support common platforms - Slack, Jira etc 3. Anomaly detection: Can it support ML powered instead of manual intervention? 4. Data Observability - Now a days, people are getting more demanding, does it support data observability functionality. 5. Showcase Lineage to learn the downstream impact. Imagine you got 100+ DQ errors which one you will prioritise first. I will do focus on the ones which are impacting my CFO dashboards :)

This is a loaded question. First, we the data community created a mess of point tools that increases data debt in an org, then we offer more point tools to solve that ;) IMO, the approach to simplify this mess is to unify solutions. Create value, solve data problems, and not just tools.

Implementing robust data quality governance tools is crucial for maintaining high data integrity standards across our organization. These tools help us enforce data quality policies uniformly, ensuring compliance and accountability. By defining clear roles and responsibilities, we streamline processes and enhance efficiency, while preparing for future challenges. This approach not only improves our data management but also ensures ongoing compliance with standards and regulations.

Not a tool, but a process or idea for a recurring campaign: data enrichment of customer data. This involves updating and completing customer data. In this way, companies benefit from having not only the current but also the complete data of their customers, which impacts on better planning of marketing and sales activities - these are more effective and cost-efficient, as customers are addressed in a more targeted manner and letter returns can be avoided. The return rate of undeliverable letters is relatively high, especially in the insurance and banking sectors. In addition, many banks and insurance companies do not have the digital contact points of their customers. Data enrichment campaigns improve the quality of customer data.

In my experience, opportunities for learning about data quality are currently less common compared to other fields such as data analysis. For example, there is still a lack of interactive platforms dedicated to data quality topics, although many up-to-date books on the subject are available. I believe that interactive data quality bootcamps are something we can expect in the near future.

Idea for a best practice use case: data enrichment of customer data. This involves updating and completing customer data. In this way, companies benefit from having not only the current but also the complete data of their customers, which impacts on better planning of marketing and sales activities - these are more effective and cost-efficient, as customers are addressed in a more targeted manner and letter returns can be avoided. The return rate of undeliverable letters is relatively high, especially in the insurance and banking sectors. In addition, many banks and insurance companies do not have the digital contact points of their customers. Data enrichment campaigns improve the quality of customer data.