Democratizing Data Quality Through a Centralized Platform

•

1 like•1,453 views

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

What's hot

Delta lake and the delta architecture

Adam Doyle

- Delta Lake is an open source project that provides ACID transactions, schema enforcement, and time travel capabilities to data stored in data lakes such as S3 and ADLS. - It allows building a "Lakehouse" architecture where the same data can be used for both batch and streaming analytics. - Key features include ACID transactions, scalable metadata handling, time travel to view past data states, schema enforcement, schema evolution, and change data capture for streaming inserts, updates and deletes.

Databricks Delta Lake and Its Benefits

Databricks

Delta Lake, an open-source innovations which brings new capabilities for transactions, version control and indexing your data lakes. We uncover how Delta Lake benefits and why it matters to you. Through this session, we showcase some of its benefits and how they can improve your modern data engineering pipelines. Delta lake provides snapshot isolation which helps concurrent read/write operations and enables efficient insert, update, deletes, and rollback capabilities. It allows background file optimization through compaction and z-order partitioning achieving better performance improvements. In this presentation, we will learn the Delta Lake benefits and how it solves common data lake challenges, and most importantly new Delta Time Travel capability.

3D: DBT using Databricks and Delta

Databricks

Data Build Tool (DBT) is an open source technology to set up your data lake using best practices from software engineering. This SQL first technology is a great marriage between Databricks and Delta. This allows you to maintain high quality data and documentation during the entire datalake life-cycle. In this talk I’ll do an introduction into DBT, and show how we can leverage Databricks to do the actual heavy lifting. Next, I’ll present how DBT supports Delta to enable upserting using SQL. Finally, we show how we integrate DBT+Databricks into the Azure cloud. Finally we show how we emit the pipeline metrics to Azure monitor to make sure that you have observability over your pipeline.

The Heart of the Data Mesh Beats in Real-Time with Apache Kafka

Kai Wähner

If there were a buzzword of the hour, it would certainly be "data mesh"! This new architectural paradigm unlocks analytic data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios. As such, the data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a data mesh infrastructure must be real-time, decoupled, reliable, and scalable. This presentation explores how Apache Kafka, as an open and scalable decentralized real-time platform, can be the basis of a data mesh infrastructure and - complemented by many other data platforms like a data warehouse, data lake, and lakehouse - solve real business problems. There is no silver bullet or single technology/product/cloud service for implementing a data mesh. The key outcome of a data mesh architecture is the ability to build data products; with the right tool for the job. A good data mesh combines data streaming technology like Apache Kafka or Confluent Cloud with cloud-native data warehouse and data lake architectures from Snowflake, Databricks, Google BigQuery, et al.

Differentiate Big Data vs Data Warehouse use cases for a cloud solution

James Serra

It can be quite challenging keeping up with the frequent updates to the Microsoft products and understanding all their use cases and how all the products fit together. In this session we will differentiate the use cases for each of the Microsoft services, explaining and demonstrating what is good and what isn't, in order for you to position, design and deliver the proper adoption use cases for each with your customers. We will cover a wide range of products such as Databricks, SQL Data Warehouse, HDInsight, Azure Data Lake Analytics, Azure Data Lake Store, Blob storage, and AAS as well as high-level concepts such as when to use a data lake. We will also review the most common reference architectures (“patterns”) witnessed in customer adoption.

Learn to Use Databricks for Data Science

Databricks

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Building Lakehouses on Delta Lake with SQL Analytics Primer

Databricks

Time to Talk about Data Mesh

LibbySchulze

This document discusses data mesh, a distributed data management approach for microservices. It outlines the challenges of implementing microservice architecture including data decoupling, sharing data across domains, and data consistency. It then introduces data mesh as a solution, describing how to build the necessary infrastructure using technologies like Kubernetes and YAML to quickly deploy data pipelines and provision data across services and applications in a distributed manner. The document provides examples of how data mesh can be used to improve legacy system integration, batch processing efficiency, multi-source data aggregation, and cross-cloud/environment integration.

What’s New with Databricks Machine Learning

Databricks

Data Lakehouse, Data Mesh, and Data Fabric (r2)

James Serra

So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I'll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I'll discuss Microsoft version of the data mesh.

Considerations for Data Access in the Lakehouse

Databricks

Organizations are increasingly exploring lakehouse architectures with Databricks to combine the best of data lakes and data warehouses. Databricks SQL Analytics introduces new innovation on the “house” to deliver data warehousing performance with the flexibility of data lakes. The lakehouse supports a diverse set of use cases and workloads that require distinct considerations for data access. On the lake side, tables with sensitive data require fine-grained access control that are enforced across the raw data and derivative data products via feature engineering or transformations. Whereas on the house side, tables can require fine-grained data access such as row level segmentation for data sharing, and additional transformations using analytics engineering tools. On the consumption side, there are additional considerations for managing access from popular BI tools such as Tableau, Power BI or Looker. The product team at Immuta, a Databricks partner, will share their experience building data access governance solutions for lakehouse architectures across different data lake and warehouse platforms to show how to set up data access for common scenarios for Databricks teams new to SQL Analytics.

Data Mesh 101

ChrisFord803185

Data Mesh is a new socio-technical approach to data architecture, first described by Zhamak Dehghani and popularised through a guest blog post on Martin Fowler's site. Since then, community interest has grown, due to Data Mesh's ability to explain and address the frustrations that many organisations are experiencing as they try to get value from their data. The 2022 publication of Zhamak's book on Data Mesh further provoked conversation, as have the growing number of experience reports from companies that have put Data Mesh into practice. So what's all the fuss about? On one hand, Data Mesh is a new approach in the field of big data. On the other hand, Data Mesh is application of the lessons we have learned from domain-driven design and microservices to a data context. In this talk, Chris and Pablo will explain how Data Mesh relates to current thinking in software architecture and the historical development of data architecture philosophies. They will outline what benefits Data Mesh brings, what trade-offs it comes with and when organisations should and should not consider adopting it.

Databricks Overview for MLOps

Databricks

1) Databricks provides a machine learning platform for MLOps that includes tools for data ingestion, model training, runtime environments, and monitoring. 2) It offers a collaborative data science workspace for data engineers, data scientists, and ML engineers to work together on projects using notebooks. 3) The platform provides end-to-end governance for machine learning including experiment tracking, reproducibility, and model governance.

Data Mesh for Dinner

Kent Graziano

Data mesh is a decentralized approach to managing and accessing analytical data at scale. It distributes responsibility for data pipelines and quality to domain experts. The key principles are domain-centric ownership, treating data as a product, and using a common self-service infrastructure platform. Snowflake is well-suited for implementing a data mesh with its capabilities for sharing data and functions securely across accounts and clouds, with built-in governance and a data marketplace for discovery. A data mesh implemented on Snowflake's data cloud can support truly global and multi-cloud data sharing and management according to data mesh principles.

How a Semantic Layer Makes Data Mesh Work at Scale

DATAVERSITY

Data Mesh is a trending approach to building a decentralized data architecture by leveraging a domain-oriented, self-service design. However, the pure definition of Data Mesh lacks a center of excellence or central data team and doesn’t address the need for a common approach for sharing data products across teams. The semantic layer is emerging as a key component to supporting a Hub and Spoke style of organizing data teams by introducing data model sharing, collaboration, and distributed ownership controls. This session will explain how data teams can define common models and definitions with a semantic layer to decentralize analytics product creation using a Hub and Spoke architecture. Attend this session to learn about: - The role of a Data Mesh in the modern cloud architecture. - How a semantic layer can serve as the binding agent to support decentralization. - How to drive self service with consistency and control.

Big data architectures and the data lake

James Serra

The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include: - Defining top-down and bottom-up approaches to data management - Explaining what a data lake is and how Hadoop can function as the data lake - Describing how a modern data warehouse combines features of a traditional data warehouse and data lake - Discussing how federated querying allows data to be accessed across multiple sources - Highlighting benefits of implementing big data solutions in the cloud - Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (

Free Training: How to Build a Lakehouse

Databricks

Doug Bateman, a principal data engineering instructor at Databricks, presented on how to build a Lakehouse architecture. He began by introducing himself and his background. He then discussed the goals of describing key Lakehouse features, explaining how Delta Lake enables it, and developing a sample Lakehouse using Databricks. The key aspects of a Lakehouse are that it supports diverse data types and workloads while enabling using BI tools directly on source data. Delta Lake provides reliability, consistency, and performance through its ACID transactions, automatic file consolidation, and integration with Spark. Bateman concluded with a demo of creating a Lakehouse.

Building a modern data warehouse

James Serra

Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.

How to Build a ML Platform Efficiently Using Open-Source

Databricks

Fast-growing startups usually face a common set of challenges when employing machine learning. Data scientists are expected to work on new products and develop new models as well as iterate on existing ones. Once in production, models should be continuously monitored and regularly maintained as the infrastructure evolves. Before too long, data scientists end up spending most of their time doing maintenance and firefighting of existing models instead of creating new ones. At GetYourGuide, we faced these challenges and decided to think about machine learning development holistically, which led us to our machine learning platform. Our platform uses MLflow to keep track of our machine learning life-cycle and ease the development experience. To integrate our models into our production environment, we also need to deal with additional requirements like API specification, SLOs and monitoring. To empower our data scientists, we have built a templating system that takes care of the heavy lifting of going to production, leveraging software engineering tools and ML-specific ones like BentoML. In this talk we will present: – Our previous approaches for deploying models and their tradeoffs – Our data science and platform principles – The main functionalities of our platform – A live demo to create a new service – Our learnings in the process

Webinar Data Mesh - Part 3

Jeffrey T. Pollock

The document discusses data mesh vs data fabric architectures. It defines data mesh as a decentralized data processing architecture with microservices and event-driven integration of enterprise data assets across multi-cloud environments. The key aspects of data mesh are that it is decentralized, processes data at the edge, uses immutable event logs and streams for integration, and can move all types of data reliably. The document then provides an overview of how data mesh architectures have evolved from hub-and-spoke models to more distributed designs using techniques like kappa architecture and describes some use cases for event streaming and complex event processing.

What's hot (20)

Delta lake and the delta architecture

Databricks Delta Lake and Its Benefits

3D: DBT using Databricks and Delta

The Heart of the Data Mesh Beats in Real-Time with Apache Kafka

Differentiate Big Data vs Data Warehouse use cases for a cloud solution

Learn to Use Databricks for Data Science

Building Lakehouses on Delta Lake with SQL Analytics Primer

Time to Talk about Data Mesh

What’s New with Databricks Machine Learning

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Considerations for Data Access in the Lakehouse

Data Mesh 101

Databricks Overview for MLOps

Data Mesh for Dinner

How a Semantic Layer Makes Data Mesh Work at Scale

Big data architectures and the data lake

Free Training: How to Build a Lakehouse

Building a modern data warehouse

How to Build a ML Platform Efficiently Using Open-Source

Webinar Data Mesh - Part 3

Similar to Democratizing Data Quality Through a Centralized Platform

Bigiota Company Profile

Krishna Kumar Maurya

Insight-2015-Session-3193

Michal Miklas

The document discusses a presentation given by Jeff Schaeffer of PPL Corporation and Michal Miklas of IBM on model driven analytics using IBM's logical data models. PPL Corporation faces challenges around data integration and governance. The presentation outlines IBM's Data Model for Energy and Utilities, which provides comprehensive data and analytics models to help utilities like PPL accelerate projects involving data warehousing and business intelligence. Adopting IBM's models allows PPL to develop a common business language, build analytics incrementally on a solid foundation, and improve consistency across reporting and analytics.

Fundamentals of BI Report Testing - Module 4

MichaelCalabrese20

The document discusses the assessment phase of a BI report testing process. It describes analyzing requirements, resources, and current tools/processes during assessment. Key steps include identifying defects, stakeholders, requirements, and recommending test strategy improvements. The assessment aims to evaluate the current state and create a plan to test BI reports in a manageable, repeatable, measurable, improvable and automated manner.

QH_SalesPitch (2).pdf

Prashant Thakur

The document provides information about QuikrHomes, a real estate marketing company in India. It details the services QuikrHomes provides, including real estate research and data collection across various Indian cities. Key services highlighted include project performance analysis, demand and supply analysis at the city and micro-market level, and consumer surveys. The document also describes QuikrHomes' methodology for collecting and auditing real estate data from various sources.

EISmartwork Plant Digitization toward Industrial 4.0

Lee Kian Lie

This document provides information about a company called PMT that offers engineering services including 3D laser scanning, dimensional control surveys, underground utility detection, 2D and 3D modeling, and engineering data management. The company was established in 2005 and has grown to 81 employees offering project-based and manpower services. It aims to become a preferred provider of engineering design and database management in oil, gas, and related industries through integrated solutions and intelligent tools.

ASUG 2014 - Big Data and Advanced Analytics

Ravindra Shukla

The document discusses a presentation by Cognilytics on big data and advanced analytics solutions. It provides biographies for two speakers, Ravindra Shukla and Ravi Sundram. It also outlines Cognilytics' offerings around big data, advanced analytics, predictive modeling and data integration. Specific use cases and industry solutions are described. The presentation covers challenges around data management and analytics, the evolution of analytics, and the role of visualization.

Accelerating the Data to Value Journey

Denodo

Watch full webinar here: [https://buff.ly/2R4JjBX] Organizations today are data rich and insights poor. There is data everywhere. ERP systems, CRM systems, external data, data lakes and ponds. The real question to ask is “Are the users getting the insights they need when they need where they need to drive successful business outcomes”. Data Integration is a core pillar of the “Data to Value” journey. In this session you will hear how enterprises across industries are grappling with data, insights challenges and how organizations have adopted data virtualization to accelerate their "data to value" journeys. Watch this Denodo DataFest 2018 session to learn: How to reduce effort to get from data to value Hope to gain faster time to Insights How to reduce overall cost of ownership

Ensure a Successful SAP Hybris Implementation – Part 2: Architecture and Buil...

Kellton Tech Solutions Ltd

This document provides information about an upcoming webinar series on SAP Hybris architecture. It discusses the topics and agenda for Part II of the series on Architecture & Building Blocks, which will cover SAP Hybris architecture models, synchronous and asynchronous integration with SAP systems, and the architecture building blocks. The presentation is by Ravi Pal from Kellton Tech and will provide an overview of SAP Hybris architecture and its key components.

2016 DSG Webinar Azure HDInsight 2 V4

Janani Eshwaran

This document discusses how to implement real-time analytics on big data using Microsoft Azure services. It begins with an introduction and overview of real-time analytics. It then discusses how real-time fraud detection in the telecom industry can help reduce losses. The presentation demonstrates using Azure Event Hubs to capture streaming call record data, Spark Streaming to process it, Hive for storage, and Power BI for visualization and real-time alerts. It provides an architecture diagram and steps for setting up such a real-time fraud detection system.

2016 DSG Webinar Azure HDInsight 2 V4

Janani Eshwaran

This document discusses how to implement real-time analytics on big data using Microsoft Azure services. It begins with an introduction and overview of real-time analytics. It then discusses how real-time fraud detection in the telecom industry can help reduce losses. The presentation demonstrates using Azure Event Hubs to capture streaming call record data, Spark Streaming to process it, storing the results in Hive, and using Power BI for visualization and real-time alerts. It provides an architecture diagram and steps for setting up such a real-time fraud detection system.

Analytics in the Cloud and the ROI for B2B

Veronica Kirn

Run IT as Business Meetup self-service BI

Mark Wu

This document summarizes NetApp's journey implementing self-service analytics. It began in 2009 by building an enterprise data warehouse and BI platform, which enabled a single source of truth but did not support discovery or self-service. In 2013, NetApp deployed Tableau and built a tier 2 data warehouse to enable self-service analytics with data mashing and faster turnaround. Today NetApp uses a dual environment with a top-down traditional BI approach for enterprise reporting and a bottom-up self-service model enabling departments to answer new questions quickly. The key is establishing governance over the self-service model through community involvement and processes for content certification, data governance, and publishing guidelines.

Preparing Your Legacy Data for Automation in S1000D

dclsocialmedia

This document discusses preparing legacy data for automation in S1000D. It outlines the challenges of converting traditional linear documents into the modular structure required by S1000D. These challenges include identifying reusable content, assigning data modules and codes, and structuring information across publications. The document recommends planning thoroughly for a conversion project, including assessing source materials, analyzing content reuse, specifying the conversion, and normalizing data. It describes setting up the conversion project, performing document analysis, and developing a detailed specification to guide the conversion process.

Oracle Project Analytics

Nitai Partners Inc

Rapid Business Analytics Implementation Services from Oracle provide organizations with comprehensive analytics solutions. Nitai Partners is an expert provider of these Oracle Business Analytics services including Oracle BI Foundation, Hyperion, and Endeca. They offer a rapid deployment timeline of 12 weeks or less to install, configure, test, validate data, build dashboards, provide user training, and deploy the analytics solution. Their expertise includes all Oracle Business Analytics modules to guarantee a successful implementation.

Data Warehousing Trends

Chris Riccomini

The document summarizes several data warehousing trends presented by speaker Chris Riccomini. Some of the trends discussed include: real-time data warehousing which handles hard deletes and replay from Kafka; data mesh which advocates for decentralized data ownership and treating data as a product; and headless BI which programs business metrics for use across systems rather than being confined within BI tools. The talk also covered data catalogs, reverse ETL, analytics engineering and other trends.

The Future of Digital Marketing and Advertising: 2023 Predictions

SG Analytics

SG Analytics is a global data analytics firm that has been in business for 16+ years. It has over 1100 employees across multiple locations globally. SG Analytics provides end-to-end data analytics services including data engineering, predictive analytics, BI visualization, and custom analytics solutions. It also offers data modernization, governance, and operations services to help clients establish lean, agile, and adaptive data ecosystems. SG Analytics prides itself on its domain expertise, proven track record with over 100 clients, and focus on business impact and flexibility for clients.

Unlocking Greater Insights with Integrated Data Quality for Collibra

Precisely

Data is arguably your company’s greatest asset, and a thoughtful data governance strategy, along with robust tools like Collibra Data Governance Center (DGC), is essential to getting the most value from that data. However, even the best data governance programs will falter without data quality. Data governance systems provide a framework for the policies, processes, rules, roles and responsibilities that help you manage your enterprise data. But they don’t give you insight into the characteristics and quality of that data – such as errors, outliers and issues – nor how the data changes over time. During this webinar, we discuss how seamlessly integrating Trillium DQ with Collibra DGC creates a complete data governance solution that delivers rapid insights into the health of your data, ensuring trust and compliance with organizational policies and plans. We demonstrate how data is automatically exchanged between the tools so users can: • Quickly establish the rules needed to support policies • Evaluate their data against those rules on an ongoing basis • Identify problems or improvements with their data quality to take action

Putting data to work

Judy Breedlove

The document discusses foundational technologies for data-driven businesses. It describes how data is growing exponentially and outlines challenges in using data due to issues like inconsistency, duplication, and size. It then presents an intelligent data lifecycle framework involving ingesting, interpreting, and transforming data. Key foundational technologies are discussed like messaging systems, data virtualization, rules engines, machine learning, business process management, and robotic process automation. An anti-money laundering use case is presented using these technologies in an open system architecture.

Rega solutions ppt [compatibility mode]

rickkhosla

REGA SOLUTIONS is a full-service IT solutions provider that offers services across the entire software development lifecycle including strategy, implementation, and support. It focuses on Microsoft applications and enterprise solutions. REGA SOLUTIONS' approach is based on an "Enterprise Value Cycle" model of understanding client needs and delivering measurable business benefits through technology-enabled process improvements and organizational change.

Empowering the Data Driven Business with Modern Business Intelligence

DATAVERSITY

By consolidating data engineering, data warehouse, and data science capabilities under a single fully-managed platform, BigQuery can accelerate computation, reduce data analysis costs, and streamline data management. Following in-depth interviews with a security services provider and a telecommunications company, Nucleus Research found that customers moving to Google Cloud BigQuery from on-premises data warehouse solutions accelerate data processing by over 75 percent while reducing data ongoing administrative expenses by over 25 percent. As BigQuery continues to optimize its platform architecture for compute efficiency and multicloud support, Nucleus expects the vendor to see rapid adoption and further penetrate the data warehouse market.

Similar to Democratizing Data Quality Through a Centralized Platform (20)

Bigiota Company Profile

Insight-2015-Session-3193

Fundamentals of BI Report Testing - Module 4

QH_SalesPitch (2).pdf

EISmartwork Plant Digitization toward Industrial 4.0

ASUG 2014 - Big Data and Advanced Analytics

Accelerating the Data to Value Journey

Ensure a Successful SAP Hybris Implementation – Part 2: Architecture and Buil...

2016 DSG Webinar Azure HDInsight 2 V4

Analytics in the Cloud and the ROI for B2B

Run IT as Business Meetup self-service BI

Preparing Your Legacy Data for Automation in S1000D

Oracle Project Analytics

Data Warehousing Trends

The Future of Digital Marketing and Advertising: 2023 Predictions

Unlocking Greater Insights with Integrated Data Quality for Collibra

Putting data to work

Rega solutions ppt [compatibility mode]

Empowering the Data Driven Business with Modern Business Intelligence

More from Databricks

Data Lakehouse Symposium | Day 1 | Part 1

Databricks

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 2

Databricks

Data Lakehouse Symposium | Day 2

Databricks

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Databricks

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

Why APM Is Not the Same As ML Monitoring

Databricks

Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications. As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored. In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Databricks

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

Stage Level Scheduling Improving Big Data and AI Integration

Databricks

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Databricks

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Databricks

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Databricks

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Sawtooth Windows for Feature Aggregations

Databricks

In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Databricks

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Re-imagine Data Monitoring with whylogs and Spark

Databricks

In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.

Raven: End-to-end Optimization of ML Prediction Queries

Databricks

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Processing Large Datasets for ADAS Applications using Apache Spark

Databricks

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Massive Data Processing in Adobe Using Delta Lake

Databricks

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Machine Learning CI/CD for Email Attack Detection

Databricks

Detecting advanced email attacks at scale is a challenging ML problem, particularly due to the rarity of attacks, adversarial nature of the problem, and scale of data. In order to move quickly and adapt to the newest threat we needed to build a Continuous Integration / Continuous Delivery pipeline for the entire ML detection stack. Our goal is to enable detection engineers and data scientists to make changes to any part of the stack including joined datasets for hydration, feature extraction code, detection logic, and develop/train ML models. In this talk, we discuss why we decided to build this pipeline, how it is used to accelerate development and ensure quality, and dive into the nitty-gritty details of building such a system on top of an Apache Spark + Databricks stack.

Jeeves Grows Up: An AI Chatbot for Performance and Quality

Databricks

Sarah: CEO-Finance-Report pipeline seems to be slow today. Why Jeeves: SparkSQL query dbt_fin_model in CEO-Finance-Report is running 53% slower on 2/28/2021. Data skew issue detected. Issue has not been seen in last 90 days. Jeeves: Adding 5 more nodes to cluster recommended for CEO-Finance-Report to finish in its 99th percentile time of 5.2 hours. Who is Jeeves? An experienced Spark developer? A seasoned administrator? No, Jeeves is a chatbot created to simplify data operations management for enterprise Spark clusters. This chatbot is powered by advanced AI algorithms and an intuitive conversational interface that together provide answers to get users in and out of problems quickly. Instead of being stuck to screens displaying logs and metrics, users can now have a more refreshing experience via a two-way conversation with their own personal Spark expert. We presented Jeeves at Spark Summit 2019. In the two years since, Jeeves has grown up a lot. Jeeves can now learn continuously as telemetry information streams in from more and more applications, especially SQL queries. Jeeves now “knows” about data pipelines that have many components. Jeeves can also answer questions about data quality in addition to performance, cost, failures, and SLAs. For example: Tom: I am not seeing any data for today in my Campaign Metrics Dashboard. Jeeves: 3/5 validations failed on the cmp_kpis table on 2/28/2021. Run of pipeline cmp_incremental_daily failed on 2/28/2021. This talk will give an overview of the newer capabilities of the chatbot, and how it now fits in a modern data stack with the emergence of new data roles like analytics engineers and machine learning engineers. You will learn how to build chatbots that tackle your complex data operations challenges.

Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue

Databricks

This presentation introduces Tune and Fugue, frameworks for intuitive and scalable hyperparameter optimization (HPO). Tune supports both non-iterative and iterative HPO problems. For non-iterative problems, Tune supports grid search, random search, and Bayesian optimization. For iterative problems, Tune generalizes algorithms like Hyperband and Asynchronous Successive Halving. Tune allows tuning models both locally and in a distributed manner without code changes. The presentation demonstrates Tune's capabilities through examples tuning Scikit-Learn and Keras models. The goal of Tune and Fugue is to make HPO development easy, testable, and scalable.

Infrastructure Agnostic Machine Learning Workload Deployment

Databricks

When it comes to Large Scale data processing and Machine Learning, Apache Spark is no doubt one of the top battle-tested frameworks out there for handling batched or streaming workloads. The ease of use, built-in Machine Learning modules, and multi-language support makes it a very attractive choice for data wonks. However bootstrapping and getting off the ground could be difficult for most teams without leveraging a Spark cluster that is already pre-provisioned and provided as a managed service in the Cloud, while this is a very attractive choice to get going, in the long run, it could be a very expensive option if it’s not well managed. As an alternative to this approach, our team has been exploring and working a lot with running Spark and all our Machine Learning workloads and pipelines as containerized Docker packages on Kubernetes. This provides an infrastructure-agnostic abstraction layer for us, and as a result, it improves our operational efficiency and reduces our overall compute cost. Most importantly, we can easily target our Spark workload deployment to run on any major Cloud or On-prem infrastructure (with Kubernetes as the common denominator) by just modifying a few configurations. In this talk, we will walk you through the process our team follows to make it easy for us to run a production deployment of our Machine Learning workloads and pipelines on Kubernetes which seamlessly allows us to port our implementation from a local Kubernetes set up on the laptop during development to either an On-prem or Cloud Kubernetes environment

More from Databricks (20)

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Why APM Is Not the Same As ML Monitoring

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Stage Level Scheduling Improving Big Data and AI Integration

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Sawtooth Windows for Feature Aggregations

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Re-imagine Data Monitoring with whylogs and Spark

Raven: End-to-end Optimization of ML Prediction Queries

Processing Large Datasets for ADAS Applications using Apache Spark

Massive Data Processing in Adobe Using Delta Lake

Machine Learning CI/CD for Email Attack Detection

Jeeves Grows Up: An AI Chatbot for Performance and Quality

Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue

Infrastructure Agnostic Machine Learning Workload Deployment

Recently uploaded

PRODUCT | RESEARCH-PRESENTATION-1.1.pptx

amazenolmedojeruel

Parcel Delivery - Intel Segmentation and Last Mile Opt.pdf

AltanAtabarut

Aws MLOps Interview Questions with answers

Sathiakumar Chandr

future-of-asset-management-future-of-asset-management

Aadee4

Audits Of Complaints Against the PPD Report_2022.pdf

evwcarr

Histology of Muscle types histology o.ppt

SamanArshad11

Vrinda store data analysis project using Excel

SantuJana12

Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data

Samuel Jackson

We present our work to improve data accessibility and performance for data-intensive tasks within the fusion research community. Our primary goal is to develop services that facilitate efficient access for data-intensive applications while ensuring compliance with FAIR principles [1], as well as adoption of interoperable tools, methods and standards. The major outcome of our work is the successful creation and deployment of a data service for the MAST (Mega Ampere Spherical Tokamak) experiment [2], leading to substantial enhancements in data discoverability, accessibility, and overall data retrieval performance, particularly in scenarios involving large-scale data access. Our work follows the principles of Analysis-Ready, Cloud Optimised (ARCO) data [3] by using cloud optimised data formats for fusion data. Our system consists of a query-able metadata catalogue, complemented with an object storage system for publicly serving data from the MAST experiment. We will show how our solution integrates with the Pandata stack [4] to enable data analysis and processing at scales that would have previously been intractable, paving the way for data-intensive workflows running routinely with minimal pre-processing on the part of the researcher. By using a cloud-optimised file format such as zarr [5] we can enable interactive data analysis and visualisation while avoiding large data transfers. Our solution integrates with common python data analysis libraries for large, complex scientific data such as xarray [6] for complex data structures and dask [7] for parallel computation and lazily working with larger that memory datasets. The incorporation of these technologies is vital for advancing simulation, design, and enabling emerging technologies like machine learning and foundation models, all of which rely on efficient access to extensive repositories of high-quality data. Relying on the FAIR guiding principles for data stewardship not only enhances data findability, accessibility, and reusability, but also fosters international cooperation on the interoperability of data and tools, driving fusion research into new realms and ensuring its relevance in an era characterised by advanced technologies in data science. [1] Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) https://doi.org/10.1038/sdata.2016.18 [2] M Cox, The Mega Amp Spherical Tokamak, Fusion Engineering and Design, Volume 46, Issues 2–4, 1999, Pages 397-404, ISSN 0920-3796, https://doi.org/10.1016/S0920-3796(99)00031-9 [3] Stern, Charles, et al. "Pangeo forge: crowdsourcing analysis-ready, cloud optimized data production." Frontiers in Climate 3 (2022): 782909. [4] Bednar, James A., and Martin Durant. "The Pandata Scalable Open-Source Analysis Stack." (2023). [5] Alistair Miles (2024) ‘zarr-developers/zarr-python: v2.17.1’. Zenodo. doi: 10.5281/zenodo.10790679 [6] Hoyer, S. & Hamman, J., (20

Big Data and Analytics Shaping the future of Payments

RuchiRathor2

Bimbingan kaunseling untuk pelajar IPTA/IPTS di Malaysia

aznidajailani

Where to order Frederick Community College diploma?

SomalyEng

DESIGN AND DEVELOPMENT OF AUTO OXYGEN CONCENTRATOR WITH SOS ALERT FOR HIKING ...

JeevanKp7

Long-term oxygen therapy (LTOT) and novel techniques of evaluating treatment efficacy have enhanced the quality of life and decreased healthcare expenses for COPD patients. The cost of a pulmonary blood gas test is comparable to the cost of two days of oxygen therapy and the cost of a hospital stay is equivalent to the cost of one month of oxygen therapy, long-term oxygen therapy (LTOT) is a cost-effective technique of treating this disease. A small number of clinical investigations on LTOT have shown that it improves the quality of life of COPD patients by reducing the loss of their respiratory capacity. A study of 8487 Danish patients found that LTOT for 1524 hours per day extended life expectancy from 1.07 to 1.40 years.

Training on CSPro and step by steps.pptx

lenjisoHussein

CT AnGIOGRAPHY of pulmonary embolism.pptx

RejoJohn2

Technology used in Ott data analysis project

49AkshitYadav

Solution Manual for First Course in Abstract Algebra A, 8th Edition by John B...

rightmanforbloodline

Data Analytics for Decision Making By District 11 Solutions

District 11 Solutions

SFBA Splunk Usergroup meeting July 17, 2024

Becky Burwell

393947940-The-Dell-EMC-PowerMax-Family-Overview.pdf

Ladislau5

SOFTWARE ENGINEERING-UNIT-1SOFTWARE ENGINEERING

PrabhuB33

Recently uploaded (20)

PRODUCT | RESEARCH-PRESENTATION-1.1.pptx

Parcel Delivery - Intel Segmentation and Last Mile Opt.pdf

Aws MLOps Interview Questions with answers

future-of-asset-management-future-of-asset-management

Audits Of Complaints Against the PPD Report_2022.pdf

Histology of Muscle types histology o.ppt

Vrinda store data analysis project using Excel

Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data

Big Data and Analytics Shaping the future of Payments

Bimbingan kaunseling untuk pelajar IPTA/IPTS di Malaysia

Where to order Frederick Community College diploma?

DESIGN AND DEVELOPMENT OF AUTO OXYGEN CONCENTRATOR WITH SOS ALERT FOR HIKING ...

Training on CSPro and step by steps.pptx

CT AnGIOGRAPHY of pulmonary embolism.pptx

Technology used in Ott data analysis project

Solution Manual for First Course in Abstract Algebra A, 8th Edition by John B...

Data Analytics for Decision Making By District 11 Solutions

SFBA Splunk Usergroup meeting July 17, 2024

393947940-The-Dell-EMC-PowerMax-Family-Overview.pdf

SOFTWARE ENGINEERING-UNIT-1SOFTWARE ENGINEERING

Democratizing Data Quality Through a Centralized Platform

1. 1 1 Smit Shah Yuliana Havryshchuk Democratizing Data Quality at Zillow through a Centralized Platform

2. 2 Who We Are Data Governance Platform Team @ Zillow Smit Shah Senior Software Development Engineer, Big Data Yuliana Havryshchuk Software Development Engineer, Big Data

3. 3 Agenda ● What is Zillow? ● Data Quality Challenges ● Centralized Data Quality Platform ○ Architecture ○ Self-Service ○ Pipeline integration ● Key Takeaways

4. Zillow

5. About Zillow ● Reimagining real estate to make it easier to unlock life’s next chapter * As of Q4-2020 ● Oﬀer customers an on-demand experience for selling, buying, renting and ﬁnancing with transparency and nearly seamless end-to-end service ● Most-visited real estate website in the United States

6. Data Quality Challenges

7. Why Monitor Data Quality? ● Data fuels many customer facing and internal services at Zillow that rely on high quality data ○ Zestimate ○ Zillow Oﬀers ○ Zillow Premier Agent ○ Econ and many more ● Reliable performance of ML and Services requires certain level of data quality

8. Challenges we Faced ● No standard way to monitor quality ● Lack of visibility into data health ● No known lineage between data and processes

9. Centralized Data Quality Platform

10. Data Quality Platform Increase Visibility of Data Health Integrate with Data Lineage Support Built-in Alerting Enable Safe Evolution of Rules Standardize Data Quality Rules 5 Pillars for Data Quality Platform

11. Platform Architecture * As of May 2021

12. Platform Architecture * As of May 2021

13. Platform Architecture * As of May 2021

14. Platform Architecture * As of May 2021

15. Platform Architecture * As of May 2021

16. Platform Architecture * As of May 2021

17. Platform Architecture * As of May 2021

18. Self-Service Capabilities

19. Self-Service Onboarding - Goals ● Must be scalable ● Must be accessible to all user archetypes ● Must require minimal conﬁguration

20. Self-Service Onboarding - Data Discovery * These values are simulated

21. Self-Service Onboarding - Example * These values are simulated id name type page_views data_date 1 123 Green St house 709 2021-05-01 2 47 Walker Rd townhouse 132 2021-05-01 1225 City St #901 condo 800 2021-05-01 4 47 Walker Ave test 600 2021-05-01

22. Self-Service Onboarding - Rule-based * These values are simulated

23. Self-Service Monitoring - Rule-based * These values are simulated

24. Self-Service Onboarding - Example * These values are simulated id name type page_views data_date 1 123 Green St house 709 2021-05-01 1 123 Green St house 820 2021-05-02 1 123 Green St house 12 2021-05-03 1 123 Green St house 760 2021-05-04

25. Self-Service Onboarding - Metrics * These values are simulated

26. Overview Metric * These values are simulated Self-Service Onboarding - Monitoring

27. Behind the Scenes ● Rule-based monitors turn into contracts ● Metrics monitors turn into ML-based anomaly detection ● Register data quality requirements in conﬁg stores ● Dynamically generate validation pipelines

28. Validation Libraries Built in-house: ● Luminaire Contract Evaluation Library (scala) for rule-based constraints ● Luminaire Anomaly Detection Library (python) for time-series metrics ○ https://github.com/zillow/luminaire

29. Pipeline Integration

30. Pipeline Integration (before) Producers Consumers

31. Pipeline Integration (after) Producers Consumers *

32. Validation Results ● Alert data users if any checks fail ● Integrate with pipeline execution to prevent propagation ● Provide visibility through data discovery tool ● Provide common understanding between producers and consumers

33. Future Direction ● Tighter integration between components ● Expand libraries to support more use-cases ● Move from detection to diagnosis ● Validation for streaming data

34. Key Takeaways

35. Key Takeaways ● 5 pillars that helped us build a robust platform: standardization, visibility, evolution, alerting, lineage ● Alerting on data quality issues early allows proactive response ● Producing quality data increases trust in data and improves decisions made ● Data quality is a shared responsibility, and collaboration is needed to be successful

36. Questions? Thank you! https://www.zillow.com/careers/

Democratizing Data Quality Through a Centralized Platform

Related slideshows

More Related Content

What's hot

What's hot (20)

Similar to Democratizing Data Quality Through a Centralized Platform

Similar to Democratizing Data Quality Through a Centralized Platform (20)

More from Databricks

More from Databricks (20)

Recently uploaded

Recently uploaded (20)

Democratizing Data Quality Through a Centralized Platform