Data Ingest Self Service and Management using Nifi and Kafka
•
12 likes•3,941 views
We’re feeling the growing pains of maintaining a large data platform. Last year we went from 50 to 150 unique data feeds by adding them all by hand. In this talk we will share the best practices developed to handle our 300% increase in feeds through self service. Having self-service capabilities will increase your teams velocity and decrease your time to value and insight. * Self service data feed design and ingest * configuration management * automatic debugging * light weight data governance
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Data Ingest Self Service and Management using Nifi and Kafka
Similar to Data Ingest Self Service and Management using Nifi and Kafka (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Data Ingest Self Service and Management using Nifi and Kafka
- 1. Data Ingest Self-Service and Management using NiFi and Kafka Imran Amjad, Principal Engineer Dave Torok, Principal Architect June 14, 2017
- 2. XFINITY TV XFINITY Internet XFINITY Voice XFINITY Home Digital & OtherOther *Minority interest and/or non-controlling interest. Slide is not comprehensive of all Comcast NBCUniversal assets Updated: December 22, 2015
- 3. Introduction and Background • Customer Experience UI with 30,000 unique internal users per month • Ingesting about 2 Billion Events / Month • Typical “Big Data Analytics” Pipeline • Data ETL, land in a data lake (e.g. HBase) • API / several channels of consumers / 14 million requests per day • Grew from a few dozen to 150+ data sources / feeds in about a year • Pipeline of 5-10 new data feeds per two week sprint Data Ingestion Self-Service and Management using NiFi and Kafka3
- 4. High Level Architecture Data Ingestion Self-Service and Management using NiFi and Kafka4 Streaming Compute Pipeline UI and Other Consumers HTTP Gateway BATCH Data Sources Analytics DB Event Storage DB Rules Enrich Standardize Detect Aggregate Real Time Data Sources “Pull” Kafka Bridge (NiFi) Store REST API Filestore Apache Flink Apache NiFi Kafka (Event Bus) NiFi Kafka (Event Bus)
- 5. Streaming Compute Pipeline UI and Other Consumers HTTP Gateway BATCH Data Sources Analytics DB Event Storage DB Rules Enrich Standardize Detect Aggregate Real Time Data Sources “Pull” Kafka Bridge (NiFi) Store REST API Filestore Apache Flink Apache NiFi Kafka (Event Bus) NiFi Kafka (Event Bus) High Level Architecture Data Ingestion Self-Service and Management using NiFi and Kafka5 Data Ingestion (“ETL”) Analytics & Active Decisioning UI / Services
- 6. Problem Statement and Motivation for Self Service Time-to-market VP: “I'd like to be able to add a new event stream in 10 minutes.” Data Ingestion Self-Service and Management using NiFi and Kafka6 Manual Processes Code Deployment
- 7. Dimensions of Ingest Variability Data Ingestion Self-Service and Management using NiFi and Kafka7 Transport Protocol Kafka Kinesis HTTP/S Files (S)FTP Format JSON XML AVRO CSV / Delimited Custom Timing [Near] Real-Time Streaming Batch / Periodic Ingest Control Pull from Source Push by Producer
- 8. Data Source Onboarding – Before Self-Service Data Ingestion Self-Service and Management using NiFi and Kafka8
- 9. Data Source Onboarding – Before Self-Service Data Ingestion Self-Service and Management using NiFi and Kafka9 Manual Process Code
- 10. Self-Service Architecture Principles Data Ingestion Self-Service and Management using NiFi and Kafka10 Metadata Driven Data Ingestion, Processing, and Rendering Driven by Metadata Automation Orchestrated Deployment for New Data Feeds Rapid Onboarding Portal for Data Source Management Light Data Governance Schema- backed Data, Schema Registry Monitoring and Metrics Ingestion, Data Quality, and Operational Status
- 11. Streaming Compute Pipeline UI and Other Consumers HTTP Gateway BATCH Data Sources Analytics DB Event Storage DB Rules Enrich Standardize Detect Aggregate Real Time Data Sources “Pull” Kafka Bridge (NiFi) Store REST API Self-Serve Metadata + Content Management DB Self Service API Filestore Apache Flink Apache NiFi Kafka (Event Bus) NiFi Self-Service UI Kafka (Event Bus) High Level Architecture with Self-Service Data Ingestion Self-Service and Management using NiFi and Kafka11 Data Ingestion (“ETL”) Analytics & Active Decisioning UI / Services Self-Service
- 12. Streaming Compute Pipeline UI and Other Consumers HTTP Gateway BATCH Data Sources Analytics DB Event Storage DB Rules Enrich Standardize Detect Aggregate Real Time Data Sources “Pull” Kafka Bridge (NiFi) Store REST API Self-Serve Metadata + Content Management DB Self Service API Filestore Apache Flink Apache NiFi Kafka (Event Bus) NiFi Self-Service UI Kafka (Event Bus) Data Ingestion (“ETL”) Analytics & Active Decisioning UI / Services Self-Service High Level Architecture with Self-Service Data Ingestion Self-Service and Management using NiFi and Kafka12
- 13. Metadata and Data Governance Data Ingestion Self-Service and Management using NiFi and Kafka13
- 14. Models and Metadata to enable Self-Service Self Service Metadata Management CORE METADATA Data Model and Data Dictionary INGEST And ETL Metadata PROCESSING Metadata Lookups, Enrichment, Aggregation, Expressions UI / RENDERING METADATA BUSINESS CONTENT Enrichment and Notification Templates and Lookups Data Ingestion Self-Service and Management using NiFi and Kafka14
- 15. Metadata Management – Example “Tags” on individual data fields Data Ingestion Self-Service and Management using NiFi and Kafka15 Display Field (true) Category (“Product”) Icon File (“x1_tv.svg”) Icon Color (“green”) UI Rendering Sensitivity (none / encrypt) Encryption KeyId (“SomeKeyId”) Security ShortName (“x1view”) Description (“X1 View Program”) Format (“json”) Data Source Information Ingest Handling (Ingest /Drop) Field Name (“viewcode”) Field JsonPath (“$.EVT.VALUE.CODE”) Source Field Information Target Domain Object (“error”) Field JsonPath (“$.data.error.viewCode) Target Field Information
- 16. Why Data Governance? Data Ingestion Self-Service and Management using NiFi and Kafka16 Validate Types against Schema Detect Structure Changes Backwards/Forwards Compatibility Universally Required Information Data Quality Data Rationalization Standard Syntax and Semantics Domains (“Customer”, “Device”) Standard Field Names and Types Message Data Format Support Integration / Correlation Data Curation Metadata Management Data Source Registry Schema Repository Support Lineage Traceability Additional Properties (e.g. Security)
- 17. Lightweight Data Governance Why JSON Schema over Avro for our use case? • Data sources / producers generally aren’t using Avro • Database Storage, UI, REST API is not Avro • Tolerate data change: Detect, Accept, Notify, Correct (Later) • Don’t “drop data on the floor” – AVRO ignores unknown fields • [Some] AWS Services – JSON friendly but not Avro friendly Data Ingestion Self-Service and Management using NiFi and Kafka17 AVRO Schema vs. JSON Schema JSON + JSON Schema AVRO + AVRO Schema Validation Validation JSON Serialization Framework Optimized for Data Storage Upfront Data Governance
- 18. Lightweight Data Governance Versioned Data and Schema • Allow ingestion of multiple versions (particularly from different sources) Data Quality • Invalid Data Types • Non-parseable Payloads • Missing Required Fields Data Change Tolerance • Detect Additional Data / “Unknown Fields” • Store alongside schema-modeled data • Allow for display in UI only after updating Schema Quality Feedback loop to data producers Data Ingestion Self-Service and Management using NiFi and Kafka18
- 19. Lightweight Data Governance Reduced schema review and approval process Generate JSON Schema Artifacts from Field Metadata Lightweight Metadata Repository • Considering Apache Atlas and Hortonworks Schema Registry Pre-defined Core Schema / Domain Objects • 5-15 (or so) domain object schema to be included in an event schema • E.g. Customer, Device, Geolocation Data Ingestion Self-Service and Management using NiFi and Kafka19
- 20. Portal for Data Source Onboarding Data Ingestion Self-Service and Management using NiFi and Kafka20
- 21. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka21
- 22. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka22
- 23. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka23
- 24. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka24
- 25. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka25
- 26. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka26
- 27. Data Source Onboarding UI Data Ingestion Self-Service and Management using NiFi and Kafka27
- 28. Platform Automation for Kafka and NiFi Data Ingestion Self-Service and Management using NiFi and Kafka28
- 29. Data Source Onboarding – With Self-Service Data Ingestion Self-Service and Management using NiFi and Kafka29 Create Schema and Metadata Register the Schema to generate the artifacts Test & Validate with Sample data Publish & Start Ingesting Live Data Data Producer
- 30. Generating NiFi Flows from Metadata Data Ingestion Self-Service and Management using NiFi and Kafka30
- 31. Finding Data with JsonPath (Example from http://petstore.swagger.io/ ) Data Ingestion Self-Service and Management using NiFi and Kafka31 { "id": 0, "category": { "id": 0, "name": “dogs" }, "name": “Fido", "photoUrls": [ “http://myimage.com" ], "tags": [ { "id": 0, "name": “friendly" }, {“id”:1, “name” : “housebroken”} ], "status": "available" } $.id = 0 $.category.name = “dogs” $.tags[?(@.id == 0)] = {id = 0, “name =“friendly”}
- 32. Example Source Data Payload { "PV": "1.1", "APP": { "APP_NAME": "XRE", "APP_VER": "X1" }, "DEV": { "DEVICE_TYPE": "Xi3", }, "ACNT": { "BILL_ID": " 1234321234 " }, "LOC": { "UTC_OFF": "-05" }, "EVT": { "ETS": 1487880967000, "NAME": "program", "VALUE": { "TYPE": "XRE", "CODE": "XRE-12345", "DESCRIPTION": "Customer started a program" } }, "PTS": 1468876861829 } Data Ingestion Self-Service and Management using NiFi and Kafka32 $.ACNT.BILL_ID $.LOC.UTC_OFF $.EVT.ETS
- 33. Example Target Schema Payload { "schema" : "comcast/xreinfo/jsonschema/1-0-0", "billingId":"1234321234", "eventSource":"xre", "eventType":"xreinfo", "timestamp":1487880967000, "data" : { "device" : { "deviceType" : "Xi3" }, "customerAccount" : { "billingId" : "1234321234", "accountType" : "residential" }, “timeInformation" : { “utcOffset" : “-5" }, "info" : { "infoCode" : "XRE-12345" } }, "unknownData" : { "EVT" : { “INFO": { "ABXXY" : “MORE DATA" } } } } Data Ingestion Self-Service and Management using NiFi and Kafka33
- 34. Simple Jsonpath Transformation FROM $.ACNT.BILL_ID TO $.billingId TO $.data.customerAccount.billingId) FROM $.EVT.ETS TO $.data.timestamp FROM $.LOC.UTC_OFF TO $.data.timeInformation.utcOffset Data Ingestion Self-Service and Management using NiFi and Kafka34 { "schema" : "comcast/xreinfo/jsonschema/1-0-0", "billingId":"1234321234", "eventSource":"xre", "eventType":"xreinfo", "timestamp":1487880967000, "data" : { "device" : { "deviceType" : “Xi3" }, "customerAccount" : { "billingId" : "1234321234" }, “timeInformation" : { “utcOffset" : “-5" }, "info" : { "infoCode" : "XRE-12345" } },
- 35. Advanced JSON Transformation using JOLT Rich DSL for transforming and manipulating JSON • Transform Structure (“Shift”) • Defaults • Sort • Remove • Expressions https://github.com/bazaarvoice/jolt Apache 2.0 License NiFi “JoltTransformJSON” Processor Data Ingestion Self-Service and Management using NiFi and Kafka35
- 36. JOLT Transformation for X1 Event Data Ingestion Self-Service and Management using NiFi and Kafka36
- 37. Advanced Transform Examples (“JOLT” Processor) INPUT: {"ChangedFields":["JobStateCd","DispatcherStatusCd"]} OUTPUT:{“ChangedFields_0”:“JobStateCd”, “ChangedFields_1”:“DispatcherStatusCd”, “ChangedReason”, “JobStateCd|DispatcherStatusCd||||” } Data Ingestion Self-Service and Management using NiFi and Kafka37
- 38. Advanced Transform Examples (“JOLT” Processor) INPUT: {"ChangedFields":["JobStateCd","DispatcherStatusCd"]} "spec": { "ChangedFields_0": "=elementAt(@(1,ChangedField),0)", "ChangedFields_1": "=elementAt(@(1,ChangedField),1)", "ChangedFields_2": "=elementAt(@(1,ChangedField),2)", "ChangedFields_3": "=elementAt(@(1,ChangedField),3)", "ChangedFields_4": "=elementAt(@(1,ChangedField),4)", "ChangedFields_5": "=elementAt(@(1,ChangedField),5)" } }, "spec": { "ChangedReason": "=concat(@(1,ChangedFields_0),'|',@(1,ChangedFields_1),'|',@(1,ChangedFields _2),'|',@(1,ChangedFields_3),'|',@(1,ChangedFields_4),'|',@(1,ChangedFields_ 5))" } OUTPUT:{“ChangedFields_0”:“JobStateCd”, “ChangedFields_1”:“DispatcherStatusCd”, “ChangedReason”, “JobStateCd|DispatcherStatusCd||||” } Data Ingestion Self-Service and Management using NiFi and Kafka38
- 39. Advanced Transform Examples (“JOLT” Processor) INPUT: {"ChangedFields":["JobStateCd","DispatcherStatusCd"]} "spec": { "ChangedFields_0": "=elementAt(@(1,ChangedField),0)", "ChangedFields_1": "=elementAt(@(1,ChangedField),1)", "ChangedFields_2": "=elementAt(@(1,ChangedField),2)", "ChangedFields_3": "=elementAt(@(1,ChangedField),3)", "ChangedFields_4": "=elementAt(@(1,ChangedField),4)", "ChangedFields_5": "=elementAt(@(1,ChangedField),5)" } }, "spec": { "ChangedReason": "=concat(@(1,ChangedFields_0),'|',@(1,ChangedFields_1),'|',@(1,ChangedFields _2),'|',@(1,ChangedFields_3),'|',@(1,ChangedFields_4),'|',@(1,ChangedFields_ 5))" } OUTPUT:{“ChangedFields_0”:“JobStateCd”, “ChangedFields_1”:“DispatcherStatusCd”, “ChangedReason”, “JobStateCd|DispatcherStatusCd||||” } Data Ingestion Self-Service and Management using NiFi and Kafka39
- 40. Advanced Transform Examples (“JOLT” Processor) “Devices” : { “array” : [ {“fulfillmentID”: “12345”, “Device”: {“schema.core.Device": { “model” : { “string”: “AX013ANM” }}}}, {“fulfillmentID”: “23456”, “Device”: {“schema": { “model” : { “string” :“PXD01ANI” }}}}]} "Devices": { "array": { "*": { "fulfillmentID": { "string": "events[0].data.fid_&2"}, "device": { “schema.core.Device": { “model": { "string": "events[0].data.model_&4" “events” : [ { “data” : { “fid_0" : “12345", "model_0" : "AX013ANM ", "fid_1" : “23456", “model_1" : "PXD01ANI” } } ] Data Ingestion Self-Service and Management using NiFi and Kafka40
- 41. NiFi REST API – Create a Process Group Data Ingestion Self-Service and Management using NiFi and Kafka41 POST http://localhost:8080/nifi-api/process-groups/root/process-groups { "revision": { "version" : 0 }, "component": { "name" : "x1info Process Group" } }
- 42. NiFi REST API – Create a ConsumeKafka Processor POST http://localhost:8080/nifi-api/process-groups/{ID}/processors { "revision": { "version": 0 }, "component": { "config": { "properties": { "bootstrap.servers": "localhost:9092", "topic": "raw.mystream.x1info", "group.id": "nifi-stage-0522", "auto.offset.reset": "latest“ }}, "name": "ConsumeKafka - x1info", "type": "org.apache.nifi.processors.kafka.pubsub.ConsumeKafka“ }} Data Ingestion Self-Service and Management using NiFi and Kafka42
- 43. Generated NiFi Flow Data Ingestion Self-Service and Management using NiFi and Kafka43
- 44. Monitoring and Metrics • Dashboard of Data Quality • Ingestion Rate Monitoring (and possibly alerting with anomaly detection) • Alerting to producers (Data Quality) Data Ingestion Self-Service and Management using NiFi and Kafka44
- 45. Ingestion Status Dashboard Data Ingestion Self-Service and Management using NiFi and Kafka45 Data Center 1 Data Center 2 Event_type1 Event_type2 MyEvent WeekendEvent
- 46. Anomaly Detection Data Ingestion Self-Service and Management using NiFi and Kafka46
- 47. Self-Service Lessons Learned Design with automation and configuration in mind Metadata-driven design reduces code deployments and custom solutions Make Simple things Simple – But allow hard things to be “code” and not UI-driven Let Data Producers be accountable for Data Quality NiFi and JOLT = Powerful Toolkit Data Ingestion Self-Service and Management using NiFi and Kafka47
- 48. Thank You!