Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

•

0 likes•751 views

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

What's hot

Apache Iceberg - A Table Format for Hige Analytic Datasets

Alluxio, Inc.

Understanding Query Plans and Spark UIs

Databricks

"The common use cases of Spark SQL include ad hoc analysis, logical warehouse, query federation, and ETL processing. Spark SQL also powers the other Spark libraries, including structured streaming for stream processing, MLlib for machine learning, and GraphFrame for graph-parallel computation. For boosting the speed of your Spark applications, you can perform the optimization efforts on the queries prior employing to the production systems. Spark query plans and Spark UIs provide you insight on the performance of your queries. This talk discloses how to read and tune the query plans for enhanced performance. It will also cover the major related features in the recent and upcoming releases of Apache Spark. "

Centralized log-management-with-elastic-stack

Rich Lee

Centralized log management is implemented using the Elastic Stack including Filebeat, Logstash, Elasticsearch, and Kibana. Filebeat ships logs to Logstash which transforms and indexes the data into Elasticsearch. Logs can then be queried and visualized in Kibana. For large volumes of logs, Kafka may be used as a buffer between the shipper and indexer. Backups are performed using Elasticsearch snapshots to a shared file system or cloud storage. Logs are indexed into time-based indices and a cron job deletes old indices to control storage usage.

Elastic Stack Introduction

Vikram Shinde

Apache Flink and what it is used for

Aljoscha Krettek

Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

Databricks

Structured Streaming has proven to be the best platform for building distributed stream processing applications. Its unified SQL/Dataset/DataFrame APIs and Spark’s built-in functions make it easy for developers to express complex computations. Delta Lake, on the other hand, is the best way to store structured data because it is a open-source storage layer that brings ACID transactions to Apache Spark and big data workloads Together, these can make it very easy to build pipelines in many common scenarios. However, expressing the business logic is only part of the larger problem of building end-to-end streaming pipelines that interact with a complex ecosystem of storage systems and workloads. It is important for the developer to truly understand the business problem that needs to be solved. Apache Spark, being a unified analytics engine doing both batch and stream processing, often provides multiples ways to solve the same problem. So understanding the requirements carefully helps you to architect your pipeline that solves your business needs in the most resource efficient manner. In this talk, I am going examine a number common streaming design patterns in the context of the following questions. WHAT are you trying to consume? What are you trying to produce? What is the final output that the business wants? What are your throughput and latency requirements? WHY do you really have those requirements? Would solving the requirements of the individual pipeline actually solve your end-to-end business requirements? HOW are going to architect the solution? And how much are you willing to pay for it? Clarity in understanding the ‘what and why’ of any problem can automatically much clarity on the ‘how’ to architect it using Structured Streaming and, in many cases, Delta Lake.

(BDT305) Amazon EMR Deep Dive and Best Practices

Amazon Web Services

Amazon EMR is one of the largest Hadoop operators in the world. In this session, we introduce you to Amazon EMR design patterns such as using Amazon S3 instead of HDFS, taking advantage of both long and short-lived clusters, and other Amazon EMR architectural best practices. We talk about how to scale your cluster up or down dynamically and introduce you to ways you can fine-tune your cluster. We will also share best practices to keep your Amazon EMR cluster cost-efficient. Finally, we dive into some of our recent launches to keep you current on our latest features.

Radical Speed for SQL Queries on Databricks: Photon Under the Hood

Databricks

Join this session to hear from the Photon product and engineering team talk about the latest developments with the project. As organizations embrace data-driven decision-making, it has become imperative for them to invest in a platform that can quickly ingest and analyze massive amounts and types of data. With their data lakes, organizations can store all their data assets in cheap cloud object storage. But data lakes alone lack robust data management and governance capabilities. Fortunately, Delta Lake brings ACID transactions to your data lakes – making them more reliable while retaining the open access and low storage cost you are used to. Using Delta Lake as its foundation, the Databricks Lakehouse platform delivers a simplified and performant experience with first-class support for all your workloads, including SQL, data engineering, data science & machine learning. With a broad set of enhancements in data access and filtering, query optimization and scheduling, as well as query execution, the Lakehouse achieves state-of-the-art performance to meet the increasing demands of data applications. In this session, we will dive into Photon, a key component responsible for efficient query execution. Photon was first introduced at Spark and AI Summit 2020 and is written from the ground up in C++ to take advantage of modern hardware. It uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications — all natively on your data lake. Photon is fully compatible with the Apache Spark™ DataFrame and SQL APIs to ensure workloads run seamlessly without code changes. Come join us to learn more about how Photon can radically speed up your queries on Databricks.

Iceberg: a fast table format for S3

DataWorks Summit

Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. With a data warehouse at this scale, it is a constant challenge to keep improving performance. This talk will focus on Iceberg, a new table metadata format that is designed for managing huge tables backed by S3 storage. Iceberg decreases job planning time from minutes to under a second, while also isolating reads from writes to guarantee jobs always use consistent table snapshots. In this session, you'll learn: • Some background about big data at Netflix • Why Iceberg is needed and the drawbacks of the current tables used by Spark and Hive • How Iceberg maintains table metadata to make queries fast and reliable • The benefits of Iceberg's design and how it is changing the way Netflix manages its data warehouse • How you can get started using Iceberg Speaker Ryan Blue, Software Engineer, Netflix

Programming in Spark using PySpark

Mostafa

This session covers how to work with PySpark interface to develop Spark applications. From loading, ingesting, and applying transformation on the data. The session covers how to work with different data sources of data, apply transformation, python best practices in developing Spark Apps. The demo covers integrating Apache Spark apps, In memory processing capabilities, working with notebooks, and integrating analytics tools into Spark Applications.

Apache Arrow: Open Source Standard Becomes an Enterprise Necessity

Wes McKinney

Producer Performance Tuning for Apache Kafka

Jiangjie Qin

What is in a Lucene index?

lucenerevolution

Presented by Adrien Grand, Software Engineer, Elasticsearch Although people usually come to Lucene and related solutions in order to make data searchable, they often realize that it can do much more for them. Indeed, its ability to handle high loads of complex queries make Lucene a perfect fit for analytics applications and, for some use-cases, even a credible replacement for a primary data-store. It is important to understand the design decisions behind Lucene in order to better understand the problems it can solve and the problems it cannot solve. This talk will explain the design decisions behind Lucene, give insights into how Lucene stores data on disk and how it differs from traditional databases. Finally, there will be highlights of recent and future changes in Lucene index file formats.

Introduction to PySpark

Russell Jurney

Grafana introduction

Rico Chen

Grafana is an open source analytics and monitoring tool that uses InfluxDB to store time series data and provide visualization dashboards. It collects metrics like application and server performance from Telegraf every 10 seconds, stores the data in InfluxDB using the line protocol format, and allows users to build dashboards in Grafana to monitor and get alerts on metrics. An example scenario is using it to collect and display load time metrics from a QA whitelist VM.

Building large scale transactional data lake using apache hudi

Bill Liu

Data is a critical infrastructure for building machine learning systems. From ensuring accurate ETAs to predicting optimal traffic routes, providing safe, seamless transportation and delivery experiences on the Uber platform requires reliable, performant large-scale data storage and analysis. In 2016, Uber developed Apache Hudi, an incremental processing framework, to power business critical data pipelines at low latency and high efficiency, and helps distributed organizations build and manage petabyte-scale data lakes. In this talk, I will describe what is APache Hudi and its architectural design, and then deep dive to improving data operations by providing features such as data versioning, time travel. We will also go over how Hudi brings kappa architecture to big data systems and enables efficient incremental processing for near real time use cases. Speaker: Satish Kotha (Uber) Apache Hudi committer and Engineer at Uber. Previously, he worked on building real time distributed storage systems like Twitter MetricsDB and BlobStore. website: https://www.aicamp.ai/event/eventdetails/W2021043010

Apache Arrow Flight Overview

Jacques Nadeau

Arrow Flight is a proposed RPC layer for Apache Arrow that allows for efficient transfer of Arrow record batches between systems. It uses GRPC as the foundation to define streams of Arrow data that can be consumed in parallel across locations. Arrow Flight supports custom actions that can be used to build services on top of the generic API. By extending GRPC, Arrow Flight aims to simplify the creation of data applications while enabling high performance data transfer and locality awareness.

ClickHouse Introduction by Alexander Zaitsev, Altinity CTO

Altinity Ltd

This document summarizes a ClickHouse meetup agenda. The meetup included an opening by Javier Santana, an introduction to ClickHouse by Alexander Zaitsev of Altinity, a presentation on 2019 new ClickHouse features by Alexey Milovidov of Yandex, a coffee break, a presentation from Idealista on migrating from a legacy system to ClickHouse, a presentation from Corunet on analyzing 1027 predictive models in 10 seconds using ClickHouse, a presentation from Adjust on shipping data from Postgres to ClickHouse, closing remarks, and a networking session. The document then provides an overview of what ClickHouse is, how fast it can be, how flexible it is in deployment options, how

Introduction to Apache Spark

Rahul Jain

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Noritaka Sekiyama

This document provides an overview and summary of Amazon S3 best practices and tuning for Hadoop/Spark in the cloud. It discusses the relationship between Hadoop/Spark and S3, the differences between HDFS and S3 and their use cases, details on how S3 behaves from the perspective of Hadoop/Spark, well-known pitfalls and tunings related to S3 consistency and multipart uploads, and recent community activities related to S3. The presentation aims to help users optimize their use of S3 storage with Hadoop/Spark frameworks.

What's hot (20)

Apache Iceberg - A Table Format for Hige Analytic Datasets

Understanding Query Plans and Spark UIs

Centralized log-management-with-elastic-stack

Elastic Stack Introduction

Apache Flink and what it is used for

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

(BDT305) Amazon EMR Deep Dive and Best Practices

Radical Speed for SQL Queries on Databricks: Photon Under the Hood

Iceberg: a fast table format for S3

Programming in Spark using PySpark

Apache Arrow: Open Source Standard Becomes an Enterprise Necessity

Producer Performance Tuning for Apache Kafka

What is in a Lucene index?

Introduction to PySpark

Grafana introduction

Building large scale transactional data lake using apache hudi

Apache Arrow Flight Overview

ClickHouse Introduction by Alexander Zaitsev, Altinity CTO

Introduction to Apache Spark

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Similar to Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Benchmarking at Parse

Travis Redman

The document discusses Parse's process for benchmarking MongoDB upgrades by replaying recorded production workloads on test servers. They found a 33-75% drop in throughput when upgrading from 2.4.10 to 2.6.3 due to query planner bugs. Working with MongoDB, they identified and helped fix several bugs, improving performance in 2.6.5 but still below 2.4.10 levels initially. Further optimization work increased throughput above 2.4.10 levels when testing with more workers and operations.

Advanced Benchmarking at Parse

MongoDB

261197832 8-performance-tuning-part i

NaviSoft

The document describes the process of performance tuning for a database application. It presents a case study where a GUI screen was taking too long to load due to an inefficient query. The case study outlines identifying the problematic query, investigating solutions, replacing the query with a tuned version, and measuring the results. The document also provides an overview of database performance concepts like response time, wait time, and throughput. It describes different components involved in SQL processing like the parser, optimizer, row source generator, and execution plan.

BTV PHP - Building Fast Websites

Jonathan Klein

This document summarizes a presentation about optimizing server-side performance. It discusses measuring performance metrics like time to first byte, optimizing databases through techniques like adding indexes and reducing joins, using caching with Memcached and APC, choosing fast web servers like Nginx and Lighttpd, and using load testing tools like JMeter to test performance before deployment. The presentation was given by a senior engineer at Wayfair to discuss their experiences optimizing their platform.

[262] netflix 빅데이터 플랫폼

NAVER D2

This document summarizes a presentation about Netflix's big data platform and Spark. The key points are: 1. Netflix uses Apache Spark on YARN and Mesos clusters to process batch and streaming data from sources like Cassandra and Kafka. 2. Netflix has contributed improvements to Spark's dynamic resource allocation, predicate pushdown, and support for S3 filesystems. 3. A use case showed Spark outperforming Pig for an iterative job that duplicated and aggregated data in multiple steps.

Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC

Kristofferson A

This document summarizes the steps taken to diagnose and resolve a sudden slow down issue affecting applications running on a two node Real Application Clusters (RAC) environment. The troubleshooting process involved systematically measuring performance at the operating system, database, and session levels. Key findings included high wait times and fragmentation issues on the network interconnect, which were resolved by replacing the network switch. Measuring performance using tools like ASH, AWR, and OS monitoring was essential to systematically diagnose the problem.

Scaling habits of ASP.NET

David Giard

The document discusses the scaling habits of ASP.NET applications over multiple versions from initial launch to large-scale business success. As an application grows from version 1 with a few users to version N with thousands of users, the key scaling challenges change from fixing logical problems to addressing performance bottlenecks and high availability requirements. The solutions also evolve from simple code optimizations to sophisticated architectures with load balancing, caching, and separate servers for web and database tiers.

PGConf APAC 2018 - Tale from Trenches

PGConf APAC

About a year ago I was caught up in line-of-fire when a production system started behaving abruptly - A batch process which would finish in 15minutes started taking 1.5 hours - We started facing OLTP read queries on standby being cancelled - We faced a sudden slowness on the Primary server and we were forced to do a forceful switch to standby. We were able to figure out that some peculiarities of the application code and batch process were responsible for this. But we could not fix the application code (as it is packaged application). In this talk I would like to share more details of how we debugged, what was the problem we were facing and how we applied a work around for it. We also learnt that a query returning in 10minutes may not be as dangerous as a query returning in 10sec but executed 100s of times in an hour. I will share in detail- - How to map the process/top stats from OS with pg_stat_activity - How to get and read explain plan - How to judge if a query is costly - What tools helped us - A peculiar autovacuum/vacuum Vs Replication conflict we ran into - Various parameters to tune autvacuum and auto-analyze process - What we have done to work-around the problem - What we have put in place for better monitoring and information gathering

Ansible benelux meetup - Amsterdam 27-5-2015

Pavel Chunyayev

1) Ansible is being used at Backbase to automate the provisioning of different server configurations for testing their Customer Experience Platform (CXP). 2) A REST API and UI allow users to easily provision new environments from available server stacks configured with Ansible for testing. 3) This enables Backbase to implement continuous delivery practices like automated testing of new versions without affecting production environments.

Continuous Application with FAIR Scheduler with Robert Xue

Databricks

This talk presents a continuous application example that relies on Spark FAIR scheduler as the conductor to orchestrate the entire “lambda architecture” in a single spark context. As a typical time series event stream analysis might involved, there are four key components: – an ETL step to store the raw data – a series of real time aggregation on the joint of streaming input and historical data to power a model – model execution – ad-hoc query for human inspection. The key benefits of this setup compared to a typical design that has a bunch of Spark application running individually are 1. Decouple streaming batches process from triggering model calculation, model calculations are triggered at a different pace from the stream processing. 2. Model is always processing the latest data, using pure rdd APIs. 3. Launch various operations in different threads on the driver node, ensuring them got submitted to the appropriate fair scheduler pool. Let FAIR scheduler to do the resource distribution. 4. Share code and time by sharing the actual data transformation (like the rdds in the intermediate steps). 5. Support adhoc queries on intermediate state without a dedicated serving layer or output protocol. 6. Only one app to monitor and tune.

Queick: A Simple Job Queue System for Python

Ryota Suenaga

Ryota SUENAGA presented Queick, a simple job queue system he created for Python. Queick uses multi-threading to asynchronously execute jobs in the background. It was designed to be lightweight and use only Python's standard libraries. Key features include asynchronous and scheduled job execution, retries of failed jobs, and checking for network connectivity to re-enqueue jobs if the connection is lost. The architecture includes a job queue, worker processes to run jobs, and a separate process to monitor network status and retry downed jobs when the connection returns.

Background processing with hangfire

Aleksandar Bozinovski

Hangfire An easy way to perform background processing in .NET and .NET Core applications. No Windows Service or separate process required. Why Background Processing? Lengthy operations like updating lot of records in DB Checking every 2 hours for new data or files Invoice generation at the end of every billing period Monthly Reporting Rebuild data, indexes or search-optimized index after data change Automatic subscription renewal Regular Mailings Send an email due to an action Background service provisioning

Give your little scripts big wings: Using cron in the cloud with Amazon Simp...

Amazon Web Services

Most developers write them and every company has them – a vast library of small and large scripts that are designed to run on a scheduled basis. These background angels help keep the lights on and the doors open. They’ve been built up over time and are forgotten little heroes that are only remembered when the machines they live on fail. They are scattered throughout a company’s IT infrastructure and do important things. In this session, we will explain how to use Ruby on Simple Workflow to quickly build a system that schedules scripts, runs them on time, retries them if they fail, and stores the history of their execution. You will walk away from this session with an understanding of how Simple Workflow brings resiliency, concurrency, and tracking to your applications.

Performance testing as part of Agile - Continius Delivery solution

Sergey Radov

Batch Processing with Amazon EC2 Container Service

Amazon Web Services

improving the performance of Rails web Applications

John McCaffrey

This presentation is the first in a series on Improving Rails application performance. This session covers the basic motivations and goals for improving performance, the best way to approach a performance assessment, and a review of the tools and techniques that will yield the best results. Tools covered include: Firebug, yslow, page speed, speed tracer, dom monster, request log analyzer, oink, rack bug, new relic rpm, rails metrics, showslow.org, msfast, webpagetest.org and gtmetrix.org. The upcoming sessions will focus on: Improving sql queries, and active record use Improving general rails/ruby code Improving the front-end And a final presentation will cover how to be a more efficient and effective developer! This series will be compressed into a best of session for the 2010 http://windycityRails.org conference

Flink Forward SF 2017: Feng Wang & Zhijiang Wang - Runtime Improvements in Bl...

Flink Forward

In 2016, we introduced Alibaba’s compute engine Blink which was based on our private branch of flink. It enalbed many large scale applications in Alibaba’s core business, such as search, recommendation and ads. With the deep and close colaboration with the flink community, we are finally close to contribute our improvements back to the flink community. In this talk, we will present our key contributions to flink runtime recently, such as the new YARN cluster mode for Flip-6, fine-grained failover for Flip-1, async i/o for Flip-12, incremental checkpoint, and the further improvements plan from Alibaba in the near future. Moreover, we will show some production use cases to illustrate how flink works in Alibaba’s large scale online applications, which includes real-time ETL as well as online machine learning. This talk is presented by Alibaba.

Internals of Presto Service

Treasure Data, Inc.

Presto is a distributed SQL query engine that Treasure Data provides as a service. Taro Saito discussed the internals of the Presto service at Treasure Data, including how the TD Presto connector optimizes scan performance from storage systems and how the service manages multi-tenancy and resource allocation for customers. Key challenges in providing a database as a service were also covered, such as balancing cost and performance.

Celery: The Distributed Task Queue

Richard Leland

Celery is an asynchronous task queue/job queue based on distributed message passing. It allows tasks to be executed concurrently on one or more worker servers to minimize request times and offload intensive processes. Some key benefits are improved user experience through faster responses, scalability by adding more workers as needed, and flexibility through many customization points. Celery uses message brokers like RabbitMQ to handle task routing and can integrate with databases, caching, and other services.

Spring Batch Performance Tuning

Gunnar Hillert

In this presentation we will examine various scalability options in order to improve the robustness and performance of your Spring Batch applications. We start out with a single threaded Spring Batch application that we will refactor so we can demonstrate how to run it using: * Concurrent Steps * Remote Chunking * AsyncItemProcessor and AsyncItemWriter * Remote Partitioning Additionally, we will show how you can deploy Spring Batch applications to Spring XD which provides high availability and failover capabilities. Spring XD also allows you to integrate Spring Batch applications with other Big Data processing needs.

Similar to Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink (20)

Benchmarking at Parse

Advanced Benchmarking at Parse

261197832 8-performance-tuning-part i

BTV PHP - Building Fast Websites

[262] netflix 빅데이터 플랫폼

Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC

Scaling habits of ASP.NET

PGConf APAC 2018 - Tale from Trenches

Ansible benelux meetup - Amsterdam 27-5-2015

Continuous Application with FAIR Scheduler with Robert Xue

Queick: A Simple Job Queue System for Python

Background processing with hangfire

Give your little scripts big wings: Using cron in the cloud with Amazon Simp...

Performance testing as part of Agile - Continius Delivery solution

Batch Processing with Amazon EC2 Container Service

improving the performance of Rails web Applications

Flink Forward SF 2017: Feng Wang & Zhijiang Wang - Runtime Improvements in Bl...

Internals of Presto Service

Celery: The Distributed Task Queue

Spring Batch Performance Tuning

More from Databricks

DW Migration Webinar-March 2022.pptx

Databricks

The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.

Data Lakehouse Symposium | Day 1 | Part 1

Databricks

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 2

Databricks

Data Lakehouse Symposium | Day 2

Databricks

Data Lakehouse Symposium | Day 4

Databricks

The document discusses the challenges of modern data, analytics, and AI workloads. Most enterprises struggle with siloed data systems that make integration and productivity difficult. The future of data lies with a data lakehouse platform that can unify data engineering, analytics, data warehousing, and machine learning workloads on a single open platform. The Databricks Lakehouse platform aims to address these challenges with its open data lake approach and capabilities for data engineering, SQL analytics, governance, and machine learning.

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Databricks

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

Democratizing Data Quality Through a Centralized Platform

Databricks

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Learn to Use Databricks for Data Science

Databricks

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Why APM Is Not the Same As ML Monitoring

Databricks

Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications. As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored. In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Databricks

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

Stage Level Scheduling Improving Big Data and AI Integration

Databricks

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Databricks

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Databricks

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Databricks

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Sawtooth Windows for Feature Aggregations

Databricks

In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.

Re-imagine Data Monitoring with whylogs and Spark

Databricks

In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.

Raven: End-to-end Optimization of ML Prediction Queries

Databricks

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Processing Large Datasets for ADAS Applications using Apache Spark

Databricks

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Massive Data Processing in Adobe Using Delta Lake

Databricks

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Machine Learning CI/CD for Email Attack Detection

Databricks

Detecting advanced email attacks at scale is a challenging ML problem, particularly due to the rarity of attacks, adversarial nature of the problem, and scale of data. In order to move quickly and adapt to the newest threat we needed to build a Continuous Integration / Continuous Delivery pipeline for the entire ML detection stack. Our goal is to enable detection engineers and data scientists to make changes to any part of the stack including joined datasets for hydration, feature extraction code, detection logic, and develop/train ML models. In this talk, we discuss why we decided to build this pipeline, how it is used to accelerate development and ensure quality, and dive into the nitty-gritty details of building such a system on top of an Apache Spark + Databricks stack.

More from Databricks (20)

DW Migration Webinar-March 2022.pptx

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 4

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Democratizing Data Quality Through a Centralized Platform

Learn to Use Databricks for Data Science

Why APM Is Not the Same As ML Monitoring

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Stage Level Scheduling Improving Big Data and AI Integration

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Sawtooth Windows for Feature Aggregations

Re-imagine Data Monitoring with whylogs and Spark

Raven: End-to-end Optimization of ML Prediction Queries

Processing Large Datasets for ADAS Applications using Apache Spark

Massive Data Processing in Adobe Using Delta Lake

Machine Learning CI/CD for Email Attack Detection

Recently uploaded

Cal Girls The Lalit Jaipur 8445551418 Khusi Top Class Girls Call Jaipur Avail...

deepikakumaridk25

Big Data and Analytics Shaping the future of Payments

RuchiRathor2

Field Diary and lab record, Importance.pdf

hritikbui

Parcel Delivery - Intel Segmentation and Last Mile Opt.pptx

AltanAtabarut

Cal Girls Hotel Safari Jaipur | | Girls Call Free Drop Service

Deepikakumari457585

Combined supervised and unsupervised neural networks for pulse shape discrimi...

Samuel Jackson

Our methodology for pulse shape discrimination is split into two steps. Firstly, we learn a model to discriminate between pulses using "clean" low-rate examples by removing pile-up & saturated events. In addition to traditional tail sum discrimination, we investigate three different choices for discrimination between γ-pulses, fast, thermal neutrons. We consider clustering the pulses directly using Gaussian Mixture Modelling (GMM), using variational autoencoders to learn a representation of the pulses and then clustering the learned representation (VAE+GMM) and using density ratio estimation to discriminate between a mixed (γ + neutron) and pure (γ only) sources using a multi-layer perceptron (MLP) as a supervised learning problem. Secondly, we aim to classify and recover pile-up events in the < 150 ns regime by training a single unified multi-label MLP. To frame the problem as a multi-label supervised learning method, we first simulate pile-up events with known components. Then, using the simulated data and combining it with single event data, we train a final multi-label MLP to output a binary code indicating both how many and which type of events are present within an event window.

Data Storytelling Final Project for MBA 635

HeidiLivengood

Training on CSPro and step by steps.pptx

lenjisoHussein

Acid Base Practice Test 4- KEY.pdfkkjkjk

talha2khan2k

Full Disclosure Board Policy.docx BRGY LICUMA

brgylicumaormoccity

Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data

Samuel Jackson

We present our work to improve data accessibility and performance for data-intensive tasks within the fusion research community. Our primary goal is to develop services that facilitate efficient access for data-intensive applications while ensuring compliance with FAIR principles [1], as well as adoption of interoperable tools, methods and standards. The major outcome of our work is the successful creation and deployment of a data service for the MAST (Mega Ampere Spherical Tokamak) experiment [2], leading to substantial enhancements in data discoverability, accessibility, and overall data retrieval performance, particularly in scenarios involving large-scale data access. Our work follows the principles of Analysis-Ready, Cloud Optimised (ARCO) data [3] by using cloud optimised data formats for fusion data. Our system consists of a query-able metadata catalogue, complemented with an object storage system for publicly serving data from the MAST experiment. We will show how our solution integrates with the Pandata stack [4] to enable data analysis and processing at scales that would have previously been intractable, paving the way for data-intensive workflows running routinely with minimal pre-processing on the part of the researcher. By using a cloud-optimised file format such as zarr [5] we can enable interactive data analysis and visualisation while avoiding large data transfers. Our solution integrates with common python data analysis libraries for large, complex scientific data such as xarray [6] for complex data structures and dask [7] for parallel computation and lazily working with larger that memory datasets. The incorporation of these technologies is vital for advancing simulation, design, and enabling emerging technologies like machine learning and foundation models, all of which rely on efficient access to extensive repositories of high-quality data. Relying on the FAIR guiding principles for data stewardship not only enhances data findability, accessibility, and reusability, but also fosters international cooperation on the interoperability of data and tools, driving fusion research into new realms and ensuring its relevance in an era characterised by advanced technologies in data science. [1] Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) https://doi.org/10.1038/sdata.2016.18 [2] M Cox, The Mega Amp Spherical Tokamak, Fusion Engineering and Design, Volume 46, Issues 2–4, 1999, Pages 397-404, ISSN 0920-3796, https://doi.org/10.1016/S0920-3796(99)00031-9 [3] Stern, Charles, et al. "Pangeo forge: crowdsourcing analysis-ready, cloud optimized data production." Frontiers in Climate 3 (2022): 782909. [4] Bednar, James A., and Martin Durant. "The Pandata Scalable Open-Source Analysis Stack." (2023). [5] Alistair Miles (2024) ‘zarr-developers/zarr-python: v2.17.1’. Zenodo. doi: 10.5281/zenodo.10790679 [6] Hoyer, S. & Hamman, J., (20

PRODUCT | RESEARCH-PRESENTATION-1.1.pptx

amazenolmedojeruel

Where to order Frederick Community College diploma?

SomalyEng

Histology of Muscle types histology o.ppt

SamanArshad11

Data Analytics for Decision Making By District 11 Solutions

District 11 Solutions

Cal Girls Mansarovar Jaipur | 08445551418 | Rajni High Profile Girls Call in ...

femim26318

How AI is Revolutionizing Data Collection.pdf

PromptCloud

Artificial Intelligence (AI) is transforming the landscape of data collection, making it more efficient, accurate, and insightful than ever before. With AI, businesses can automate the extraction of vast amounts of data from diverse sources, analyze patterns in real-time, and gain deeper insights with minimal human intervention. This revolution in data collection enables companies to make faster, data-driven decisions, enhance their competitive edge, and unlock new opportunities for growth. AI-powered tools can handle complex and dynamic web content, adapt to changes in website structures, and even understand the context of data through natural language processing. This means that data collection is not only faster but also more precise, reducing the time and effort required for manual data extraction. Furthermore, AI can process unstructured data, such as social media posts and customer reviews, providing valuable insights into customer sentiment and market trends. Embrace the future of data collection with AI and stay ahead of the curve. Learn more about how PromptCloud’s AI-driven web scraping solutions can transform your data strategy. https://www.promptcloud.com/contact/

Getting Started with Interactive Brokers API and Python.pdf

Riya Sen

In the fast-paced world of finance, automation is key to staying ahead of the curve. Traders and investors are increasingly turning to programming languages like Python to streamline their strategies and enhance their decision-making processes. In this blog post, we will delve into the integration of Python with Interactive Brokers, one of the leading brokerage platforms, and explore how this dynamic duo can revolutionize your trading experience.

CT AnGIOGRAPHY of pulmonary embolism.pptx

RejoJohn2

DESIGN AND DEVELOPMENT OF AUTO OXYGEN CONCENTRATOR WITH SOS ALERT FOR HIKING ...

JeevanKp7

Long-term oxygen therapy (LTOT) and novel techniques of evaluating treatment efficacy have enhanced the quality of life and decreased healthcare expenses for COPD patients. The cost of a pulmonary blood gas test is comparable to the cost of two days of oxygen therapy and the cost of a hospital stay is equivalent to the cost of one month of oxygen therapy, long-term oxygen therapy (LTOT) is a cost-effective technique of treating this disease. A small number of clinical investigations on LTOT have shown that it improves the quality of life of COPD patients by reducing the loss of their respiratory capacity. A study of 8487 Danish patients found that LTOT for 1524 hours per day extended life expectancy from 1.07 to 1.40 years.

Recently uploaded (20)

Cal Girls The Lalit Jaipur 8445551418 Khusi Top Class Girls Call Jaipur Avail...

Big Data and Analytics Shaping the future of Payments

Field Diary and lab record, Importance.pdf

Parcel Delivery - Intel Segmentation and Last Mile Opt.pptx

Cal Girls Hotel Safari Jaipur | | Girls Call Free Drop Service

Combined supervised and unsupervised neural networks for pulse shape discrimi...

Data Storytelling Final Project for MBA 635

Training on CSPro and step by steps.pptx

Acid Base Practice Test 4- KEY.pdfkkjkjk

Full Disclosure Board Policy.docx BRGY LICUMA

Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data

PRODUCT | RESEARCH-PRESENTATION-1.1.pptx

Where to order Frederick Community College diploma?

Histology of Muscle types histology o.ppt

Data Analytics for Decision Making By District 11 Solutions

Cal Girls Mansarovar Jaipur | 08445551418 | Rajni High Profile Girls Call in ...

How AI is Revolutionizing Data Collection.pdf

Getting Started with Interactive Brokers API and Python.pdf

CT AnGIOGRAPHY of pulmonary embolism.pptx

DESIGN AND DEVELOPMENT OF AUTO OXYGEN CONCENTRATOR WITH SOS ALERT FOR HIKING ...

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

1. Redis + Apache Spark = Swiss Army Knife meets Kitchen Sink Yeshwanth Vijayakumar Sr. Engineering Manager/Architect @ Adobe

2. Agenda § Niche 1 ▪ Long Running Spark Batch Job - Dispatch New Jobs by polling a Redis Queue § Niche 2 : ▪ Distributed Counters

3. • Niche 1 : Long Running Spark Batch Job

4. Problem Context

5. Run as many queries as possible in parallel on top a denormalized dataframe • foo = 1 Query 1 • bar.baz > 120 Query 2 • state in [CA, NY] Query 3 Query 1000 ProfileIds field1 field1000 eventsArray a@a.com a x [e1,2,3] b@g.com b x [e1] d@d.com d y [e1,2,3] z@z.com z y [e1,2,3,5,7]

6. What do we need? • Long Running Spark Batch Job • Dispatch New Jobs by polling a Redis Queue • We want to parametrize a Spark Action repeatedly for interactive results • E.g. Submit custom queries on top a table • We load the data once query N times • Bringing up a Spark Cluster per job has a latency cost • Wasted time doing same initialization actions multiple times. • Possible Multi tenancy

7. Why not Apache Livy et. al?

8. Why not Structured Streaming? • Lack of access to Spark Context within executor context • Can’t do a spark action on top of dataframe that is already loaded in the driver unless you do a join • Doing a join is extremely limited

9. Working Solution Summary • Blocking POP on Redis inside driver and use Command Pattern to send queries to rediscover queue • Consume the commands and trigger spark actions using a FAIR scheduler • Communicate status of job through a micro service/database or Redis itself!

10. Session Workflow – Spark Continuous Session 10 Submit Query API Spark Driver Executor 1 Executor N Fetch Results Executor Logic API 1. POST /preview 2. Check if result in Cache 1. GET /preview/<previewID> 2. Fetch Counters from Redis 3. Push <query> into queue 4. Pop queries till queue is empty [q1, q2, q3, q100] Sample Dataframe Sample Dataframe partition 1 partition 2 partition 1

11. Working Solution – Code View

12. Working Solution – Code View

13. Working Solution – Code View

14. • Niche 2 : Distributed Counters

15. What is wrong with Accumulators? • Repeated Task Execution - Non idempotency • Task Failures and Retries • Re-using stage in repeated operations • Speculative Execution • Memory pressure on driver on collect() • Can’t access per partition stats programmatically AFAIK

16. What is wrong with Accumulators? - Example

17. Utilize Redis Hashes as distributed counters

18. Utilize Redis Hashes as distributed counters

19. Excellent Throughput

20. Digging into Redis Pipelining + Spark From https://redis.io/topics/pipelining Without Pipelining With Pipelining

22. Important Config Optimizations Off-Heap Allocation

23. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Related slideshows

More Related Content

What's hot

What's hot (20)

Similar to Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Similar to Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink (20)

More from Databricks

More from Databricks (20)

Recently uploaded

Recently uploaded (20)

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink