This document discusses common mistakes made when writing Spark applications and provides recommendations to address them. It covers issues like having executors that are too small or large, shuffle blocks exceeding size limits, data skew slowing jobs, and excessive stages. The key recommendations are to optimize executor and partition sizes, increase partitions to reduce skew, use techniques like salting to address skew, and favor transformations like ReduceByKey over GroupByKey to minimize shuffles and memory usage.
The document discusses Spark job failures and Spark/YARN architecture. It describes a Spark job failure due to a task failing 4 times with a NumberFormatException when parsing a string. It then explains that Spark jobs are divided into stages made up of tasks, and the entire job fails if a stage fails. The document also provides an overview of the Spark and YARN architectures, showing how Spark jobs are submitted to and run via the YARN resource manager.
Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital KediaDatabricks
Apache Spark is a fast and flexible compute engine for a variety of diverse workloads. Optimizing performance for different applications often requires an understanding of Spark internals and can be challenging for Spark application developers. In this session, learn how Facebook tunes Spark to run large-scale workloads reliably and efficiently. The speakers will begin by explaining the various tools and techniques they use to discover performance bottlenecks in Spark jobs. Next, you’ll hear about important configuration parameters and their experiments tuning these parameters on large-scale production workload. You’ll also learn about Facebook’s new efforts towards automatically tuning several important configurations based on nature of the workload. The speakers will conclude by sharing their results with automatic tuning and future directions for the project.ing several important configurations based on nature of the workload. We will conclude by sharing our result with automatic tuning and future directions for the project.
Apache Spark presentation at HasGeek FifthElelephant
https://fifthelephant.talkfunnel.com/2015/15-processing-large-data-with-apache-spark
Covering Big Data Overview, Spark Overview, Spark Internals and its supported libraries
Deep Dive: Memory Management in Apache SparkDatabricks
Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.
Using Apache Spark to analyze large datasets in the cloud presents a range of challenges. Different stages of your pipeline may be constrained by CPU, memory, disk and/or network IO. But what if all those stages have to run on the same cluster? In the cloud, you have limited control over the hardware your cluster runs on.
You may have even less control over the size and format of your raw input files. Performance tuning is an iterative and experimental process. It’s frustrating with very large datasets: what worked great with 30 billion rows may not work at all with 400 billion rows. But with strategic optimizations and compromises, 50+ TiB datasets can be no big deal.
By using Spark UI and simple metrics, explore how to diagnose and remedy issues on jobs:
Sizing the cluster based on your dataset (shuffle partitions)
Ingestion challenges – well begun is half done (globbing S3, small files)
Managing memory (sorting GC – when to go parallel, when to go G1, when offheap can help you)
Shuffle (give a little to get a lot – configs for better out of box shuffle) – Spill (partitioning for the win)
Scheduling (FAIR vs FIFO, is there a difference for your pipeline?)
Caching and persistence (it’s the cost of doing business, so what are your options?)
Fault tolerance (blacklisting, speculation, task reaping)
Making the best of a bad deal (skew joins, windowing, UDFs, very large query plans)
Writing to S3 (dealing with write partitions, HDFS and s3DistCp vs writing directly to S3)
This document discusses Spark shuffle, which is an expensive operation that involves data partitioning, serialization/deserialization, compression, and disk I/O. It provides an overview of how shuffle works in Spark and the history of optimizations like sort-based shuffle and an external shuffle service. Key concepts discussed include shuffle writers, readers, and the pluggable block transfer service that handles data transfer. The document also covers shuffle-related configuration options and potential future work.
Cosco: An Efficient Facebook-Scale Shuffle ServiceDatabricks
Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).
Performance Troubleshooting Using Apache Spark MetricsDatabricks
Luca Canali, a data engineer at CERN, presented on performance troubleshooting using Apache Spark metrics at the UnifiedDataAnalytics #SparkAISummit. CERN runs large Hadoop and Spark clusters to process over 300 PB of data from the Large Hadron Collider experiments. Luca discussed how to gather, analyze, and visualize Spark metrics to identify bottlenecks and improve performance.
How to Actually Tune Your Spark Jobs So They WorkIlya Ganelin
This document summarizes a USF Spark workshop that covers Spark internals and how to optimize Spark jobs. It discusses how Spark works with partitions, caching, serialization and shuffling data. It provides lessons on using less memory by partitioning wisely, avoiding shuffles, using the driver carefully, and caching strategically to speed up jobs. The workshop emphasizes understanding Spark and tuning configurations to improve performance and stability.
EMR Spark tuning involves configuring Spark and YARN parameters like executor memory and cores to optimize performance. The default Spark configurations depend on the deployment method (Thrift, Zeppelin etc). YARN is used for resource management in cluster mode, and allocates resources to containers based on minimum and maximum thresholds. When tuning, factors like available cluster resources, executor instances and cores should be considered to avoid overcommitting resources.
Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...Databricks
This document discusses best practices for optimizing Apache Spark applications. It covers techniques for speeding up file loading, optimizing file storage and layout, identifying bottlenecks in queries, dealing with many partitions, using datasource tables, managing schema inference, file types and compression, partitioning and bucketing files, managing shuffle partitions with adaptive execution, optimizing unions, using the cost-based optimizer, and leveraging the data skipping index. The presentation aims to help Spark developers apply these techniques to improve performance.
In Spark SQL the physical plan provides the fundamental information about the execution of the query. The objective of this talk is to convey understanding and familiarity of query plans in Spark SQL, and use that knowledge to achieve better performance of Apache Spark queries. We will walk you through the most common operators you might find in the query plan and explain some relevant information that can be useful in order to understand some details about the execution. If you understand the query plan, you can look for the weak spot and try to rewrite the query to achieve a more optimal plan that leads to more efficient execution.
The main content of this talk is based on Spark source code but it will reflect some real-life queries that we run while processing data. We will show some examples of query plans and explain how to interpret them and what information can be taken from them. We will also describe what is happening under the hood when the plan is generated focusing mainly on the phase of physical planning. In general, in this talk we want to share what we have learned from both Spark source code and real-life queries that we run in our daily data processing.
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Databricks
The document discusses optimizations made to Spark SQL performance when working with Parquet files at ByteDance. It describes how Spark originally reads Parquet files and identifies two main areas for optimization: Parquet filter pushdown and the Parquet reader. For filter pushdown, sorting columns improved statistics and reduced data reads by 30%. For the reader, splitting it to first filter then read other columns prevented loading unnecessary data. These changes improved Spark SQL performance at ByteDance without changing jobs.
Top 5 mistakes when writing Spark applicationshadooparchbook
This document discusses common mistakes people make when writing Spark applications and provides recommendations to address them. It covers issues related to executor configuration, application failures due to shuffle block sizes exceeding limits, slow jobs caused by data skew, and managing the DAG to avoid excessive shuffles and stages. Recommendations include using smaller executors, increasing the number of partitions, addressing skew through techniques like salting, and preferring ReduceByKey over GroupByKey and TreeReduce over Reduce to improve performance and resource usage.
Fine Tuning and Enhancing Performance of Apache Spark JobsDatabricks
Apache Spark defaults provide decent performance for large data sets but leave room for significant performance gains if able to tune parameters based on resources and job.
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc.
The document discusses 5 common mistakes people make when writing Spark applications:
1) Not properly sizing executors for memory and cores.
2) Having shuffle blocks larger than 2GB which can cause jobs to fail.
3) Not addressing data skew which can cause joins and shuffles to be very slow.
4) Not properly managing the DAG to minimize shuffles and stages.
5) Classpath conflicts from mismatched dependencies causing errors.
This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
The Parquet Format and Performance Optimization OpportunitiesDatabricks
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Top 5 mistakes when writing Spark applicationsmarkgrover
This document discusses 5 common mistakes people make when writing Spark applications.
The first mistake is improperly sizing Spark executors by not considering factors like the number of cores, amount of memory, and overhead needed. The second mistake is running into the 2GB limit on Spark shuffle blocks, which can cause jobs to fail. The third mistake is not addressing data skew during joins and shuffles, which can cause some tasks to be much slower than others. The fourth mistake is poorly managing the DAG by overusing shuffles, not using techniques like ReduceByKey instead of GroupByKey, and not using complex data types. The fifth mistake is classpath conflicts between the versions of libraries used by Spark and those added by the user.
Top 5 mistakes when writing Spark applicationsmarkgrover
This is a talk given at Advanced Spark meetup in San Francisco (http://www.meetup.com/Advanced-Apache-Spark-Meetup/events/223668878/). It focusses on common mistakes when writing Spark applications and how to avoid them.
Top 5 Mistakes When Writing Spark Applications by Mark Grover and Ted MalaskaSpark Summit
The document discusses 5 common mistakes people make when writing Spark applications:
1) Not properly sizing executors for memory and cores.
2) Having shuffle blocks larger than 2GB which can cause jobs to fail.
3) Not addressing data skew which can cause joins and shuffles to be very slow.
4) Not properly managing the DAG to minimize shuffles and stages.
5) Classpath conflicts from conflicting dependencies like Guava which can cause errors.
This document provides tips and best practices for optimizing Apache Spark performance and resource allocation. It discusses:
- The components of Spark including executors, drivers, and tasks
- Configuring Spark on YARN and dynamic resource allocation
- Optimizing memory usage, avoiding data skew, and reducing serialization costs
- Best practices for Spark Streaming around microbatching, fault tolerance, and performance
- Recommendations for running Spark on cloud object stores like S3
Hadoop World 2011: Hadoop Troubleshooting 101 - Kate Ting - ClouderaCloudera, Inc.
Attend this session and walk away armed with solutions to the most common customer problems. Learn proactive configuration tweaks and best practices to keep your cluster free of fetch failures, job tracker hangs, and the like.
This document discusses improving Spark performance on many-core machines by implementing an in-memory shuffle. It finds that Spark's disk-based shuffle is inefficient on such hardware due to serialization, I/O contention, and garbage collection overhead. An in-memory shuffle avoids these issues by copying data directly between memory pages. This results in a 31% median performance improvement on TPC-DS queries compared to the default Spark shuffle. However, more work is needed to address other performance bottlenecks and extend the in-memory shuffle to multi-node clusters.
Why the Address Translation Scheme Matters?Jiaqing Du
This document discusses address translation in memory and its impact on performance. It first describes the multi-dimensional structure of DRAM devices and how logical addresses map to physical locations. An experiment is then described that revealed the memory translation scheme by observing the impact of small offsets on throughput. The results help explain an observed imbalance between small and large packets. Potential solutions proposed include introducing random offsets in the network adapter driver or maintaining buffers of varying sizes in the memory pool.
Presented at Spark+AI Summit Europe 2019
https://databricks.com/session_eu19/apache-spark-at-scale-in-the-cloud
Using Apache Spark to analyze large datasets in the cloud presents a range of challenges. Different stages of your pipeline may be constrained by CPU, memory, disk and/or network IO. But what if all those stages have to run on the same cluster? In the cloud, you have limited control over the hardware your cluster runs on.
You may have even less control over the size and format of your raw input files. Performance tuning is an iterative and experimental process. It’s frustrating with very large datasets: what worked great with 30 billion rows may not work at all with 400 billion rows. But with strategic optimizations and compromises, 50+ TiB datasets can be no big deal.
By using Spark UI and simple metrics, explore how to diagnose and remedy issues on jobs:
Sizing the cluster based on your dataset (shuffle partitions)
Ingestion challenges – well begun is half done (globbing S3, small files)
Managing memory (sorting GC – when to go parallel, when to go G1, when offheap can help you)
Shuffle (give a little to get a lot – configs for better out of box shuffle) – Spill (partitioning for the win)
Scheduling (FAIR vs FIFO, is there a difference for your pipeline?)
Caching and persistence (it’s the cost of doing business, so what are your options?)
Fault tolerance (blacklisting, speculation, task reaping)
Making the best of a bad deal (skew joins, windowing, UDFs, very large query plans)
The document discusses dense linear algebra solvers and algorithms. It provides an overview of existing software for dense linear algebra including LINPACK, EISPACK, LAPACK, ScaLAPACK, PLASMA, and MAGMA. It then discusses challenges with dense linear algebra on modern hardware including distributed memory, heterogeneity, and the high cost of communication. It introduces tile algorithms as an approach to address these challenges compared to traditional LAPACK algorithms.
Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)Spark Summit
This document discusses lessons learned from working with Spark's machine learning library (ML Lib) for collaborative filtering on a large dataset. It covers four main lessons:
1. Spark uses more memory than expected due to JVM overhead, metadata for shuffles and jobs, and Scala vs Java. This can be addressed through careful partitioning, serialization with Kryo, and cleaning up long-running jobs.
2. Shuffles between nodes are expensive and can cause out of memory errors, so it is best to avoid them by using the driver for collection, broadcast variables, and accumulators.
3. Sending data through the driver has memory limits, so partitions and akka frame sizes must be configured based
Bharath Mundlapudi presented on Disk Fail Inplace in Hadoop. He discussed how a single disk failure currently causes an entire node to be blacklisted. With newer hardware trends of more disks per node, this wastes significant resources. His team developed a Disk Fail Inplace approach where Hadoop can tolerate disk failures until a threshold. This included separating critical and user files, handling failures at startup and runtime in DataNode and TaskTracker, and rigorous testing of the new approach.
This document discusses common mistakes made when implementing Oracle Exadata systems. It describes improperly sized SGAs which can hurt performance on data warehouses. It also discusses issues like not using huge pages, over or under use of indexing, too much parallelization, selecting the wrong disk types, failing to patch systems, and not implementing tools like Automatic Service Request and exachk. The document provides guidance on optimizing these areas to get the best performance from Exadata.
Using apache spark for processing trillions of records each day at DatadogVadim Semenov
This document discusses Datadog's use of Apache Spark to process trillions of records daily. It describes their initial Spark setup using AWS EMR with large clusters. It then covers common out of memory errors, measuring memory usage, handling spot instances, and lessons learned around monitoring jobs and ensuring resilience.
The document summarizes the Hoard memory allocator designed for multithreaded applications. Hoard aims to provide fast, scalable memory allocation that avoids fragmentation and false sharing. It uses per-processor heaps to avoid contention and a global heap to balance memory usage. Superblocks of same-sized objects are allocated from the OS in pages and objects within are handed out with low synchronization. Experimental results show Hoard outperforms other allocators on speed, scalability, fragmentation and false sharing avoidance.
This document provides performance optimization tips for Hadoop jobs, including recommendations around compression, speculative execution, number of maps/reducers, block size, sort size, JVM tuning, and more. It suggests how to configure properties like mapred.compress.map.output, mapred.map/reduce.tasks.speculative.execution, and dfs.block.size based on factors like cluster size, job characteristics, and data size. It also identifies antipatterns to avoid like processing thousands of small files or using many maps with very short runtimes.
London Spark Meetup Project Tungsten Oct 12 2015Chris Fregly
Building on a previous talk about how Spark beat Hadoop @ 100TB Daytona GraySort, we present low-level details of Project Tungsten which includes many CPU and Memory optimizations.
This document describes Pigasus, a system that achieves 100Gbps intrusion prevention on a single server using an FPGA-based smart NIC. It represents an order of magnitude improvement over state-of-the-art systems that require 4-21 servers. Pigasus utilizes a novel "FPGA-first" architecture where the FPGA is the main processing unit, addressing memory constraints through hierarchical multi-string pattern matching. Evaluation shows Pigasus needs only 1 server and 16 cores compared to 4-21 servers for state-of-the-art systems, resulting in much lower costs and power consumption to achieve the same throughput.
Performance Scenario: Diagnosing and resolving sudden slow down on two node RACKristofferson A
This document summarizes the steps taken to diagnose and resolve a sudden slow down issue affecting applications running on a two node Real Application Clusters (RAC) environment. The troubleshooting process involved systematically measuring performance at the operating system, database, and session levels. Key findings included high wait times and fragmentation issues on the network interconnect, which were resolved by replacing the network switch. Measuring performance using tools like ASH, AWR, and OS monitoring was essential to systematically diagnose the problem.
Adaptive Query Execution: Speeding Up Spark SQL at RuntimeDatabricks
Over the years, there has been extensive and continuous effort on improving Spark SQL’s query optimizer and planner, in order to generate high quality query execution plans. One of the biggest improvements is the cost-based optimization framework that collects and leverages a variety of data statistics (e.g., row count, number of distinct values, NULL values, max/min values, etc.) to help Spark make better decisions in picking the most optimal query plan.
Similar to Top 5 mistakes when writing Spark applications (20)
Architecting a next-generation data platformhadooparchbook
This document discusses a high-level architecture for analyzing taxi trip data in real-time and batch using Apache Hadoop and streaming technologies. The architecture includes ingesting data from multiple sources using Kafka, processing streaming data using stream processing engines, storing data in data stores like HDFS, and enabling real-time and batch querying and analytics. Key considerations discussed are choosing data transport and stream processing technologies, scaling and reliability, and processing both streaming and batch data.
Top 5 mistakes when writing Streaming applicationshadooparchbook
This document discusses 5 common mistakes when writing streaming applications and provides solutions. It covers: 1) Not shutting down apps gracefully by using thread hooks or external markers to stop processing after batches finish. 2) Assuming exactly-once semantics when things can fail at multiple points requiring offsets and idempotent operations. 3) Using streaming for everything when batch processing is better for some goals. 4) Not preventing data loss by enabling checkpointing and write-ahead logs. 5) Not monitoring jobs by using tools like Spark Streaming UI, Graphite and YARN cluster mode for automatic restarts.
Architecting a Next Generation Data Platformhadooparchbook
This document discusses a presentation on architecting Hadoop application architectures for a next generation data platform. It provides an overview of the presentation topics which include a case study on using Hadoop for an Internet of Things and entity 360 application. It introduces the key components of the proposed high level architecture including ingesting streaming and batch data using Kafka and Flume, stream processing with Kafka streams and storage in Hadoop.
What no one tells you about writing a streaming apphadooparchbook
This document discusses 5 things that are often not addressed when writing streaming applications:
1. Managing and monitoring long-running streaming jobs can be challenging as frameworks were not originally designed for streaming workloads. Options include using cluster mode to ensure jobs continue if clients disconnect and leveraging monitoring tools to track metrics.
2. Preventing data loss requires different approaches depending on the data source. File and receiver-based sources benefit from checkpointing while Kafka's commit log ensures data is not lost.
3. Spark Streaming is well-suited for tasks involving windowing, aggregations, and machine learning but may not be needed for all streaming use cases.
4. Achieving exactly-once semantics requires techniques
Architecting next generation big data platformhadooparchbook
A tutorial on architecting next generation big data platform by the authors of O'Reilly's Hadoop Application Architectures book. This tutorial discusses how to build a customer 360 (or entity 360) big data application.
Audience: Technical.
Hadoop application architectures - using Customer 360 as an examplehadooparchbook
Hadoop application architectures - using Customer 360 (more generally, Entity 360) as an example. By Ted Malaska, Jonathan Seidman and Mark Grover at Strata + Hadoop World 2016 in NYC.
The document discusses best practices for streaming applications. It covers common streaming use cases like ingestion, transformations, and counting. It also discusses advanced streaming use cases that involve machine learning. The document provides an overview of streaming architectures and compares different streaming engines like Spark Streaming, Flink, Storm, and Kafka Streams. It discusses when to use different storage systems and message brokers like Kafka for ingestion pipelines. The goal is to understand common streaming use cases and their architectures.
This document discusses a presentation on fraud detection application architectures using Hadoop. It provides an overview of different fraud use cases and challenges in implementing Hadoop-based solutions. Requirements for the applications include handling high volumes, velocities and varieties of data, generating real-time alerts with low latency, and performing both stream and batch processing. A high-level architecture is proposed using Hadoop, HBase, HDFS, Kafka and Spark to meet the requirements. Storage layer choices and considerations are also discussed.
Building a fraud detection application using the tools in the Hadoop ecosystem. Presentation given by authors of O'Reilly's Hadoop Application Architectures book at Strata + Hadoop World in San Jose, CA 2016.
This document discusses a case study on fraud detection using Hadoop. It begins with an overview of fraud detection requirements, including the need for real-time and near real-time processing of large volumes and varieties of data. It then covers considerations for the system architecture, including using HDFS and HBase for storage, Kafka for ingestion, and Spark and Storm for stream and batch processing. Data modeling with HBase and caching options are also discussed.
Architecting applications with Hadoop - Fraud Detectionhadooparchbook
This document discusses architectures for fraud detection applications using Hadoop. It provides an overview of requirements for such an application, including the need for real-time alerts and batch processing. It proposes using Kafka for ingestion due to its high throughput and partitioning. HBase and HDFS would be used for storage, with HBase better supporting random access for profiles. The document outlines using Flume, Spark Streaming, and HBase for near real-time processing and alerting on incoming events. Batch processing would use HDFS, Impala, and Spark. Caching profiles in memory is also suggested to improve performance.
The document discusses real-time fraud detection patterns and architectures. It provides an overview of key technologies like Kafka, Flume, and Spark Streaming used for real-time event processing. It then describes a high-level architecture involving ingesting events through Flume and Kafka into Spark Streaming for real-time processing, with results stored in HBase, HDFS, and Solr. The document also covers partitioning strategies, micro-batching, complex topologies, and ingestion of real-time and batch data.
The document discusses architectural considerations for Hadoop applications based on a case study of clickstream analysis. It covers requirements for data ingestion, storage, processing, and orchestration. For data storage, it recommends storing raw clickstream data in HDFS using the Avro file format with Snappy compression. For processed data, it recommends using the Parquet columnar storage format to enable efficient analytical queries. The document also discusses partitioning strategies and HDFS directory layout design.
The document provides an agenda and slides for a presentation on architectural considerations for data warehousing with Hadoop. The presentation discusses typical data warehouse architectures and challenges, how Hadoop can complement existing architectures, and provides an example use case of implementing a data warehouse with Hadoop using the Movielens dataset. Key aspects covered include ingestion of data from various sources using tools like Flume and Sqoop, data modeling and storage formats in Hadoop, processing the data using tools like Hive and Spark, and exporting results to a data warehouse.
Hadoop Application Architectures tutorial at Big DataService 2015hadooparchbook
This document outlines a presentation on architectural considerations for Hadoop applications. It introduces the presenters who are experts from Cloudera and contributors to Apache Hadoop projects. It then discusses a case study on clickstream analysis, how this was challenging before Hadoop due to data storage limitations, and how Hadoop provides a better solution by enabling active archiving of large volumes and varieties of data at scale. Finally, it covers some of the challenges in implementing Hadoop, such as choices around storage managers, data modeling and file formats, data movement workflows, metadata management, and data access and processing frameworks.
Architectural considerations for Hadoop Applicationshadooparchbook
The document discusses architectural considerations for Hadoop applications using a case study on clickstream analysis. It covers requirements for data ingestion, storage, processing, and orchestration. For data storage, it considers HDFS vs HBase, file formats, and compression formats. SequenceFiles are identified as a good choice for raw data storage as they allow for splittable compression.
This document discusses application architectures using Hadoop. It provides an example case study of clickstream analysis. It covers challenges of Hadoop implementation and various architectural considerations for data storage and modeling, data ingestion, and data processing. For data processing, it discusses different processing engines like MapReduce, Pig, Hive, Spark and Impala. It also discusses what specific processing needs to be done for the clickstream data like sessionization and filtering.
Manufacturing is the process of converting raw materials into finished goods through various production methods. Historically, manufacturing occurred on a small scale through apprenticeships or putting-out systems, but the Industrial Revolution led to large-scale manufacturing using machines powered by steam engines
internship project presentation for reference.pptxSaieJadhav1
Purpose :- Debian-based Linux distribution for advanced Penetration Testing and Security Auditing.- Developed by Offensive Security, known for security training.- Released in March 2013, a rebuild of BackTrack Linux following Debian standards.Maintained by Offensive Security:- Offensive Security oversees Kali Linux, ensuring it includes the latest security tools.- Known for certifications like OSCP (Offensive Security Certified Professional).Usage:- Penetration Testing: Simulates cyber-attacks to find system vulnerabilities.- Security Research: Helps researchers understand exploits and security techniques.- Computer Forensics: Used in digital crime investigations and data recovery.
Predicting damage in notched functionally graded materials plates thr...Barhm Mohamad
Presently, Functionally Graded Materials (FGMs) are extensively utilised in several industrial sectors, and the modelling of their mechanical behaviour is consistently advancing. Most studies investigate the impact of layers on the mechanical characteristics, resulting in a discontinuity in the material. In the present study, the extended Finite Element Method (XFEM) technique is used to analyse the damage in a Metal/Ceramic plate (FGM-Al/SiC) with a circular central notch. The plate is subjected to a uniaxial tensile force. The maximum stress criterion was employed for fracture initiation and the energy criterion for its propagation and evolution. The FGM (Al/SiC) structure is graded based on its thickness using a modified power law. The plastic characteristics of the structure were estimated using the Tamura-Tomota-Ozawa (TTO) model in a user-defined field variables (USDFLD) subroutine. Validation of the numerical model in the form of a stress-strain curve with the findings of the experimental tests was established following a mesh sensitivity investigation and demonstrated good convergence. The influence of the notch dimensions and gradation exponent on the structural response and damage development was also explored. Additionally, force-displacement curves were employed to display the data, highlighting the fracture propagation pattern within the FGM structure.
Maintaining asset integrity, maximising performance, and reaching peak production outcomes all depend on effective defect elimination management.
The difficulties of locating, comprehending, and resolving asset flaws will be examined in this article, with a focus on the significance of having a complete awareness of their nature, implications, and possible outcomes.
The significance of maintenance departments creating a framework for integrating defect elimination and bad actor analysis into their proactive maintenance strategies, enabling them to make more informed decisions, becomes clear when we delve into the subtleties of defect analysis.
Defects in an asset are characterised as flaws, inadequacies, or departures from standard operating characteristics or design specifications.
The kind of defect, where it is located on the assembly of the asset, and what happens when it exists determine how one defect impacts the performance of the asset as a whole.
For example, a minor corrosion patch on a panel that is readily replaceable and does not pose a structural risk to the asset overall is unlikely to have an adverse effect on the asset's longevity, performance, or safety.
Effective Defect Elimination Management is critical for maintaining asset integrity, optimizing performance, and achieving peak production results. It involves identifying, understanding, and addressing asset defects systematically.
Understanding asset defects requires accurate identification and comprehensive documentation in the CMMS, including risk assessments that evaluate both the consequence and likelihood of defects leading to failures.
Defect Elimination Management (DEM) is a comprehensive approach that goes beyond traditional maintenance practices, focusing on root cause analysis and implementing long-term solutions to prevent defect recurrence.
"Bad Actors" in defect elimination refer to equipment, systems, or components that consistently underperform, require frequent maintenance, or cause repeated operational reliability or quality issues.
Advanced diagnostic tools and technologies, such as vibration analysis systems, infrared thermography, and AI-based analytics, have transformed the way asset defects are identified and managed.
Quality and timely repairs and clear business processes for managing defects are crucial, along with maintaining quality maintenance history data to provide valuable insights for future defect elimination processes.
To learn more, you can read my article via this link: https://www.cmmssuccess.com/defect-elimination-management/
Structural Dynamics and Earthquake Engineeringtushardatta
Slides are prepared with a lot of text material to help young teachers to teach the course for the first time. This also includes solved problems. This can be used to teach a first course on structural dynamics and earthquake engineering. The lecture notes based on which slides are prepared are available in SCRIBD.
8. 8
Decisions, decisions, decisions
• Number of executors (--num-executors)
• Cores for each executor (--executor-cores)
• Memory for each executor (--executor-memory)
• 6 nodes
• 16 cores each
• 64 GB of RAM
20. 20
#1 – Memory overhead
• --executor-memory controls the heap size
• Need some overhead (controlled by
spark.yarn.executor.memory.overhead) for off heap memory
• Default is max(384MB, .07 * spark.executor.memory)
23. 23
#3 HDFS Throughput
• 15 cores per executor can lead to bad HDFS I/O throughput.
• Best is to keep under 5 cores per executor
24. 24
Calculations
• 5 cores per executor
– For max HDFS throughput
• Cluster has 6 * 15 = 90 cores in total
after taking out Hadoop/Yarn daemon cores)
• 90 cores / 5 cores/executor
= 18 executors
• Each node has 3 executors
• 63 GB/3 = 21 GB, 21 x (1-0.07)
~ 19 GB
• 1 executor for AM => 17 executors
Overhead
Worker node
Executor 3
Executor 2
Executor 1
25. 25
Correct answer
• 17 executors in total
• 19 GB memory/executor
• 5 cores/executor
* Not etched in stone
Overhead
Worker node
Executor 3
Executor 2
Executor 1
26. 26
Dynamic allocation helps with though, right?
• Dynamic allocation allows Spark to dynamically scale the cluster
resources allocated to your application based on the workload.
• Works with Spark-On-Yarn
27. 27
Decisions with Dynamic Allocation
• Number of executors (--num-executors)
• Cores for each executor (--executor-cores)
• Memory for each executor (--executor-memory)
• 6 nodes
• 16 cores each
• 64 GB of RAM
28. 28
Read more
• From a great blog post on this topic by Sandy Ryza:
http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-
part-2/
30. 30
Application failure
15/04/16 14:13:03 WARN scheduler.TaskSetManager: Lost task 19.0 in stage
6.0 (TID 120, 10.215.149.47): java.lang.IllegalArgumentException: Size
exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:828) at
org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:123) at
org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:132) at
org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:517)
at
org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:432)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:618) at
org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:146)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
32. 32
Ok, what’s a shuffle block again?
• In MapReduce terminology, a file written from one Mapper for a Reducer
• The Reducer makes a local copy of this file (reducer local copy) and then
‘reduces’ it
33. 33
Defining shuffle and partition
Each yellow arrow
in this diagram
represents a shuffle
block.
Each blue block is a
partition.
35. 35
What’s going on here?
• Spark uses ByteBuffer as abstraction for blocks
val buf = ByteBuffer.allocate(length.toInt)
• ByteBuffer is limited by Integer.MAX_SIZE (2 GB)!
36. 36
Spark SQL
• Especially problematic for Spark SQL
• Default number of partitions to use when doing shuffles is 200
– This low number of partitions leads to high shuffle block size
37. 37
Umm, ok, so what can I do?
1. Increase the number of partitions
– Thereby, reducing the average partition size
2. Get rid of skew in your data
– More on that later
38. 38
Umm, how exactly?
• In Spark SQL, increase the value of
spark.sql.shuffle.partitions
• In regular Spark applications, use rdd.repartition() or
rdd.coalesce()(latter to reduce #partitions, if needed)
39. 39
But, how many partitions should I have?
• Rule of thumb is around 128 MB per partition
40. 40
But! There’s more!
• Spark uses a different data structure for bookkeeping during shuffles, when
the number of partitions is less than 2000, vs. more than 2000.
42. 42
Ok, so what are you saying?
If number of partitions < 2000, but not by much, bump it to be slightly higher
than 2000.
43. 43
Can you summarize, please?
• Don’t have too big partitions
–Your job will fail due to 2 GB limit
• Don’t have too few partitions
–Your job will be slow, not making using of parallelism
• Rule of thumb: ~128 MB per partition
• If #partitions < 2000, but close, bump to just > 2000
• Track SPARK-6235 for removing various 2 GB limits
45. 45
Slow jobs on Join/Shuffle
• Your dataset takes 20 seconds to run over with a map job, but take 4 hours
when joined or shuffled. What wrong?
46. 46
Mistake - Skew
Single Thread
Single Thread
Single Thread
Single Thread
Single Thread
Single Thread
Single Thread
Normal
Distributed
The Holy Grail of Distributed Systems
53. 53
• Two Stage Aggregation
– Stage one to do operations on the salted keys
– Stage two to do operation access unsalted key results
Mistake – Skew : Salting
Data Source Map
Convert to
Salted Key & Value
Tuple
Reduce
By Salted Key
Map Convert results
to
Key & Value
Tuple
Reduce
By Key
Results
54. 54
• Second Stage only required for Isolated Keys
Mistake – Skew : Isolated Salting
Data Source Map
Convert to
Key & Value
Isolate Key and
convert to
Salted Key & Value
Tuple
Reduce
By Key & Salted
Key
Filter Isolated
Keys
From Salted
Keys
Map Convert
results to
Key & Value
Tuple
Reduce
By Key
Union to Results
55. 55
• Filter Out Isolated Keys and use Map Join/Aggregate on
those
• And normal reduce on the rest of the data
• This can remove a large amount of data being shuffled
Mistake – Skew : Isolated Map Join
Data Source Filter Normal
Keys
From Isolated
Keys
Reduce
By Normal Key
Union to Results
Map Join
For Isolated
Keys
57. 57
Table YTable X
• How To fight Cartesian Join
– Nested Structures
Managing Parallelism
A, 1
A, 2
A, 3
A, 4
A, 5
A, 6
Table X
A, 1, 4
A, 2, 4
A, 3, 4
A, 1, 5
A, 2, 5
A, 3, 5
A, 1, 6
A, 2, 6
A, 3, 6
JOIN OR
Table X
A
A, 1
A, 2
A, 3
A, 4
A, 5
A, 6
60. 60
Out of luck?
• Do you every run out of memory?
• Do you every have more then 20 stages?
• Is your driver doing a lot of work?
61. 61
Mistake – DAG Management
• Shuffles are to be avoided
• ReduceByKey over GroupByKey
• TreeReduce over Reduce
• Use Complex/Nested Types
62. 62
Mistake – DAG Management: Shuffles
• Map Side reduction, where possible
• Think about partitioning/bucketing ahead of time
• Do as much as possible with a single shuffle
• Only send what you have to send
• Avoid Skew and Cartesians
63. 63
ReduceByKey over GroupByKey
• ReduceByKey can do almost anything that GroupByKey
can do
• Aggregations
• Windowing
• Use memory
• But you have more control
• ReduceByKey has a fixed limit of Memory requirements
• GroupByKey is unbound and dependent on data
64. 64
TreeReduce over Reduce
• TreeReduce & Reduce return some result to driver
• TreeReduce does more work on the executors
• While Reduce bring everything back to the driver
Partition
Partition
Partition
Partition
Driver
100%
Partition
Partition
Partition
Partition
Driver
4
25%
25%
25%
25%
65. 65
Complex Types
• Top N List
• Multiple types of Aggregations
• Windowing operations
• All in one pass
68. 68
Ever seen this?
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode;
at org.apache.spark.util.collection.OpenHashSet.org
$apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261)
at
org.apache.spark.util.collection.OpenHashSet$mcI$sp.getPos$mcI$sp(OpenHashSet.scala:165)
at
org.apache.spark.util.collection.OpenHashSet$mcI$sp.contains$mcI$sp(OpenHashSet.scala:102)
at
org.apache.spark.util.SizeEstimator$$anonfun$visitArray$2.apply$mcVI$sp(SizeEstimator.scala:214)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at
org.apache.spark.util.SizeEstimator$.visitArray(SizeEstimator.scala:210)
at…....
74. 74
5 Mistakes
• Size up your executors right
• 2 GB limit on Spark shuffle blocks
• Evil thing about skew and cartesians
• Learn to manage your DAG, yo!
• Do shady stuff, don’t let classpath leaks mess
you up