SlideShare a Scribd company logo

HDP2 and YARN operations point

•
•
Treasure Data, Inc.
Treasure Data, Inc.

This document summarizes Ryu Kobayashi's presentation on HDP2 and YARN operations. The presentation introduced YARN, the resource management framework in Hadoop 2.0, describing its architecture and how it differs from the previous MapReduce v1 framework. It highlighted important considerations for YARN resource management and potential bugs in older versions of Hadoop.

Related slideshows

Large Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraphLarge Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraph
 
My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
 
1 of 36
Download to read offline
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. HDP2 and YARN operations point Ryu Kobayashi Treasure Data Tech Talk 11 and 12 Mar 2015
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Who am I? • Ryu Kobayashi • @ryu_kobayashi • https://github.com/ryukobayashi • Treasure Data, Inc. • Software Engineer • Background • Hadoop, Cassandra, Machine Learning, ... • I developed Huahin(Hadoop) Framework. 
 http://huahinframework.org/
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. What is YARN?
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YARN(Yet Another Resource Negotiator) Architecture
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • MRv1 • JobTracker • TaskTracker
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • YARN • ResourceManager • NodeManager • ApplicationMaster • Job History Server • YARN Timeline Server
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • MRv1 • JobTracker • TaskTracker • YARN • ResourceManager • NodeManager • ApplicationMaster • Job History Server                                          (We  can  not  see  the  log  job  history  If  it  do  not  install)   • YARN Timeline Server                                          (We  can  not  see  the  log  YARN  history  If  it  do  not  install)
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YARN Timeline Server • It is included container info
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Note!!!
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Use the Hadoop 2.4.0 and later!!!
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • The versions which must not be used • Apache Hadoop 2.2.0 • Apache Hadoop 2.3.0 • HDP 2.0(2.2.0 based)
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Currently • Apache Hadoop 2.6.0 • CDH 5.3.2(2.5.0 based and patch) • HDP 2.2(2.6.0 based and patch)
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Why should not use? • Capacity Scheduler • There is a bug • Fair Scheduler • There is a bug
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Any bugs? • Each Scheduler will cause a deadlock
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. •In fact, there is a bug in 2.4.0 and 2.6.0… •It is better to use the new version. •Note: 2.7.0 and later is a different thing
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • I was backport some patch • https://github.com/ryukobayashi/patches
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • Included dead lock patch • Format of the counter • Application kill in Web UI.
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Format of the counter
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Format of the counter
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI • Job kill in Web UI • (default false) • Application kill in Web UI • (default true) mapreduce.jobtracker.webinterface.trusted yarn.resourcemanager.webapp.ui-actions.enabled
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • We want the next… • Job task attempt kill in Web UI patch (in development) • Currently, only command line $ mapred job -kill-task attempt_*
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 2
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 2 Cluster   Application App   Master Container Application App   Master Container
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master Livelock!
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Container Application App   Master Kill
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • ^ squeeze the number of applications
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • ^ squeeze the number of applications • set the root maxRunningApps
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • root maxRunningApps = 2 Cluster   Application App   Master Container Application App   Master Application App   Master Container Application App   Master Pending
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YANR Resource Management yarn-site.xml • yarn.nodemanager.resource.memory-mb • (yarn.nodenamager.vmem-pmem-ratio) • (yarn.scheduler.minimum-allocation-mb) mapred-site.xml • yarn.app.mapreduce.am.resource.mb • mapreduce.map.memory.mb • mapreduce.reduce.memory.mb fair-scheduler.xml • maxResources, minResources etc…
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. e.g. Use hdp-configuration-utils.py script http://goo.gl/L2hxyq Use Ambari http://ambari.apache.org/ See the Cloudera’s document http://goo.gl/EBreca YANR Resource Management
Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Thanks!!!

More Related Content

What's hot

Large Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraphLarge Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraph
P. Taylor Goetz
 
My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)
Wes McKinney
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
Taro L. Saito
 
Using PostgreSQL with Bibliographic Data
Using PostgreSQL with Bibliographic DataUsing PostgreSQL with Bibliographic Data
Using PostgreSQL with Bibliographic Data
Jimmy Angelakos
 
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto: SQL-on-Anything. Netherlands Hadoop User Group MeetupPresto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Wojciech Biela
 
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
Spark Summit
 
A Day in the Life of a Druid Implementor and Druid's Roadmap
A Day in the Life of a Druid Implementor and Druid's RoadmapA Day in the Life of a Druid Implementor and Druid's Roadmap
A Day in the Life of a Druid Implementor and Druid's Roadmap
Itai Yaffe
 
Presto at Twitter
Presto at TwitterPresto at Twitter
Presto at Twitter
Bill Graham
 
Kafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn MeetupKafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn Meetup
Gwen (Chen) Shapira
 
Real-time analytics with Druid at Appsflyer
Real-time analytics with Druid at AppsflyerReal-time analytics with Druid at Appsflyer
Real-time analytics with Druid at Appsflyer
Michael Spector
 
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with DaskAUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
Víctor Zabalza
 
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
Nathan Bijnens
 
Programmatic Bidding Data Streams & Druid
Programmatic Bidding Data Streams & DruidProgrammatic Bidding Data Streams & Druid
Programmatic Bidding Data Streams & Druid
Charles Allen
 
Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou
Fast Data with Apache Ignite and Apache Spark with Christos ErotocritouFast Data with Apache Ignite and Apache Spark with Christos Erotocritou
Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou
Spark Summit
 
Apache Arrow and Python: The latest
Apache Arrow and Python: The latestApache Arrow and Python: The latest
Apache Arrow and Python: The latest
Wes McKinney
 
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgetsLens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Víctor Zabalza
 
Big Telco - Yousun Jeong
Big Telco - Yousun JeongBig Telco - Yousun Jeong
Big Telco - Yousun Jeong
Spark Summit
 
Presto @ Uber Hadoop summit2017
Presto @ Uber Hadoop summit2017Presto @ Uber Hadoop summit2017
Presto @ Uber Hadoop summit2017
Zhenxiao Luo
 
Presto+MySQLで分散SQL
Presto+MySQLで分散SQLPresto+MySQLで分散SQL
Presto+MySQLで分散SQL
Sadayuki Furuhashi
 
Presto for the Enterprise @ Hadoop Meetup
Presto for the Enterprise @ Hadoop MeetupPresto for the Enterprise @ Hadoop Meetup
Presto for the Enterprise @ Hadoop Meetup
Wojciech Biela
 

What's hot (20)

Large Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraphLarge Scale Graph Analytics with JanusGraph
Large Scale Graph Analytics with JanusGraph
 
My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)My Data Journey with Python (SciPy 2015 Keynote)
My Data Journey with Python (SciPy 2015 Keynote)
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
 
Using PostgreSQL with Bibliographic Data
Using PostgreSQL with Bibliographic DataUsing PostgreSQL with Bibliographic Data
Using PostgreSQL with Bibliographic Data
 
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto: SQL-on-Anything. Netherlands Hadoop User Group MeetupPresto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
 
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...
 
A Day in the Life of a Druid Implementor and Druid's Roadmap
A Day in the Life of a Druid Implementor and Druid's RoadmapA Day in the Life of a Druid Implementor and Druid's Roadmap
A Day in the Life of a Druid Implementor and Druid's Roadmap
 
Presto at Twitter
Presto at TwitterPresto at Twitter
Presto at Twitter
 
Kafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn MeetupKafka and Hadoop at LinkedIn Meetup
Kafka and Hadoop at LinkedIn Meetup
 
Real-time analytics with Druid at Appsflyer
Real-time analytics with Druid at AppsflyerReal-time analytics with Druid at Appsflyer
Real-time analytics with Druid at Appsflyer
 
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with DaskAUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask
 
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
 
Programmatic Bidding Data Streams & Druid
Programmatic Bidding Data Streams & DruidProgrammatic Bidding Data Streams & Druid
Programmatic Bidding Data Streams & Druid
 
Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou
Fast Data with Apache Ignite and Apache Spark with Christos ErotocritouFast Data with Apache Ignite and Apache Spark with Christos Erotocritou
Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou
 
Apache Arrow and Python: The latest
Apache Arrow and Python: The latestApache Arrow and Python: The latest
Apache Arrow and Python: The latest
 
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgetsLens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
 
Big Telco - Yousun Jeong
Big Telco - Yousun JeongBig Telco - Yousun Jeong
Big Telco - Yousun Jeong
 
Presto @ Uber Hadoop summit2017
Presto @ Uber Hadoop summit2017Presto @ Uber Hadoop summit2017
Presto @ Uber Hadoop summit2017
 
Presto+MySQLで分散SQL
Presto+MySQLで分散SQLPresto+MySQLで分散SQL
Presto+MySQLで分散SQL
 
Presto for the Enterprise @ Hadoop Meetup
Presto for the Enterprise @ Hadoop MeetupPresto for the Enterprise @ Hadoop Meetup
Presto for the Enterprise @ Hadoop Meetup
 

Viewers also liked

Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
Treasure Data, Inc.
 
Internals of Presto Service
Internals of Presto ServiceInternals of Presto Service
Internals of Presto Service
Treasure Data, Inc.
 
Diary of Support Engineer
Diary of Support EngineerDiary of Support Engineer
Diary of Support Engineer
Treasure Data, Inc.
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
VARUN SAXENA
 
Apache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and FutureApache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and Future
DataWorks Summit
 
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFSDiscover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Hortonworks
 
20140708 オンラインゲームソリューション
20140708 オンラインゲームソリューション20140708 オンラインゲームソリューション
20140708 オンラインゲームソリューション
Takahiro Inoue
 
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
Hortonworks
 
トレジャーデータ株式会社について（for all Data_Enthusiast!!）
トレジャーデータ株式会社について（for all Data_Enthusiast!!）トレジャーデータ株式会社について（for all Data_Enthusiast!!）
トレジャーデータ株式会社について（for all Data_Enthusiast!!）
Takahiro Inoue
 
事例で学ぶトレジャーデータ 20140612
事例で学ぶトレジャーデータ 20140612事例で学ぶトレジャーデータ 20140612
事例で学ぶトレジャーデータ 20140612
Takahiro Inoue
 
オンラインゲームソリューション@トレジャーデータ
オンラインゲームソリューション@トレジャーデータオンラインゲームソリューション@トレジャーデータ
オンラインゲームソリューション@トレジャーデータ
Takahiro Inoue
 
この Visualization がすごい2014 〜データ世界を彩るツール6選〜
この Visualization がすごい2014 〜データ世界を彩るツール6選〜この Visualization がすごい2014 〜データ世界を彩るツール6選〜
この Visualization がすごい2014 〜データ世界を彩るツール6選〜
Takahiro Inoue
 
Tableauが魅せる Data Visualization の世界
Tableauが魅せる Data Visualization の世界Tableauが魅せる Data Visualization の世界
Tableauが魅せる Data Visualization の世界
Takahiro Inoue
 
MongoDBを用いたソーシャルア��リのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
Takahiro Inoue
 
トレジャーデータ流，データ分析の始め方
トレジャーデータ流，データ分析の始め方トレジャーデータ流，データ分析の始め方
トレジャーデータ流，データ分析の始め方
Takahiro Inoue
 
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
Takahiro Inoue
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014
Sadayuki Furuhashi
 
AWS Lambda
AWS LambdaAWS Lambda
AWS Lambda
Scott Leberknight
 
トレジャーデータのバッチクエリとアドホッククエリを理解��る
トレジャーデータのバッチクエリとアドホッククエリを理解するトレジャーデータのバッチクエリとアドホッククエリを理解する
トレジャーデータのバッチクエリとアドホッククエリを理解する
Takahiro Inoue
 
トレジャーデータとtableau実現する自動レポーティング
トレジャーデータとtableau実現する自動レポーティングトレジャーデータとtableau実現する自動レポーティング
トレジャーデータとtableau実現する自動レポーティング
Takahiro Inoue
 

Viewers also liked (20)

Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
 
Internals of Presto Service
Internals of Presto ServiceInternals of Presto Service
Internals of Presto Service
 
Diary of Support Engineer
Diary of Support EngineerDiary of Support Engineer
Diary of Support Engineer
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
 
Apache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and FutureApache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and Future
 
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFSDiscover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
 
20140708 オンラインゲームソリューション
20140708 オンラインゲームソリューション20140708 オンラインゲームソリューション
20140708 オンラインゲームソリューション
 
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
 
トレジャーデータ株式会社について（for all Data_Enthusiast!!）
トレジャーデータ株式会社について（for all Data_Enthusiast!!）トレジャーデータ株式会社について（for all Data_Enthusiast!!）
トレジャーデータ株式会社について（for all Data_Enthusiast!!）
 
事例で学ぶトレジャーデータ 20140612
事例で学ぶトレジャーデータ 20140612事例で学ぶトレジャーデータ 20140612
事例で学ぶトレジャーデータ 20140612
 
オンラインゲームソリューション@トレジャーデータ
オンラインゲームソリューション@トレジャーデータオンラインゲームソリューション@トレジャーデータ
オンラインゲームソリューション@トレジャーデータ
 
この Visualization がすごい2014 〜データ世界を彩るツール6選〜
この Visualization がすごい2014 〜データ世界を彩るツール6選〜この Visualization がすごい2014 〜データ世界を彩るツール6選〜
この Visualization がすごい2014 〜データ世界を彩るツール6選〜
 
Tableauが魅せる Data Visualization の世界
Tableauが魅せる Data Visualization の世界Tableauが魅せる Data Visualization の世界
Tableauが魅せる Data Visualization の世界
 
MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
MongoDBを用いたソーシャルアプリのログ解析 〜解析基盤構築からフロントUIまで、MongoDBを最大限に活用する〜
 
トレジャーデータ流，データ分析の始め方
トレジャーデータ流，データ分析の始め方トレジャーデータ流，データ分析の始め方
トレジャーデータ流，データ分析の始め方
 
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014
 
AWS Lambda
AWS LambdaAWS Lambda
AWS Lambda
 
トレジャーデータのバッチクエリとアドホッククエリを理解する
トレジャーデータのバッチクエリとアドホッククエリを理解するトレジャーデータのバッチクエリとアドホッククエリを理解する
トレジャーデータのバッチクエリとアドホッククエリを理解する
 
トレジャーデータとtableau実現する自動レポーティング
トレジャーデータとtableau実現する自動レポーティングトレジャーデータとtableau実現する自動レポーティング
トレジャーデータとtableau実現する自動レポーティング
 

Similar to HDP2 and YARN operations point

YARN
YARNYARN
YARN
Alex Moundalexis
 
Hadoop Summit Europe 2015 - YARN Present and Future
Hadoop Summit Europe 2015 - YARN Present and FutureHadoop Summit Europe 2015 - YARN Present and Future
Hadoop Summit Europe 2015 - YARN Present and Future
Vinod Kumar Vavilapalli
 
Apache Hadoop YARN 2015: Present and Future
Apache Hadoop YARN 2015: Present and FutureApache Hadoop YARN 2015: Present and Future
Apache Hadoop YARN 2015: Present and Future
DataWorks Summit
 
Inside Of Mbga Open Platform
Inside Of Mbga Open PlatformInside Of Mbga Open Platform
Inside Of Mbga Open Platform
Hideo Kimura
 
Shrinking the container_zurich_july_2018
Shrinking the container_zurich_july_2018Shrinking the container_zurich_july_2018
Shrinking the container_zurich_july_2018
Ewan Slater
 
Architectural considerations for Hadoop Applications
Architectural considerations for Hadoop ApplicationsArchitectural considerations for Hadoop Applications
Architectural considerations for Hadoop Applications
hadooparchbook
 
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E... Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
Yahoo!デベロッパーネットワーク
 
Accumulo Summit 2014: Accumulo with Distributed SQL queries
Accumulo Summit 2014: Accumulo with Distributed SQL queriesAccumulo Summit 2014: Accumulo with Distributed SQL queries
Accumulo Summit 2014: Accumulo with Distributed SQL queries
Accumulo Summit
 
Soft-Shake 2013 : Enabling Realtime Queries to End Users
Soft-Shake 2013 : Enabling Realtime Queries to End UsersSoft-Shake 2013 : Enabling Realtime Queries to End Users
Soft-Shake 2013 : Enabling Realtime Queries to End Users
Benoit Perroud
 
Scaling to 1,000,000 concurrent users on the JVM
Scaling to 1,000,000 concurrent users on the JVMScaling to 1,000,000 concurrent users on the JVM
Scaling to 1,000,000 concurrent users on the JVM
Pursuit Consulting
 
Hadoop Application Architectures tutorial - Strata London
Hadoop Application Architectures tutorial - Strata LondonHadoop Application Architectures tutorial - Strata London
Hadoop Application Architectures tutorial - Strata London
hadooparchbook
 
Strata NY 2014 - Architectural considerations for Hadoop applications tutorial
Strata NY 2014 - Architectural considerations for Hadoop applications tutorialStrata NY 2014 - Architectural considerations for Hadoop applications tutorial
Strata NY 2014 - Architectural considerations for Hadoop applications tutorial
hadooparchbook
 
Trend Micro Big Data Platform and Apache Bigtop
Trend Micro Big Data Platform and Apache BigtopTrend Micro Big Data Platform and Apache Bigtop
Trend Micro Big Data Platform and Apache Bigtop
Evans Ye
 
Hadoop Application Architectures tutorial at Big DataService 2015
Hadoop Application Architectures tutorial at Big DataService 2015Hadoop Application Architectures tutorial at Big DataService 2015
Hadoop Application Architectures tutorial at Big DataService 2015
hadooparchbook
 
Lattice yapc-slideshare
Lattice yapc-slideshareLattice yapc-slideshare
Lattice yapc-slideshare
Gwenn Etourneau
 
High throughput data replication over RAFT
High throughput data replication over RAFTHigh throughput data replication over RAFT
High throughput data replication over RAFT
DataWorks Summit
 
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal GemfireIMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
In-Memory Computing Summit
 
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
Big Data Spain
 
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web ScaleSaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
SaltStack
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoWorld
 

Similar to HDP2 and YARN operations point (20)

YARN
YARNYARN
YARN
 
Hadoop Summit Europe 2015 - YARN Present and Future
Hadoop Summit Europe 2015 - YARN Present and FutureHadoop Summit Europe 2015 - YARN Present and Future
Hadoop Summit Europe 2015 - YARN Present and Future
 
Apache Hadoop YARN 2015: Present and Future
Apache Hadoop YARN 2015: Present and FutureApache Hadoop YARN 2015: Present and Future
Apache Hadoop YARN 2015: Present and Future
 
Inside Of Mbga Open Platform
Inside Of Mbga Open PlatformInside Of Mbga Open Platform
Inside Of Mbga Open Platform
 
Shrinking the container_zurich_july_2018
Shrinking the container_zurich_july_2018Shrinking the container_zurich_july_2018
Shrinking the container_zurich_july_2018
 
Architectural considerations for Hadoop Applications
Architectural considerations for Hadoop ApplicationsArchitectural considerations for Hadoop Applications
Architectural considerations for Hadoop Applications
 
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E... Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
Dragon: A Distributed Object Storage at Yahoo! JAPAN (WebDB Forum 2017 / E...
 
Accumulo Summit 2014: Accumulo with Distributed SQL queries
Accumulo Summit 2014: Accumulo with Distributed SQL queriesAccumulo Summit 2014: Accumulo with Distributed SQL queries
Accumulo Summit 2014: Accumulo with Distributed SQL queries
 
Soft-Shake 2013 : Enabling Realtime Queries to End Users
Soft-Shake 2013 : Enabling Realtime Queries to End UsersSoft-Shake 2013 : Enabling Realtime Queries to End Users
Soft-Shake 2013 : Enabling Realtime Queries to End Users
 
Scaling to 1,000,000 concurrent users on the JVM
Scaling to 1,000,000 concurrent users on the JVMScaling to 1,000,000 concurrent users on the JVM
Scaling to 1,000,000 concurrent users on the JVM
 
Hadoop Application Architectures tutorial - Strata London
Hadoop Application Architectures tutorial - Strata LondonHadoop Application Architectures tutorial - Strata London
Hadoop Application Architectures tutorial - Strata London
 
Strata NY 2014 - Architectural considerations for Hadoop applications tutorial
Strata NY 2014 - Architectural considerations for Hadoop applications tutorialStrata NY 2014 - Architectural considerations for Hadoop applications tutorial
Strata NY 2014 - Architectural considerations for Hadoop applications tutorial
 
Trend Micro Big Data Platform and Apache Bigtop
Trend Micro Big Data Platform and Apache BigtopTrend Micro Big Data Platform and Apache Bigtop
Trend Micro Big Data Platform and Apache Bigtop
 
Hadoop Application Architectures tutorial at Big DataService 2015
Hadoop Application Architectures tutorial at Big DataService 2015Hadoop Application Architectures tutorial at Big DataService 2015
Hadoop Application Architectures tutorial at Big DataService 2015
 
Lattice yapc-slideshare
Lattice yapc-slideshareLattice yapc-slideshare
Lattice yapc-slideshare
 
High throughput data replication over RAFT
High throughput data replication over RAFTHigh throughput data replication over RAFT
High throughput data replication over RAFT
 
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal GemfireIMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire
 
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...
 
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web ScaleSaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
 

More from Treasure Data, Inc.

GDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for MarketersGDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for Marketers
Treasure Data, Inc.
 
AR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and MarketAR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and Market
Treasure Data, Inc.
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data Platforms
Treasure Data, Inc.
 
Hands On: Javascript SDK
Hands On: Javascript SDKHands On: Javascript SDK
Hands On: Javascript SDK
Treasure Data, Inc.
 
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowHands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
Treasure Data, Inc.
 
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and AppsBrand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Treasure Data, Inc.
 
How to Power Your Customer Experience with Data
How to Power Your Customer Experience with DataHow to Power Your Customer Experience with Data
How to Power Your Customer Experience with Data
Treasure Data, Inc.
 
Why Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without DataWhy Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without Data
Treasure Data, Inc.
 
Connecting the Customer Data Dots
Connecting the Customer Data DotsConnecting the Customer Data Dots
Connecting the Customer Data Dots
Treasure Data, Inc.
 
Harnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company SuccessHarnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company Success
Treasure Data, Inc.
 
Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017
Treasure Data, Inc.
 
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
Treasure Data, Inc.
 
Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14
Treasure Data, Inc.
 
Introduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of HivemallIntroduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of Hivemall
Treasure Data, Inc.
 
Scalable Hadoop in the cloud
Scalable Hadoop in the cloudScalable Hadoop in the cloud
Scalable Hadoop in the cloud
Treasure Data, Inc.
 
Using Embulk at Treasure Data
Using Embulk at Treasure DataUsing Embulk at Treasure Data
Using Embulk at Treasure Data
Treasure Data, Inc.
 
Scaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big DataScaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big Data
Treasure Data, Inc.
 
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data, Inc.
 
Treasure Data From MySQL to Redshift
Treasure Data From MySQL to RedshiftTreasure Data From MySQL to Redshift
Treasure Data From MySQL to Redshift
Treasure Data, Inc.
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
Treasure Data, Inc.
 

More from Treasure Data, Inc. (20)

GDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for MarketersGDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for Marketers
 
AR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and MarketAR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and Market
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data Platforms
 
Hands On: Javascript SDK
Hands On: Javascript SDKHands On: Javascript SDK
Hands On: Javascript SDK
 
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowHands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
 
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and AppsBrand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
 
How to Power Your Customer Experience with Data
How to Power Your Customer Experience with DataHow to Power Your Customer Experience with Data
How to Power Your Customer Experience with Data
 
Why Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without DataWhy Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without Data
 
Connecting the Customer Data Dots
Connecting the Customer Data DotsConnecting the Customer Data Dots
Connecting the Customer Data Dots
 
Harnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company SuccessHarnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company Success
 
Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017
 
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
 
Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14
 
Introduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of HivemallIntroduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of Hivemall
 
Scalable Hadoop in the cloud
Scalable Hadoop in the cloudScalable Hadoop in the cloud
Scalable Hadoop in the cloud
 
Using Embulk at Treasure Data
Using Embulk at Treasure DataUsing Embulk at Treasure Data
Using Embulk at Treasure Data
 
Scaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big DataScaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big Data
 
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
 
Treasure Data From MySQL to Redshift
Treasure Data From MySQL to RedshiftTreasure Data From MySQL to Redshift
Treasure Data From MySQL to Redshift
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
 

Recently uploaded

CATIA V5 Automation VB script.........pdf
CATIA V5 Automation VB script.........pdfCATIA V5 Automation VB script.........pdf
CATIA V5 Automation VB script.........pdf
shahidad729
 
02 - Method Statement for Concrete pouring.docx
02 - Method Statement for Concrete pouring.docx02 - Method Statement for Concrete pouring.docx
02 - Method Statement for Concrete pouring.docx
RAHEEL KHALID
 
internship project presentation for reference.pptx
internship project presentation for reference.pptxinternship project presentation for reference.pptx
internship project presentation for reference.pptx
SaieJadhav1
 
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
IJAEMSJORNAL
 
Youtube Transcript Sumariser- application of API
Youtube Transcript Sumariser- application of APIYoutube Transcript Sumariser- application of API
Youtube Transcript Sumariser- application of API
AnamikaRani12
 
THERMAL POWER PLANT its applications and advantages
THERMAL POWER PLANT its applications and advantagesTHERMAL POWER PLANT its applications and advantages
THERMAL POWER PLANT its applications and advantages
VikramSingh6251
 
Gen AI with LLM for construction technology
Gen AI with LLM for construction technologyGen AI with LLM for construction technology
Gen AI with LLM for construction technology
Tae wook kang
 
Defect Elimination Management - CMMS Success.pdf
Defect Elimination Management - CMMS Success.pdfDefect Elimination Management - CMMS Success.pdf
Defect Elimination Management - CMMS Success.pdf
David Johnston
 
The Pennsylvania State University degree Cert diploma offer
The Pennsylvania State University degree Cert diploma offerThe Pennsylvania State University degree Cert diploma offer
The Pennsylvania State University degree Cert diploma offer
ekyhonz
 
Human_assault project using jetson nano new
Human_assault project using jetson nano newHuman_assault project using jetson nano new
Human_assault project using jetson nano new
frostflash010
 
UNIT-I-METAL CASTING PROCESSES -Manufact
UNIT-I-METAL CASTING PROCESSES -ManufactUNIT-I-METAL CASTING PROCESSES -Manufact
UNIT-I-METAL CASTING PROCESSES -Manufact
Mr.C.Dineshbabu
 
Introduction to Course_1 for Cyber Security
Introduction to Course_1 for Cyber SecurityIntroduction to Course_1 for Cyber Security
Introduction to Course_1 for Cyber Security
kuldeephule1989
 
Mobile Forensics challenges and Extraction process
Mobile Forensics challenges and Extraction processMobile Forensics challenges and Extraction process
Mobile Forensics challenges and Extraction process
Swapnil Gharat
 
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
Kiran Kumar Manigam
 
Cyber security detailed ppt and understand
Cyber security detailed ppt and understandCyber security detailed ppt and understand
Cyber security detailed ppt and understand
docpain605501
 
III B.TECH CSE_flutter Lab manual (1).docx
III B.TECH CSE_flutter Lab manual (1).docxIII B.TECH CSE_flutter Lab manual (1).docx
III B.TECH CSE_flutter Lab manual (1).docx
divijareddy0502
 
PBL _PPT _final year for engineerin student
PBL _PPT _final year for engineerin studentPBL _PPT _final year for engineerin student
PBL _PPT _final year for engineerin student
nikitalohar549
 
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
NoeAranel
 
07 - Method Statement for Plastering Works.pdf
07 - Method Statement for Plastering Works.pdf07 - Method Statement for Plastering Works.pdf
07 - Method Statement for Plastering Works.pdf
RAHEEL KHALID
 
03 - Method Statement for block masonry.pdf
03 - Method Statement for block masonry.pdf03 - Method Statement for block masonry.pdf
03 - Method Statement for block masonry.pdf
RAHEEL KHALID
 

Recently uploaded (20)

CATIA V5 Automation VB script.........pdf
CATIA V5 Automation VB script.........pdfCATIA V5 Automation VB script.........pdf
CATIA V5 Automation VB script.........pdf
 
02 - Method Statement for Concrete pouring.docx
02 - Method Statement for Concrete pouring.docx02 - Method Statement for Concrete pouring.docx
02 - Method Statement for Concrete pouring.docx
 
internship project presentation for reference.pptx
internship project presentation for reference.pptxinternship project presentation for reference.pptx
internship project presentation for reference.pptx
 
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
Agricultural Profitability through Resilience: Smallholder Farmers' Strategie...
 
Youtube Transcript Sumariser- application of API
Youtube Transcript Sumariser- application of APIYoutube Transcript Sumariser- application of API
Youtube Transcript Sumariser- application of API
 
THERMAL POWER PLANT its applications and advantages
THERMAL POWER PLANT its applications and advantagesTHERMAL POWER PLANT its applications and advantages
THERMAL POWER PLANT its applications and advantages
 
Gen AI with LLM for construction technology
Gen AI with LLM for construction technologyGen AI with LLM for construction technology
Gen AI with LLM for construction technology
 
Defect Elimination Management - CMMS Success.pdf
Defect Elimination Management - CMMS Success.pdfDefect Elimination Management - CMMS Success.pdf
Defect Elimination Management - CMMS Success.pdf
 
The Pennsylvania State University degree Cert diploma offer
The Pennsylvania State University degree Cert diploma offerThe Pennsylvania State University degree Cert diploma offer
The Pennsylvania State University degree Cert diploma offer
 
Human_assault project using jetson nano new
Human_assault project using jetson nano newHuman_assault project using jetson nano new
Human_assault project using jetson nano new
 
UNIT-I-METAL CASTING PROCESSES -Manufact
UNIT-I-METAL CASTING PROCESSES -ManufactUNIT-I-METAL CASTING PROCESSES -Manufact
UNIT-I-METAL CASTING PROCESSES -Manufact
 
Introduction to Course_1 for Cyber Security
Introduction to Course_1 for Cyber SecurityIntroduction to Course_1 for Cyber Security
Introduction to Course_1 for Cyber Security
 
Mobile Forensics challenges and Extraction process
Mobile Forensics challenges and Extraction processMobile Forensics challenges and Extraction process
Mobile Forensics challenges and Extraction process
 
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
RAILWAYS, a vital part of our infrastructure, play a crucial role in ensuring...
 
Cyber security detailed ppt and understand
Cyber security detailed ppt and understandCyber security detailed ppt and understand
Cyber security detailed ppt and understand
 
III B.TECH CSE_flutter Lab manual (1).docx
III B.TECH CSE_flutter Lab manual (1).docxIII B.TECH CSE_flutter Lab manual (1).docx
III B.TECH CSE_flutter Lab manual (1).docx
 
PBL _PPT _final year for engineerin student
PBL _PPT _final year for engineerin studentPBL _PPT _final year for engineerin student
PBL _PPT _final year for engineerin student
 
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
HSE-BMS-009 COSHH & MSDS.pptHSE-BMS-009 COSHH & MSDS.pptSE-BMS-009 COSHH & MSDS.
 
07 - Method Statement for Plastering Works.pdf
07 - Method Statement for Plastering Works.pdf07 - Method Statement for Plastering Works.pdf
07 - Method Statement for Plastering Works.pdf
 
03 - Method Statement for block masonry.pdf
03 - Method Statement for block masonry.pdf03 - Method Statement for block masonry.pdf
03 - Method Statement for block masonry.pdf
 

HDP2 and YARN operations point

  • 1. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. HDP2 and YARN operations point Ryu Kobayashi Treasure Data Tech Talk 11 and 12 Mar 2015
  • 2. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Who am I? • Ryu Kobayashi • @ryu_kobayashi • https://github.com/ryukobayashi • Treasure Data, Inc. • Software Engineer • Background • Hadoop, Cassandra, Machine Learning, ... • I developed Huahin(Hadoop) Framework. 
 http://huahinframework.org/
  • 3. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. What is YARN?
  • 4. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YARN(Yet Another Resource Negotiator) Architecture
  • 5. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • MRv1 • JobTracker • TaskTracker
  • 6. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • YARN • ResourceManager • NodeManager • ApplicationMaster • Job History Server • YARN Timeline Server
  • 7. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • MRv1 • JobTracker • TaskTracker • YARN • ResourceManager • NodeManager • ApplicationMaster • Job History Server                                          (We  can  not  see  the  log  job  history  If  it  do  not  install)   • YARN Timeline Server                                          (We  can  not  see  the  log  YARN  history  If  it  do  not  install)
  • 8. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YARN Timeline Server • It is included container info
  • 9. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Note!!!
  • 10. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Use the Hadoop 2.4.0 and later!!!
  • 11. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • The versions which must not be used • Apache Hadoop 2.2.0 • Apache Hadoop 2.3.0 • HDP 2.0(2.2.0 based)
  • 12. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Currently • Apache Hadoop 2.6.0 • CDH 5.3.2(2.5.0 based and patch) • HDP 2.2(2.6.0 based and patch)
  • 13. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Why should not use? • Capacity Scheduler • There is a bug • Fair Scheduler • There is a bug
  • 14. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. • Any bugs? • Each Scheduler will cause a deadlock
  • 15. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. •In fact, there is a bug in 2.4.0 and 2.6.0… •It is better to use the new version. •Note: 2.7.0 and later is a different thing
  • 16. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • I was backport some patch • https://github.com/ryukobayashi/patches
  • 17. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • Included dead lock patch • Format of the counter • Application kill in Web UI.
  • 18. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Format of the counter
  • 19. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Format of the counter
  • 20. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI
  • 21. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI
  • 22. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Application kill in Web UI • Job kill in Web UI • (default false) • Application kill in Web UI • (default true) mapreduce.jobtracker.webinterface.trusted yarn.resourcemanager.webapp.ui-actions.enabled
  • 23. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Backport Patch • We want the next… • Job task attempt kill in Web UI patch (in development) • Currently, only command line $ mapred job -kill-task attempt_*
  • 24. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 2
  • 25. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 2 Cluster   Application App   Master Container Application App   Master Container
  • 26. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4
  • 27. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master
  • 28. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master
  • 29. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Application App   Master Livelock!
  • 30. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 Cluster   Application App   Master Application App   Master Application App   Master Container Application App   Master Kill
  • 31. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • ^ squeeze the number of applications
  • 32. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • ^ squeeze the number of applications • set the root maxRunningApps
  • 33. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Matter of resources • total container = 4 • concurrent application = 4 • root maxRunningApps = 2 Cluster   Application App   Master Container Application App   Master Application App   Master Container Application App   Master Pending
  • 34. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. YANR Resource Management yarn-site.xml • yarn.nodemanager.resource.memory-mb • (yarn.nodenamager.vmem-pmem-ratio) • (yarn.scheduler.minimum-allocation-mb) mapred-site.xml • yarn.app.mapreduce.am.resource.mb • mapreduce.map.memory.mb • mapreduce.reduce.memory.mb fair-scheduler.xml • maxResources, minResources etc…
  • 35. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. e.g. Use hdp-configuration-utils.py script http://goo.gl/L2hxyq Use Ambari http://ambari.apache.org/ See the Cloudera’s document http://goo.gl/EBreca YANR Resource Management
  • 36. Copyright  ©2015  Treasure  Data.    All  Rights  Reserved. Thanks!!!