From the course: Learning Hadoop
Join today to access over 23,200 courses taught by industry experts.
Tune a Mapper - Hadoop Tutorial
From the course: Learning Hadoop
Tune a Mapper
- [Instructor] When we think about possibilities for tuning the map proportion of our map reduce job, let's recall that maps use keys and values as input and then they produce intermediate values, one list per node. One common optimization is a local reducer. What this will do is this will do a aggregate on each worker node, which reduces traffic going across the network and allows the reducers to work more efficiently. So it's an aggregator of the information on that particular node. Other types of mapper task optimizations include the following. Subdivide the tasks so you can change jobs as I was talking about previous video, and rule of thumb is one to three minutes per map task run. Obviously, this is going to vary, depends on the complexity of your computation and the size of your load, but the whole idea is to make really small units of computation. You could also implement a custom partitioner and you could…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.