From the course: Learning Hadoop
Join today to access over 23,200 courses taught by industry experts.
Scale a Spark job to calculate Pi - Hadoop Tutorial
From the course: Learning Hadoop
Scale a Spark job to calculate Pi
- [Instructor] So previously we ran the calculate pi with Spark calculating the digits of pi in the very beginning of this course. And what I have done is I have selected that job and then I have created a new job with additional parameters based on that job. So let's first review the original job. So if we look at that job the configuration was to use the included jar file with the main class with a thousand iterations which gives us this result. And that job took 25 seconds. Now what I have done is I cloned that job and in the new job, which already succeeded the original job took 25 seconds. This one took one minute and 19 seconds. What I did is I changed the configuration and I set it to 10,000 arguments so that we would get a more precise value of pi. Now, I did that also to be able to compare and contrast in terms of scaling in terms of the resource usage. So of course we could just look here at the monitoring to…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.