❗The content presented here is sourced directly from Cognitive Class platform. For comprehensive course details, including enrollment information, simply click on the 'Go to class' link on our website.
Updated in [July 11th, 2023]
Apache Hadoop is one of the most popular tools for big data processing It has been successfully deployed in production by many companies for several years Though Hadoop is considered a reliable scalable and cost-effective solution it is constantly being improved by a large community of developers As a result the 20 version offers several revolutionary features including Yet Another Resource Negotiator (YARN) HDFS Federation and high availability which make the Hadoop cluster much more efficient powerful and reliableThe most serious limitations of classical MapReduce are primarily related to scalability resource utilization and the support of workloads different from MapReduce In the MapReduce framework the job execution is controlled by two types of processes: a single master process called JobTracker and a number of subordinate processes called TaskTrackersApache Hadoop 20 includes YARN which separates the resource management and processing components The YARN-based architecture is not constrained to MapReduce In YARN MapReduce is simply degraded to a role of a distributed application (but still a very popular and useful one) and is now called MRv2 MRv2 is simply the re-implementation of the classic MapReduce engine now called MRv1 which runs on top of YARNThe course reviews MapReduce and provides insight into the design and implementation of YARN: ResourceManager instead of a cluster manager ApplicationMaster instead of a dedicated and short-lived JobTracker NodeManager instead of TaskTracker a distributed application instead of a MapReduce job
