Lemuntu

[Play Docker]How to build up an Hadoop server cluster in docker

1 1468
Edited by john.wen at Jul 20, 2016 02:20

How to build up an Hadoop server cluster in docker




1. Introduction

Hadoop is a software platform that can develops and runs mass data. It is an open source software framework that was implemented using java language by Apache, achieves a massive data distributed computing in computer cluster. The core design of Hadoop framework is HDFS and MapReduce; HDFS provides mass data storage, and MapReduce provides a calculation of the data.


2. Install Docker

Please refer 《How to install docker on the Lemuntu》 to learn how to install docker. In this article, we will teach you how to install docker on Lemuntu.



3. Install Hadoop

(1) Create a hadoop system user
  1. addgroup Hadoop
  2. sudo adduser --ingroup hadoop hadoop
Copy the Code
After click "Enter", will promot you to input hadoop's passwrod, type password: hadoop.


(2) Add administrator privilege to hadoop
  1. usermod -aG root hadoop
Copy the Code

(3) Install SSH
  1. apt-get install openssh-server
Copy the Code

(4) Start SSH server
  1. service ssh start
Copy the Code

(5) View SSH server
  1. ps –ef | grep ssh
Copy the Code

(6) Generate public key and private key
  1. ssh-keygen -t rsa -P ""
Copy the Code

id_rsa is a private key and id_rsa.pub is a public key. We add the public key to authorized_keys:
  1. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Copy the Code

(7) Install Java JDK
  1. apt-get install openjdk-8-jdk
Copy the Code


(8) View Java version
  1. java -version
Copy the Code


(9) Install Hadoop
  1. wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-2.7.2.tar.gz
  2. tar –xvf hadoop-2.7.2.tar.gz
  3. mv hadoop-2.7.2 /usr/local/Hadoop

  4. chown -R hadoop:hadoop /usr/local/Hadoop
  5. su hadoop
  6. cd /usr/local/hadoop
Copy the Code


(10) View Java installation path
  1. update-alternatives - -config java
Copy the Code


得到java的安装目录是:/usr/lib/jvm/java-8-openjdk-armhf
设定etc/hadoop/hadoop-env.sh
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-armhf
HADOOP_HOME=/usr/local/hadoop
PATH=$PATH:/usr/local/hadoop/bin
  1. echo “export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-armhf” >> ./ etc/hadoop/hadoop-env.sh
  2. echo “export HADOOP_HOME=/usr/local/hadoop” >> ./ etc/hadoop/hadoop-env.sh
  3. echo “export PATH=$PATH:/usr/local/hadoop/bin” >> ./ etc/hadoop/hadoop-env.sh
Copy the Code

(11) Update enviroment variables
  1. source /usr/local/hadoop/conf/hadoop-env.sh  
Copy the Code
至此,hadoop的单机模式已经安装成功。


4. WordCount test

We can use WordCount tool to check whether hadoop has been installed sucessfully.
  1. mkdir input
  2. cp README.txt ./input
Copy the Code


View the word counting result
  1. cat ./output
Copy the Code




You have to log in before you can reply Login | Sign Up

Points Rules