[Play Docker]How to build up an Hadoop server cluster in docker

2 2847
Edited by john.wen at Jul 20, 2016 02:20

How to build up an Hadoop server cluster in docker

1. Introduction

Hadoop is a software platform that can develops and runs mass data. It is an open source software framework that was implemented using java language by Apache, achieves a massive data distributed computing in computer cluster. The core design of Hadoop framework is HDFS and MapReduce; HDFS provides mass data storage, and MapReduce provides a calculation of the data.

2. Install Docker

Please refer 《How to install docker on the Lemuntu》 to learn how to install docker. In this article, we will teach you how to install docker on Lemuntu.

3. Install Hadoop

(1) Create a hadoop system user
  1. addgroup Hadoop
  2. sudo adduser --ingroup hadoop hadoop
Copy the Code
After click "Enter", will promot you to input hadoop's passwrod, type password: hadoop.

(2) Add administrator privilege to hadoop
  1. usermod -aG root hadoop
Copy the Code

(3) Install SSH
  1. apt-get install openssh-server
Copy the Code

(4) Start SSH server
  1. service ssh start
Copy the Code

(5) View SSH server
  1. ps –ef | grep ssh
Copy the Code

(6) Generate public key and private key
  1. ssh-keygen -t rsa -P ""
Copy the Code

id_rsa is a private key and id_rsa.pub is a public key. We add the public key to authorized_keys:
  1. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Copy the Code

(7) Install Java JDK
  1. apt-get install openjdk-8-jdk
Copy the Code

(8) View Java version
  1. java -version
Copy the Code

(9) Install Hadoop
  1. wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-2.7.2.tar.gz
  2. tar –xvf hadoop-2.7.2.tar.gz
  3. mv hadoop-2.7.2 /usr/local/Hadoop

  4. chown -R hadoop:hadoop /usr/local/Hadoop
  5. su hadoop
  6. cd /usr/local/hadoop
Copy the Code

(10) View Java installation path
  1. update-alternatives - -config java
Copy the Code

  1. echo “export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-armhf” >> ./ etc/hadoop/hadoop-env.sh
  2. echo “export HADOOP_HOME=/usr/local/hadoop” >> ./ etc/hadoop/hadoop-env.sh
  3. echo “export PATH=$PATH:/usr/local/hadoop/bin” >> ./ etc/hadoop/hadoop-env.sh
Copy the Code

(11) Update enviroment variables
  1. source /usr/local/hadoop/conf/hadoop-env.sh  
Copy the Code

4. WordCount test

We can use WordCount tool to check whether hadoop has been installed sucessfully.
  1. mkdir input
  2. cp README.txt ./input
Copy the Code

View the word counting result
  1. cat ./output
Copy the Code

Accidentally i have come across this website and am a little bit confused about the details shared here. This particular post shares the details regarding how to build up a Hadoop server cluster in lyft accident lawyer docker. The step by step process is available here and i am looking here for more updates on that.

You have to log in before you can reply Login | Sign Up

Points Rules