NAS

Using Ganglia to monitor Banana Pi clusters

8 4751
Pyplate  
Edited by Pyplate at Thu Nov 6, 2014 13:55

Ganglia is a tool used for monitoring clusters of servers.  I've set it up on some Banana Pi clusters that I've been working with so that I can make sure everything's working well.



Set up the master node

I'm using one Banana Pi as a master node which will collect data from all the other Banana PI servers and display it in a web UI.

Install Apache and Ganglia:
  1. sudo apt-get install apache2 php5 libapache2-mod-php5 php5-json
  2. sudo apt-get install ganglia-frontendweb gmetad gmetad rrdtool ganglia-monitor
Copy the Code
Set up the control node's Ganglia configuration, /etc/ganglia/gmetad.conf:
  1. sudo nano /etc/ganglia/gmetad.conf
Copy the Code
Edit the datasource lines with the name of the clusters to be monitored, and the hosts in those clusters:
  1. data_source "cmd_cluster" 60 192.168.0.8:8655 192.168.0.9:8655
  2. data_source "db_cluster" 60 192.168.0.35:8650 192.168.0.36:8650 192.168.0.37:8650 192.168.0.38:8650
  3. data_source "g_cluster" 60 192.168.0.30:8656 192.168.0.31:8656 192.168.0.32:8656 192.168.0.33:8656
Copy the Code
Edit the gridname variable:
  1. gridname = "ARM_Farm"
Copy the Code
I also edited the trustedhosts field to list the IP addresses of nodes in the clusters.  

Set up the configuration for Ganglia's web frontend in Apache:
  1. sudo cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-available/ganglia.conf
  2. sudo a2ensite ganglia.conf
Copy the Code
Restart all the relevant services on the master:
  1. sudo service ganglia-monitor restart ; sudo service gmetad restart ; sudo service apache2 restart
Copy the Code
Set up monitoring on each node

On each client, install ganglia-monitor:
  1. sudo apt-get install ganglia-monitor
Copy the Code
Edit the monitor configuration file, /etc/ganglia/gmond.conf:
  1. sudo nano /etc/ganglia/gmond.conf
Copy the Code
Set the cluster name:
  1. cluster {
  2.   name = "cmd_cluster"
  3.   owner = "unspecified"
  4.   latlong = "unspecified"
  5.   url = "unspecified"
  6. }
Copy the Code
Set the port that Ganglia will use to connect to this node:
  1. /* Feel free to specify as many udp_send_channels as you like.  Gmond
  2.    used to only support having a single channel */
  3. udp_send_channel {
  4.   mcast_join = 239.2.11.71
  5.   port = 8650
  6.   ttl = 1
  7. }

  8. /* You can specify as many udp_recv_channels as you like as well. */
  9. udp_recv_channel {
  10.   mcast_join = 239.2.11.71
  11.   port = 8650
  12.   bind = 239.2.11.71
  13. }

  14. /* You can specify as many tcp_accept_channels as you like to share
  15.    an xml description of the state of the cluster */
  16. tcp_accept_channel {
  17.   port = 8650
  18. }
Copy the Code
Restart the Ganglia monitor:
  1. sudo service ganglia-monitor restart
  2. Stopping Ganglia Monitor Daemon: gmond.
  3. Starting Ganglia Monitor Daemon: gmond.
Copy the Code
Now visit http://<ipaddress>/ganglia, and you should see information about your cluster.



Thanks for sharing!

cxy  
Hi,
I am very interested in the cluster,but i don't known how to build ?  which software do you use to build the cluster ?
how many BananaPi did you use in your clusters of servers ?
Thank you very much

Pyplate  
Edited by Pyplate at Fri Nov 7, 2014 03:17

So far I have built a cluster of web servers, a cluster used for file storage, and a cluster of databases.

You can read about the server cluster here: http://banoffeepiserver.com/banana-pi-server-cluster/. I used Nginx on each node.  Nginx doesn't handle dynamic pages, so I also set up uWSGI on each node, along with my Python CMS (www.pyplate.com).  You don't have to use Nginx, I could have installed any web server.  I used Apache on my Raspberry Pi cluster (http://raspberrywebserver.com/raspberrypicluster/).  

When you build a web server cluster, you need to build a load-balancer to distribute HTTP requests between servers.  My load balancer uses Apache in reverse proxy mode to distribute requests.

I built the next cluster using MySQL replication:http://banoffeepiserver.com/mysq ... -on-four-nodes.html. Again, there are many different databases that support replication. I chose MySQL because it's well documented and widely used.  

I built a cluster using the GlusterFS distributed file system: http://banoffeepiserver.com/glus ... ana-pi-servers.html.  The good thing about gluster is that if I modify a file on one node, the update will automatically be replicated on the other nodes where the file is stored.  If one disk dies or a server crashes, my data is still accessible from any of the other nodes.  

I used Fedora on the web server cluster, but the others are currently running Raspbian.  There are 4 nodes in each cluster at the moment.  The next step is to use all these clusters as one large server.  The web servers will be updated so that they use the database cluster instead of an SQLite database on each server node.  The web servers will use the gluster servers to store files.  Offloading file storage and processing database queries from the web servers means they can run more efficiently and process more simultaneous requests.

tkaiser  
Edited by tkaiser at Mon Nov 10, 2014 04:13
Pyplate replied at Fri Nov 7, 2014 03:15
The good thing about gluster is that if I modify a file on one node, the update will automatically be replicated on the other nodes where the file is stored.


The bad thing about running all sorts of file services on unrealiable hardware is that when a bit flip occurs on one node the errors will be distributed automagically (garbage in / garbage out): http://en.wikipedia.org/wiki/ECC_memory#Problem_background

Pyplate  
tkaiser replied at Fri Nov 7, 2014 04:59
The bad thing about running all sorts of file services un unrealiable hardware is that when a bit  ...

Hmm, I hadn't thought of that. I guess I need to back up my site regularly so that I can restore it if it gets corrupted.

cxy  
Edited by cxy at Fri Nov 7, 2014 22:42

Thanks for your answer first, it is very cool, but I want to build a cluster of services about distributed compilation.
Do you have any good suggestions?

Pyplate  
I'd probably start by looking at distcc: https://code.google.com/p/distcc/. Distcc seems to be widely used.

cxy  
Well,Thanks

You have to log in before you can reply Login | Sign Up

Points Rules