Sunday, December 8, 2013

My Private Cloud

It is costly to rent systems from Azure/Amazon to do regular experiment at home for learning purposes when I need many machines. I have four very powerful machines lying around at home with a lot of RAM and 32 total processor cores. I bought them second hand from ebay for less than 500$.  These machines are not going to be economical for datacenters because of low processing power and RAM capability but for experimental purposes I can create hundreds of virtual machines easily (most of them wont be running most of the time).

Now installing operating system on each of the virtual machines is very time consuming task. So I decided to go with MAAS inside ubuntu server so it handles OS installation part. Also with juju I can deploy a lot of services (hadoop, mongodb, django and many many more)  with a single command.

I would not be exposing the services outside my home for other people. So I do not require high bandwidth. But I still want to see how it works from internet.

I split my home network in two subnetworks ( and One powers my home equipments and other is for my private cloud machines.

I have a router ( that is connected to Internet and works as gateway to Internet.

I have another system (cloud gateway) running Linux with two network interfaces (, and configured as a router for my private cloud and works as gateway. It has forward packet rule enabled in iptables.

I have one virtual machine on the cloud gateway that I have configured as MAAS server.

Cloud Gateway

Dual interface, is connected to Internet router by direct link [GW + DNS/DHCP:  ]. is connected to cloud machines switch (no DNS, DHCP - managed by MAAS server


#while br0 is using dhcp the IP is static and managed by router DHCP server
auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

auto br1
iface br1 inet static
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

Enable forwarding:

sudo su
echo 1 > /proc/sys/net/ipv4/ip_forward

To make it run on boot

# turn ip_forward on
echo 1 > /proc/sys/net/ipv4/ip_forward

sudo chmod +x /etc/init.d/lekhonicloud
sudo update-rc.d lekhonicloud defaults

iptables -A FORWARD -p tcp -m state -d --state RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -p udp -m state -d --state RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -p tcp -m state -d --state RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -p udp -m state -d --state RELATED,ESTABLISHED -j ACCEPT

Internet router

Dual Interface:
WAN [Static IP mapped to public domain name]

Destination     Gateway         Netmask         Flags Metric Ref    Use Iface   UG    1      0        0 LAN     *        U     0      0        0 LAN
default         [ISP Gtwy IP]         UG    0      0        0 WAN

 MAAS System Settings

From MAAS Network Config page
Interface eth0
Management: Manage DHCP and DNS
Subnet mask
Router :

IP Range:


# The primary network interface
auto eth0
iface eth0 inet static

auto eth1
iface eth1 inet dhcp

Install and configure MAAS

Installing MAAS is easy. Details are availabe at . I am putting here the basic steps. After you boot a system with Ubuntu server bootable disk from first screen select

Multiple server install with MAAS

Then follow the setup. When Install or enlist with Ubuntu MAAS Server dialog box select

Create a new MAAS on this server .

When you are asked for Ubuntu MAAS API address on the Configuring maas-cluster-controller provide the right IP. It should be automatically selected and shown. But if it is not correct change it:

http://<maas ip address>/MAAS/

If you want MAAS to manage DHCP and DNS install those components:

sudo apt-get install maas-dhcp maas-dns

It makes things much easier.

After thse you want to create an admin user and import boot images-

$ sudo maas createadmin --username=root --email=<adminemail>
$ maas-cli maas node-groups import-boot-images

You may now login to http://<maas ip address>/MAAS/ and start configuring.

Commission and start your servers

When you first start a server that is not yet added to MAAS service and it has PXE boot enabled it will boot using an image that it receives from MAAS maintained DHCP/TFTP server and collect some data and tell MAAS about its various hardware configuration and shutdown itself. The node will be shown in MAAS node list. You can changes some of the parameters like host name, power options and press commission button to start commission. If the system is not powered up for some reason you may manually power up. The system will again boot itself using boot image from MAAS and poweroff itself. The system will now be shown as Ready in MASS console. You may start the node now. On boot it will install Ubuntu (you can set which version from MAAS web console).

You may keep any node in ready state so that juju can pick up for deploying any service on it. We will configure juju shortly. Since I have only 4 physical servers I'll be using virtual machines to create juju node pool. MAAS treats both type of machines in the same way.

MAAS inserts the SSH keys, that we set in the MAAS Preferences page, for the default user "ubuntu". But in case the network interface is down it is hard to get the machine back. While it may be a security bad practice, I create a user  on the physical server so that I can login through console.

Install KVM on each Server

First install the required components and setup user:

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
sudo adduser `id -un` kvm
sudo adduser `id -un` libvirtd
Now logout and login back and verify installation using:
ubuntu@Server06:~$ virsh -c qemu:///system list

 Id    Name                           State


Setting up virsh on MAAS system

From MAAS system the query 
$ virsh -c qemu+ssh://<username>@<serverip>/system list --all 
should show all virtual machines on the server when you are logged in as maas 
user (sudo su - maas). If it doers not show up we must do the following steps.

First make sure the user maas has its home directory (/home/maas). 
If not create them and assign /bin/bash as default shell:
$ mkdir /home/maas
$ chown maas:maas /home/maas 
$ sudo chsh maas /bin/bash 

Now sudo as mass

$ sudo su - maas
Now generate the key and assign copy it to the server: 
$ ssh-keygen
$ ssh-copy-id -i ~/.ssh/id_rsa <username>@<server_ip>
The username here can be any user name. If that user is not already setup with ssh keys 
you'll be prompted for password.
Now test again if the command works:
$ virsh -c qemu+ssh://<username>@<serverip>/system list --all 
If it still fails look at for  more troubleshooting information.

Now you may use qemu+ssh://<username>@<serverip>/system in the node configuration page on MAAS portal.

Power Type: virsh (virtual systems)
Address: qemu+ssh://<username>@<serverip>/system
Power Id: <Virtual machine name>

The <virtual machine name> should be same as it is shown for the command

virsh -c qemu+ssh://<username>@<serverip>/system list --all

With this setup MAAS should automatically start and stop the system. When it says "This node is now allocated to you. It has been asked to start up." make sure the system is actually has been powered up.

Install and Configure Juju

First install the components:

sudo apt-get install juju-core juju

Then create a configuration file ~/.juju/environments.yaml

default: maas
    type: maas
    # Change this to where your MAAS server lives.  It must specify the base path.
    maas-server: 'http://<MAAS Server IP>/MAAS/'
    maas-oauth: '<MAAS API key>'
    admin-secret: '<admin secret>
    default-series: precise
    authorized-keys-path: ~/.ssh/ #authorized_keys # or any file you want.
    # Or:
    # authorized-keys: ssh-rsa keymaterialhere

You can get MAAS API Key from MAAS preferences page.

You can generate this using juju generate-config command and then edit the file to keep only configuration for MAAS and putting in the keys.

Now do the bootstrap using

$ juju bootstrap --upload-tools

And see the status using

$ juju status 

You may now deploy hadoop

$ juju deploy hadoop

This will pick a machine from MAAS machine pool  and install linux na dthen hadoop in it.