wutitded88 / hadoop-ambari-automation

Hadoop cluster installation via Ambari on Vagrant VMs with Ansible.

Hadoop via Ambari automation

Hadoop (HDP) installation via Ambari on Vagrant VMs with Ansible to configure VMs and work on APIs.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contributing
5. License
6. Contributors

About The Project

This project aims at deploying a Hadoop single node cluster using Vagrant to setup virtual machines and Ansible to configure the cluster and the different services. These Ansible playbooks will build a Hortonworks cluster (Hortonworks Data Platform and / or Hortonworks DataFlow) using Ambari Blueprints.

Disclaimer

We decided to implement a single node cluster because our computers could not handle more complex installations (we tried). Feel free to modify the inventory and the blueprint to modify the cluster.

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

To continue with the project, you will need the following software installed on your machine :

• VirtualBox
• Vagrant
• Ansible

Installation

Clone the repo

git clone https://github.com/alexhff/hadoop-ambari-automation.git

Navigate to the repo

cd hadoop-ambari-automation

Copy the private key to your home directory (or some place convenient for you) so that it’s easily accessible for uploading via Ambari Web:

vagrant

The above command shows the command usage and also creates a private key as ~/.vagrant.d/insecure_private_key. This key will be used in the following steps.

Copy the private key to the local repo:

cp ~/.vagrant.d/insecure_private_key .

Start VMs

Now you can start VM with the following command:

vagrant up

You can check the status of your VM using:

vagrant status

The VM should be up and running.

Launch Ansible installation

To build the cluster using the Ansible playbooks, use the following command:

./install_cluster.sh

Usage

Adding nodes to the cluster

If you want to add nodes, you have to edit two files. First, update the Vagrantfile to add more VMs. Second, edit the inventory/static file to update the Ansible configuration.

Manually installing Ambari

Ansible is responsible for installing Ambari. However, if you want to install Ambari manually, follow the procedures:

Log into the VM:

vagrant ssh c7401

Note that this logs you in as user vagrant. Once you are logged in, you can run:

sudo su -

to make yourself root. Download the Ambari repository file to a directory on your installation host:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

Install ambari:

yum install ambari-server -y

Run the following to set up and start ambari-server:

ambari-server setup -s
ambari-server start

Once Ambari Server is started, hit http://http://192.168.74.101:8080/ (URL depends on the OS being tested) from your browser on your local computer. Note that Ambari Server can take some time to fully come up and ready to accept connections. Keep hitting the URL until you get the login page.

Once you are at the login page, login with the default username admin and password admin.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contributors

• Alexander Hoffmann
• Adrien Bourget
• Jean-Baptiste Mader