The Hortonworks Data Platform (HDP) Savanna plugin provides a way to provision HDP clusters on OpenStack using templates in a single click and in an easily repeatable fashion. As seen from the architecture diagram below, the Savanna controller serves as the glue between Hadoop and OpenStack. The HDP plugin mediates between the Savanna controller and Apache Ambari in order to deploy and configure Hadoop on OpenStack. Core to the HDP Plugin is Apache Ambari that is used as the orchestrator for deploying the HDP stack on OpenStack.
The HDP plugin uses Ambari Blueprints aka templates for cluster provisioning.
Apache Ambari Blueprints is a portable document definition, which provides a complete definition for an Apache Hadoop cluster, including cluster topology, components, services and their configurations. Ambari Blueprints can be consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The benefits of this approach is that it allows for Hadoop clusters to be configured and deployed using an Ambari native format that can be used with as well as outside of OpenStack allowing for clusters to be re-instantiated in a variety of environments.
For more information about Apache Ambari Blueprints, refer to: https://issues.apache.org/jira/browse/AMBARI-1783. Note that Apache Ambari Blueprints are not yet finalized.
The HDP Plugin performs the following four primary functions during cluster creation:
The Savanna HDP plugin can make use of either minimal (operating system only) images or pre-populated HDP images. The base requirement for both is that the image is cloud-init enabled and contains a supported operating system (see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
The advantage of a pre-populated image is that provisioning time is accelerated, as packages do not need to be downloaded and installed which make up the majority of the time spent in the provisioning cycle.
As with the provided pre-populated image, a pre-populated image can install any of the following packages:
Any packages that are not installed in a pre-populated image will automatically be installed during the HDP provisioning process. There are two VM images provided for use with the HDP Plugin:
HDP plugin requires an image to be tagged in Savanna Image Registry with two tags: ‘hdp’ and ‘<hdp version>’ (e.g. ‘1.3.2’).
Also in the Image Registry you will need to specify username for an image. It should be ‘root’ for both images.
Please refer to the reference VM image provided for specific details.
The HDP plugin currently has the following limitations:
Currently, the HDP plugin provides support for HDP 1.3. Once HDP2 is released, support for this version will be provided.
Swift integration is not yet implemented.
It is not possible to decrement the number of node-groups or hosts per node group in a Savanna generated cluster.
Note: Other services may be added using Ambari after initial cluster creation.
Prior to Hadoop cluster creation, the HDP plugin will perform the following validation checks to ensure a successful Hadoop deployment:
A Hortonworks supported version of HDP OpenStack plugin will become available at a future date. For more information, please contact Hortonworks.