Here is OpenStack in a realistic test configuration with multiple physical servers.

Prerequisites Linux & Network

Minimal Install

You need to have a system with a fresh install of Linux. You can download the Minimal CD for Ubuntu releases since DevStack will download & install all the additional dependencies. The netinstall ISO is available for Fedora and CentOS/RHEL.

Install a couple of packages to bootstrap configuration:

apt-get install -y git sudo || yum install -y git sudo

Network Configuration

The first iteration of the lab uses OpenStack’s FlatDHCP network controller so only a single network will be required. It should be on its own subnet without DHCP; the host IPs and floating IP pool(s) will come out of this block. This example uses the following:

Gateway: 192.168.42.1

Physical nodes: 192.168.42.11-192.168.42.99

Floating IPs: 192.168.42.128-192.168.42.254

Configure each node with a static IP. For Ubuntu edit /etc/network/interfaces:

auto eth0

iface eth0 inet static

    address 192.168.42.11

    netmask 255.255.255.0

    gateway 192.168.42.1

For Fedora and CentOS/RHEL edit /etc/sysconfig/network-scripts/ifcfg-eth0:

BOOTPROTO=static

IPADDR=192.168.42.11

NETMASK=255.255.255.0

GATEWAY=192.168.42.1

Installation shake and bake

Add the DevStack User

OpenStack runs as a non-root user that has sudo access to root. There is nothing special about the name, we’ll use stack here. Every node must use the same name and preferably uid. If you created a user during the OS install you can use it and give it sudo privileges below. Otherwise create the stack user:

groupadd stack

useradd -g stack -s /bin/bash -d /opt/stack -m stack

This user will be making many changes to your system during installation and operation so it needs to have sudo privileges to root without a password:

echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

From here on use the stack user. Logout and login as the stack user.

Set Up Ssh

Set up the stack user on each node with an ssh key for access:

mkdir ~/.ssh; chmod 700 ~/.ssh

echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys

Download DevStack

Grab the latest version of DevStack:

git clone https://git.openstack.org/openstack-dev/devstack

cd devstack

Up to this point all of the steps apply to each node in the cluster. From here on there are some differences between the cluster controller (aka ‘head node’) and the compute nodes.

Configure Cluster Controller

The cluster controller runs all OpenStack services. Configure the cluster controller’s DevStack in local.conf:

[[local|localrc]]

HOST_IP=192.168.42.11

FLAT_INTERFACE=eth0

FIXED_RANGE=10.4.128.0/20

FIXED_NETWORK_SIZE=4096

FLOATING_RANGE=192.168.42.128/25

MULTI_HOST=1

LOGFILE=/opt/stack/logs/stack.sh.log

ADMIN_PASSWORD=labstack

DATABASE_PASSWORD=supersecret

RABBIT_PASSWORD=supersecrete

SERVICE_PASSWORD=supersecrete

SERVICE_TOKEN=xyzpdqlazydog

In the multi-node configuration the first 10 or so IPs in the private subnet are usually reserved. Add this to local.sh to have it run after every stack.sh run:

for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done

Fire up OpenStack:

./stack.sh

A stream of activity ensues. When complete you will see a summary of stack.sh‘s work, including the relevant URLs, accounts and passwords to poke at your shiny new OpenStack. The most recent log file is available in stack.sh.log.

Configure Compute Nodes

The compute nodes only run the OpenStack worker services. For additional machines, create a local.conf with:

[[local|localrc]]

HOST_IP=192.168.42.12 # change this per compute node

FLAT_INTERFACE=eth0

FIXED_RANGE=10.4.128.0/20

FIXED_NETWORK_SIZE=4096

FLOATING_RANGE=192.168.42.128/25

MULTI_HOST=1

LOGFILE=/opt/stack/logs/stack.sh.log

ADMIN_PASSWORD=labstack

DATABASE_PASSWORD=supersecret

RABBIT_PASSWORD=supersecrete

SERVICE_PASSWORD=supersecrete

SERVICE_TOKEN=xyzpdqlazydog

DATABASE_TYPE=mysql

SERVICE_HOST=192.168.42.11

MYSQL_HOST=$SERVICE_HOST

RABBIT_HOST=$SERVICE_HOST

GLANCE_HOSTPORT=$SERVICE_HOST:9292

ENABLED_SERVICES=n-cpu,n-net,n-api-meta,c-vol

NOVA_VNC_ENABLED=True

NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"

VNCSERVER_LISTEN=$HOST_IP

VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

Note: the n-api-meta service is a version of the api server that only serves the metadata service. It’s needed because the computes created won’t have a routing path to the metadata service on the controller.

Fire up OpenStack:

./stack.sh

A stream of activity ensues. When complete you will see a summary of stack.sh‘s work, including the relevant URLs, accounts and passwords to poke at your shiny new OpenStack. The most recent log file is available in stack.sh.log.

Cleaning Up After DevStack

Shutting down OpenStack is now as simple as running the included unstack.sh script:

./unstack.sh

A more aggressive cleanup can be performed using clean.sh. It removes certain troublesome packages and attempts to leave the system in a state where changing the database or queue manager can be reliably performed.

./clean.sh

Sometimes running instances are not cleaned up. DevStack attempts to do this when it runs but there are times it needs to still be done by hand:

sudo rm -rf /etc/libvirt/qemu/inst*

sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy

Options pimp your stack

Additional Users

DevStack creates two OpenStack users (admin and demo) and two projects (also admin and demo). admin is exactly what it sounds like, a privileged administrative account that is a member of both the admin and demo projects. demo is a normal user account that is only a member of the demo project. Creating additional OpenStack users can be done through the dashboard, sometimes it is easier to do them in bulk from a script, especially since they get blown away every time stack.sh runs. The following steps are ripe for scripting:

# Get admin creds

. openrc admin admin

# List existing projects

openstack project list

# List existing users

openstack user list

# Add a user and project

NAME=bob

PASSWORD=BigSecrete

PROJECT=$NAME

openstack project create $PROJECT

openstack user create $NAME --password=$PASSWORD --project $PROJECT

openstack role add Member --user $NAME --project $PROJECT

# The Member role is created by stack.sh

# openstack role list

Swift

Swift, OpenStack Object Storage, requires a significant amount of resources and is disabled by default in DevStack. The support in DevStack is geared toward a minimal installation but can be used for testing. To implement a true multi-node test of swift, additional steps will be required. Enabling it is as simple as enabling the swift service in local.conf:

enable_service s-proxy s-object s-container s-account

Swift, OpenStack Object Storage, will put its data files in SWIFT_DATA_DIR (default /opt/stack/data/swift). The size of the data ‘partition’ created (really a loop-mounted file) is set by SWIFT_LOOPBACK_DISK_SIZE. The Swift config files are located in SWIFT_CONF_DIR (default /etc/swift). All of these settings can be overridden in (wait for it...) local.conf.

Volumes

DevStack will automatically use an existing LVM volume group named stack-volumes to store cloud-created volumes. If stack-volumes doesn’t exist, DevStack will set up a 10Gb loop-mounted file to contain it. This obviously limits the number and size of volumes that can be created inside OpenStack. The size can be overridden by setting VOLUME_BACKING_FILE_SIZE in local.conf.

stack-volumes can be pre-created on any physical volume supported by Linux’s LVM. The name of the volume group can be changed by setting VOLUME_GROUP in localrc. stack.sh deletes all logical volumes in VOLUME_GROUP that begin with VOLUME_NAME_PREFIX as part of cleaning up from previous runs. It is recommended to not use the root volume group as VOLUME_GROUP.

The details of creating the volume group depends on the server hardware involved but looks something like this:

pvcreate /dev/sdc

vgcreate stack-volumes /dev/sdc

Syslog

DevStack is capable of using rsyslog to aggregate logging across the cluster. It is off by default; to turn it on set SYSLOG=True in local.conf. SYSLOG_HOST defaults to HOST_IP; on the compute nodes it must be set to the IP of the cluster controller to send syslog output there. In the example above, add this to the compute node local.conf:

SYSLOG_HOST=192.168.42.11

Using Alternate Repositories/Branches

The git repositories for all of the OpenStack services are defined in stackrc. Since this file is a part of the DevStack package changes to it will probably be overwritten as updates are applied. Every setting in stackrc can be redefined in local.conf.

To change the repository or branch that a particular OpenStack service is created from, simply change the value of *_REPO or *_BRANCH corresponding to that service.

After making changes to the repository or branch, if RECLONE is not set in localrc it may be necessary to remove the corresponding directory from /opt/stack to force git to re-clone the repository.

For example, to pull nova, OpenStack Compute, from a proposed release candidate in the primary nova repository:

NOVA_BRANCH=rc-proposed

To pull glance, OpenStack Image service, from an experimental fork:

GLANCE_BRANCH=try-something-big

GLANCE_REPO=https://github.com/mcuser/glance.git

Notes stuff you might need to know

Reset the Bridge

How to reset the bridge configuration:

sudo brctl delif br100 eth0.926

sudo ip link set dev br100 down

sudo brctl delbr br100

Set MySQL Password

If you forgot to set the root password you can do this:

mysqladmin -u root -pnova password 'supersecret'