2. Getting started

Familiarize yourself with Fabric and Puppet.

2.1. Installation

Install this package via:

  • Mercurial:

    hg clone https://www.bitbucket.org/sourpoi/minimin /tmp/minimin
  • pip:

    pip install minimin
  • source:

    wget http://pypi.python.org/packages/...

For this quickstart, we’ll assume that the extracted package is located at /tmp/minimin and that our target “fabfile” is /tmp/minimin/lib/minimin/fabfile.py (we’ll set this as the default fabfile in Admin user’s configuration).

2.2. Layout

We’ll be using a layout where our admin user’s home is at /home/admin, our primary project is at /tmp/myproj, a common project with shared configurations is at /tmp/common, and 3rd-party support is in /tmp/support:

+-- home
|   +-- admin
|       +-- .fabricrc
|       +-- .ssh
|           +-- config
|           +-- id_some_key
|           +-- id_some_key.pub
+-- tmp
    +-- myproj
    |   +-- etc
    |   |   +-- site.ini
    |   +-- puppet
    |       +-- modules
    |           +-- myproj
    |               +-- manifests
    |                   +-- site.ini
    +-- common
    |   +-- etc
    |   |   +-- common.ini
    |   +-- puppet
    |       +-- modules/
    +-- support

2.2.1. Skeleton

Create a sample environment with directories for third-party support, shared and reusable configurations, private configurations, and the admin user’s home:

mkdir -p /tmp/support                      # public, third-party packages
mkdir -p /tmp/common/{etc,puppet/modules}  # common, shared configs
mkdir -p /tmp/myproj/{etc,puppet/modules}  # project, site-specific configs
mkdir -p /home/admin                       # admin's home
touch /home/admin/.fabricrc                # admin's Fabric config
mkdir /home/admin/.ssh                     # admin's SSH home
touch /home/admin/config                   # admin's SSH config

2.2.2. Admin user’s configuration

Include some suggested environment settings in the Fabric rcfile (~/.fabricrc):

cat >> /home/admin/.fabricrc <<EOF
fabfile = /tmp/minimin/lib/minimin/fabfile.py
ssh_config_path = /home/admin/.ssh/config
use_ssh_config = True
warn_only = True

Include the required Fabric environment setting in the Fabric rcfile (~/.fabricrc):

cat >> /home/admin/.fabricrc <<EOF
admin_support_path = /tmp/support
admin_puppet_path = /tmp/myproj/puppet/modules /tmp/common/puppet/modules
admin_cfg_path = /tmp/myproj/etc /tmp/common/etc
admin_cfg_include = site.ini common.ini
admin_ssh_home = /home/admin/.ssh

2.2.3. Project configuration

First let’s satisfy our site.ini inclusion. We intend this to be a private, project-specific configuration so we put it in /tmp/myproj/etc (instead of /tmp/common/etc). Our admin_cfg_path will first look for site.ini in /tmp/myproj/etc:

cat > /tmp/myproj/etc/site.ini <<EOF
websvr = f15
mailsvr = s10

common.nodes = websvr mailsvr
common.puppet.mods = myproj
common.puppet.modpath = /etc/puppet/modules
common.puppet.site = /etc/puppet/modules/myproj/manifests/site.pp

kick-web.txt = restart the web server
kick-web.f15 = service httpd restart

kick-mail.txt = restart the mail server
kick-mail.s10 = svcadm restart sendmail

2.2.4. Common/shared configuration

Next let’s include our shared configuration common.ini in /tmp/common/etc. This contains options that aren’t considered private, may be shared between projects, and may have come from a public source:

cat > /tmp/common/etc/common.ini <<EOF
pkg-grep.txt = look for a package name matching {str}
pkg-grep.f15 = rpm -qa | grep {str}
pkg-grep.s10 = pkginfo | grep -i {str}

url-sfw10 = ftp://ftp.sunfreeware.com/pub/freeware/intel/10

nc.txt.s10 = Netcat, the all-purpose network tool SMCnc
nc.url.s10 = %(url-sfw10)s/nc-110-sol10-x86-local.gz
nc.src.s10 = nc-110-sol10-x86-local.gz
nc.dst.s10 = /tmp/nc-110-sol10-x86-local.gz
nc.pth.s10 = /usr/sbin:/usr/bin
nc.req.s10 = which pkgadd pkgrm gunzip
nc.chk.s10 = /usr/local/bin/nc -h 2>&1 | grep 'v1.10'
nc.del.s10 = pkgrm SMCnc
nc.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip nc-110-sol10-x86-local.gz && \
    pkgadd -d nc-110-sol10-x86-local all

2.3. Workflow

  1. Install Minimin. See Installation.
  2. Set up the environment. See Layout.
  3. Generate SSH keys, configure nodes to use them. See SSH key distribution.
  4. Install Puppet on nodes. See Puppet installation and [install].
  5. Apply Puppet configurations to nodes. See Applying Puppet configuration and [role]. Use a quick-n-dirty Puppet module to jumpstart service configurations. See Quick-n-dirty Puppet modules.
  6. Query nodes, manage services using Fabric. See Ad-hoc commands, queries, and service management and [exe].

2.3.1. SSH key distribution

Generate an SSH key to use on both nodes by providing the local username (admin), username on the remote node (root), and an indicator of the key’s purpose (SHARED):

fab sshkeygen:admin,root,SHARED

This will create:


Note the suggested ssh_config additions, and add the following to /home/admin/.ssh/config:

cat >> /home/admin/.ssh/config <<EOF
Host *
IdentitiesOnly yes
GSSAPIAuthentication no

Host websvr
IdentityFile /home/admin/.ssh/id_admin_root_SHARED
User root

Host mailsvr
IdentityFile /home/admin/.ssh/id_admin_root_SHARED
User root

Next, configure your target nodes to respect your SSH key. This generally consists of:

  1. Configuring the firewall to permit SSH access and restarting the firewall.
  2. Configuring /etc/ssh/sshd_config to use public keys and restarting sshd.
  3. Appending your key(s) to the appropriate user’s ~/.ssh/authorized_keys file.

The details will vary between operating systems, but if you have SSH access and can login manually via root password, you might try using a known-good sshd_config and the included “pure-ssh” push/append included.

  1. Create a basic sshd_config to distribute:

    cat >> /tmp/sshd_config <<EOF
    Protocol 2
    Port 22
    HostbasedAuthentication yes
    HostKey /etc/ssh/ssh_host_rsa_key
    PasswordAuthentication yes
    PermitRootLogin yes
    AuthorizedKeysFile .ssh/authorized_keys
    ChallengeResponseAuthentication no
    GSSAPIAuthentication no
    Subsystem sftp /usr/libexec/openssh/sftp-server
    AllowTcpForwarding no
    X11Forwarding no
  2. Reconfigure/restart sshd and append your new SSH key to root’s authorized_keys:

    fab -uroot -Rcommon -- cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
    fab -uroot -Rcommon -- ssh-keygen -trsa -f /etc/ssh/ssh_host_rsa_key
    fab -uroot -Rcommon sshput:/tmp/sshd_config,/etc/ssh/sshd_config
    fab -uroot -Rcommon -- service sshd restart
    fab -uroot -Rcommon sshput:$PUBKEY,$AUTHKEY,append=True
    fab -uroot -Rcommon -- chmod 600 $AUTHKEY
    fab sshup

2.3.2. SSH headache

Firewall access and SSH support on each node is part of the minimal overhead, but it can be tedious. Many OS installs do not include firewall access, SSH public key or SSH root access by default. You can ease the pain by:

  1. enabling SSH during the OS install with a kickstart <http://fedoraproject.org/wiki/Anaconda/Kickstart> file or installation server,
  2. speeding up manual password entries using screen or tmux copy-paste, or
  3. scripting the password entries using expect or key distribution with ssh-copy-id.

2.3.3. Puppet installation

If a package manager is configured and the node has internet access (for example, on our websvr above running Fedora 15), you might be able to install Puppet by running:

fab -Hwebsvr -- yum -y install puppet

However, for our mailsvr node running Solaris 10, we’ll configure manual installs. This is a little involved, and requires the following packages to satisfy dependencies libiconv -> gcc -> ruby -> facter -> puppet:

url-ibs10 = http://www.ibiblio.org/pub/packages/solaris/freeware/intel/10
url-ruby18 = http://ftp.ruby-lang.org/pub/ruby/1.8
url-puplab = http://downloads.puppetlabs.com

libiconv.txt.s10 = libiconv x86 SMCliconv
libiconv.url.s10 = %(url-ibs10)s/libiconv-1.11-sol10-x86-local.gz
libiconv.src.s10 = libiconv-1.11-sol10-x86-local.gz
libiconv.dst.s10 = /tmp/libiconv-1.11-sol10-x86-local.gz
libiconv.chk.s10 = pkginfo -l SMCliconv | grep VERSION | grep '1.11'
libiconv.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip libiconv-1.11-sol10-x86-local.gz && \
    pkgadd -d libiconv-1.11-sol10-x86-local all

gcc86.txt.s10 = GNU x86 C compiler SMCgcc
gcc86.url.s10 = %(url-ibs10)s/gcc-3.4.6-sol10-x86-local.gz
gcc86.src.s10 = gcc-3.4.6-sol10-x86-local.gz
gcc86.dst.s10 = /tmp/gcc-3.4.6-sol10-x86-local.gz
gcc86.req.s10 = libiconv.s10
gcc86.chk.s10 = /usr/local/bin/gcc -v 2>&1 | grep 'gcc version 3.4.6'
gcc86.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip gcc-3.4.6-sol10-x86-local.gz && \
    pkgadd -d gcc-3.4.6-sol10-x86-local all

ruby.txt.s10 = Ruby general-purpose programming language from %(url-sfw10)s
ruby.url.s10 = %(url-ruby18)s/ruby-1.8.7-p72.tar.gz
ruby.src.s10 = ruby-1.8.7-p72.tar.gz
ruby.dst.s10 = /tmp/ruby-1.8.7-p72.tar.gz
ruby.req.s10 = gcc86.s10
ruby.chk.s10 = /usr/local/bin/ruby -v | grep "ruby 1.8.7"
ruby.pth.s10 = /usr/sbin:/usr/bin:/usr/sfw/bin:/usr/ccs/bin
; installs to /usr/local/bin/ruby
ruby.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip ruby-1.8.7-p72.tar.gz && \
    tar -xvf ruby-1.8.7-p72.tar && \
    rm -r ruby-1.8.7-p72.tar && \
    cd ruby-1.8.7-p72 && \
    ./configure && \
    make && \
    make install && \
    ln -s /usr/local/bin/ruby /usr/bin/ruby

facter.txt.s10 = Facter host-assessment tool from %(url-sfw10)s
facter.url.s10 = %(url-puplab)s/facter/facter-1.6.6rc2.tar.gz
facter.src.s10 = facter-1.6.6rc2.tar
facter.dst.s10 = /tmp/facter-1.6.6rc2.tar
facter.req.s10 = ruby.s10
facter.chk.s10 = /usr/local/bin/facter -v | grep "1.6.6"
facter.pth.s10 = /usr/sbin:/usr/bin:/usr/local/bin
facter.exe.s10 = #!/bin/sh
    cd /tmp && \
    tar -xf facter-1.6.6rc2.tar && \
    cd facter-1.6.6rc2 &&
    ./install.rb && \
    ln -s /usr/local/bin/facter /usr/bin/facter

; Note underscore to dash change after tarball extract.
puppet.txt.s10 = Puppet configuration manager from source
puppet.url.s10 = %(url-puplab)s/puppet/puppet_2.7.10.orig.tar.gz
puppet.src.s10 = puppet_2.7.10.orig.tar.gz
puppet.dst.s10 = /tmp/puppet_2.7.10.orig.tar.gz
puppet.req.s10 = ruby.s10 facter.s10
puppet.chk.s10 = /usr/local/bin/puppet -V | grep "2.7.10"
puppet.pth.s10 = /usr/sbin:/usr/bin:/usr/local/bin
puppet.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip puppet_2.7.10.orig.tar.gz && \
    tar -xf puppet_2.7.10.orig.tar && \
    cd puppet-2.7.10 && \
    ./install.rb && \
    ln -s /usr/local/bin/puppet /usr/bin/puppet

With all of the above in our Minimin config, we can install Puppet on mailsvr using:

fab -Hmailsvr install:libiconv
fab -Hmailsvr install:gcc86
fab -Hmailsvr install:ruby
fab -Hmailsvr install:facter
fab -Hmailsvr install:puppet

Note: The URL options (url-ibs10, url-ruby18 and url-puplab) aren’t used for anything but documentation (yet).

More information on install configuration in [install].

2.3.4. Applying Puppet configuration

To apply Puppet configurations, run updates on all nodes sharing a role:

fab -Rcommon apply

..or specify one or more hosts (nodes) as well as the role type:

fab -Rcommon -Hwebsvr apply

A dummy role named nodes contains all the configured nodes. If the puppet options are defined for the dummy role nodes:

nodes.puppet.mods = myproj
nodes.puppet.modpath = /etc/puppet/modules
nodes.puppet.site = /etc/puppet/modules/myproj/manifests/site.pp

..then you may use the nodes role as you would any other:

fab -Rnodes -Hwebsvr apply

More information on Puppet configuration using roles in [role].

Rationale behind the role dependency in Role dependency.

2.3.5. Ad-hoc commands, queries, and service management

List configured commands and descriptions from the [exe] section:

fab showexe

Run the commands on specific nodes:

fab -Hwebsvr exe:kick-web
fab -Hmailsvr exe:kick-mail

Given our configuration above, we “just know” that kick-web won’t work on node mailsvr because [exe] doesn’t have the option kick-web.s10 and kick-mail won’t work on websvr because [exe] doesn’t have the option kick-mail.f15.

A configured command may require keyword arguments:

net-ping.txt = ping {node} 3 times and show stats
net-ping.f15 = /bin/ping -c3 {node}
net-ping.s10 = /usr/sbin/ping -s {node} 56 3

Since net-ping is configured for all our node suffixes, we can ping a common target ( from both nodes like this:

fab -Rcommon exe:net-ping,node=

Include the required keywords in the description of the configured command.

More information on Puppet configuration using roles in [exe].

2.3.6. On my laptop

Here’s my typical workflow:

fab vbmlist                     # available VirtualBox VMs
fab vbmstart:{vmname}           # start a VM
fab vbmscan:      # reconcile VM DCHP results with ssh_config
fab sshup                       # confirm access
fab showcfg:flat | grep role.   # remind me what I'm working on
fab install:{pkg}               # install something on our nodes
fab -H{node} config             # apply Puppet configuration to a test node
fab -R{role} config             # apply Puppet configuration role nodes
fab showexe | grep {cmd}        # describe pre-configured {cmd}
fab -R{role} exe:{cmd},opt=val  # check some aspect of our nodes