5. Modules

5.1. minimin.fabfile

Sample fabfile.py.

This may be used to set default Fabric behavior setting the fabfile option in ~/.fabricrc:

fabfile = /home/admin/support/minimin.hg/lib/minimin/fabfile.py

See the source for details.

This may be used to set default Fabric behavior setting the fabfile option in ~/.fabricrc:

fabfile = /home/admin/support/minimin.hg/lib/minimin/fabfile.py

See the source for details.

5.2. minimin.fabcfg

Shared configuration for Fabric tasks.

This module intends to serve as a singleton configuration object by providing the following attributes:

  • cfg_incl: list of .INI files contributing to our effective configuration
  • cfg_path: list of directories to find included .INI files (cfg_incl)
  • puppet_path: list of directories to find Puppet modules
  • support_path: list of directories to find support software/installers
  • ini: ConfigParser.SafeConfigParser object, our effective configuration
  • inimap: dictionary with nested ini dotted-options
  • extmap: dictionary mapping nodes to their OS/arch extensions
  • nodes: list of node names in our effective configuration
  • roledefs: dictionary mapping role names to node lists
  • role: active role (in roledefs)
  • sshcfg: ssh.SSHConfig dictionary representing our ssh_config
class minimin.fabcfg.IniMap(ini)[source]

Bases: object

A flat key, val mapper that pulls info from a dict.

gen(*match)[source]
minimin.fabcfg.ext()[source]

Retrieve current node’s OS/arch extension.

Returns :node’s OS/arch extension as mapped in .extmap.
Exception :Raises KeyError if extension is not found.

5.3. minimin.fabmin

Minimin’s basic adminisrative commands

Three commands are the basis of minimal administration - manual installs, one-off commands, and system configuration:

  • install() - check, copy, and install packages in [install]
  • exists() - run the package’s .chk script to test for existence
  • delete() - run the package’s .del script to delete it
  • exe() - interface to commands configured in [exe]
  • config() - apply Puppet configurations defined in [role]

To help make sense of your configuration and environment:

minimin.fabmin.config(noop=False)[source]

Configure nodes by copying Puppet modules and running ‘puppet apply’.

Parameters :
  • noop: do a dry-run instead of applying the manifest (default False)

Applying the Puppet configuration will copy modules to the node’s modulepath and run the site manifest on the node, distinguishing the node with using the node_name_value.

We only copy modules (using the convenient fabric.api.put() - we do not prune. This means that a new manifest tree might not take effect if, for example, a new manifests/ntp.pp is prefered over an existing but outdated manifests/ntp/init.pp when using class {'mod::ntp':}.

Todo

An rsync --delete might be useful here.

Example:

fab -Rrole config                 # apply to hosts sharing role
fab -Rrole -Hh1,h2 config         # apply to given hosts using role
fab -Rrole -Hh1 config:noop=True  # dry-run only, don't actually apply
minimin.fabmin.delete(pkg)[source]

Delete a manually-installed package.

Parameters :
  • pkg: package name (first option segment in [install] section)
Returns :

output from {pkg}.del.{os} command

Example:

fab delete:monit
minimin.fabmin.exe(cmd, final=False, sshlog=None, **kw)[source]

Execute a pre-configured command.

Parameters :
  • cmd: command option name
  • final: skip command name look-up if True (optional, default False)
  • sshlog: file to log SSH debugging info
Keywords :

optcmd command keywords (see exeinfo())

This exists to fill the gap between elegant configuration management and ad-hoc, one-off commands by way of configuring those commands as simple INI-style options in this format:

[exe]
{cmd} = command summary
{cmd}.{os} = mycmd -a {arg1}

Example:

fab -Hh1,h2 exe:sys-who
fab -Hlocalhost exe:sys-who-f15  # unconfigured localhost requires os
minimin.fabmin.exists(pkg)[source]

Check the existence of a manually-installed package.

Parameters :
  • pkg: package name (first option segment in [install] section)
Returns :

output from {pkg}.chk.{os} command

Example:

fab exists:monit
minimin.fabmin.install(pkg)[source]

Install a package manually.

Parameters :
  • pkg: package name (first option segment in [install] section)
Returns :

output from exe() or None if check {pkg}.chk.{os} succeeds

Install options for package {pkg} on OS/architecture {os}:

[install]
{pkg} = Short description, include version in long form
{pkg}.url.{os} = http://host/app.tgz      ; download URL
{pkg}.src.{os} = app.tgz                  ; file in admin_support_path
{pkg}.dst.{os} = /tmp/app.tgz             ; absolute destination path
{pkg}.pth.{os} = /bin:/sbin               ; remote $PATH
{pkg}.req.{os} = which tar                ; test for requirements 
{pkg}.chk.{os} = app -V | grep '1.0'      ; test for existence
{pkg}.exe.{os} = tar -C/tmp -xzf app.tgz  ; install script
{pkg}.del.{os} = rm -rf /tmp/app          ; deletion script

This function will:

  1. Run .req and .chk scripts using the .pth $PATH on the node and continue the installation only if .req succeeds and .chk fails. If requirements are not met we abort, erring on the side of uniformity.
  2. Copy the first matching .src file found in env.admin_support_path to remote .dst.
  3. Run .exe using .pth on the node.

Example:

fab install:gcc
minimin.fabmin.qndput(osname=None, *pkgs)[source]

Copy quick-n-dirty installers to the expected package source paths.

Parameters :
  • osname: operating system name (third portion of quick-n-dirty option)
  • pkgs: list of package to install (requires an *.install.* option)

Untested: use with a sense of humor.

minimin.fabmin.showcfg(*args, **kwargs)[source]

Show the effective configuration in raw format.

Parameters :
  • flat: replace default INI-style output with a flat list of opts prepended by section names (optional, default False)

Example:

fab --hide=running,status showcfg > /tmp/showcfg.ini
fab showcfg:flat | grep role
minimin.fabmin.showenv(*args, **kwargs)[source]

Show Fabric’s environment settings (fabric.api.env).

Example:

fab --hide=running,status showenv
fab showenv | grep admin_
minimin.fabmin.showexe(*args, **kwargs)[source]

Print info on a configured command (or list all summaries).

Parameters :
  • cmd: command name (optional, default None)
Keywords :

keywords to populate command arguments (optional)

Example:

fab --hide=status,running showexe # list all command names and summaries
fab showexe:proc-listen,port=22   # keyword fills {port} placeholder
minimin.fabmin.showpaths(*args, **kwargs)[source]

Show our potential and actual file inclusions for a given role.

We use our configured paths and file inclusions (set in fabricrc or provided via fab --set) and provide the list of paths we look for followed by the list of files and directories we actually have. This is a quick way to double check running path sanity. Compare to:

fab showenv | grep admin_

Example:

fab --hide=running,status showpaths
minimin.fabmin.support(*srcs)[source]

Copy/sync support sourcepackages.

Parameters :
  • srcs: a list of specific packages to sync (optional)

Example:

fab -Rrole support
fab -Rrole support:awstats-7.0.tar.gz

5.4. minimin.fabssh

SSH commands

SSH helpers include:

  • sshkeygen() - create local SSH keys as id_localuser_remotuser_host
  • sshput() - use basic SSH to copy a file (when node is missing SCP/SFTP)
  • sshup() - check access to nodes configured in ssh_config
minimin.fabssh.sshfwdlocal(src, dst=None)[source]

Forward a local port to a port on a remote host.

Parameters :
  • src: source port (port on localhost)
  • dst: destination port (port on remote host)

This is intended to make it easier to use local commands to manage remote services:

fab -Hcouch sshfwdlocal:5984
curl -XPUT 'http://localhost:5984/newdb'

This function relies on an ssh_config to determine remote user, host, and port.

Example:

fab -Hhost sshfwdlocal:8080
fab -Hhost sshfwdlocal:8080,80
minimin.fabssh.sshkeygen(*args, **kwargs)[source]

Create a new SSH key pair: write keys to SSH home and print config.

Parameters :
  • localuser: username of local admin (you)
  • remoteuser: username of user on remote node
  • nodes: list of nodes to generate keys for

For nodes h1 and h2, create SSH keys in your SSH home (configured as the admin_ssh_home value in ~/.fabricrc):

fab sshkeygen:me,root,h1,h2  # create id_me_root_h1 and id_me_root_h2

Using manual password logins to begin with, we copy the appropriate SSH server configuration (enabling the use keys) and our keys to the target nodes and restart the service:

fab -Hh1,h2 sshput:sshd_config,/etc/ssh/sshd_config,backup
fab -Hh1,h2 sshput:id_admin.pub,/root/.ssh/authorized_keys
fab -Hh1,h2 -- service sshd restart

Example:

fab sshkeygen:admin,root,n1,n2
fab sshkeygen:admin,root,SHARED
fab --set admin_ssh_home='/alt/ssh' sshkeygen:admin,root,n1
minimin.fabssh.sshput(src, dst, append=False)[source]

Copy a file using local SSH (work around lack of sftp on target node).

Parameters :
  • src: path to local file
  • dst: path of destination file on remote node
  • append: (optional) append to file rather than overwriting it

This bare-bones copying method addresses SSH configurations on new nodes which do not support sftp or scp (and frustrate conveniences like Fabric’s put and append).

Ideally, most file copies should be syncs via CMS.

This function relies on an ssh_config to determine remote user, host, and port.

Example:

fab -Hh1 sshput:id_admin.pub,/root/.ssh/authorized_keys,append
fab -Hh1 sshput:motd,/etc/motd
minimin.fabssh.sshup(*args, **kwargs)[source]

Check availability of configured SSH nodes/hosts.

For each host configured in the ssh_config_path (typically ~/.ssh/ssh_config), retrieve the host key and attempt to run uname -a to confirm a connection.

Example:

fab sshup

5.5. minimin.fabvbm

Fabric VirtualBox commands

minimin.fabvbm.vbmlist(*args, **kwargs)[source]

Print a list of VirtualBox VMs (existing and running).

Example:

fab --hide=running,status vbmlist
fab vbmlist
minimin.fabvbm.vbmscan(*args, **kwargs)[source]

Scan addresses and match them to listening VM names and MACs.

Parameters :
  • addrs: range of host address to scan using Nmap

This function helps work with VMs that use DHCP and wind up with an address that isn’t configured in ssh_config. We run a ping sweep over a range of addresses to collect MAC addresses in our ARP table, then we match MACs with those in our guest VM configurations.

See sshup() and vbmlist().

Example:

fab --hide=running,status vbmscan:192.168.1-40
fab vbmscan:192.168.1.1-40
minimin.fabvbm.vbmshow(*args, **kwargs)[source]

Print information on a VirtualBox VM.

Parameters :
  • vmname: name of the virtual machine

Example:

fab vbmshow:vmname
minimin.fabvbm.vbmstart(*args, **kwargs)[source]

Start a VirtualBox VM in headless mode.

Parameters :
  • vmname: name of the virtual machine

This is mainly for convenience after a VM is set up and SSH is running.

See vbmlist().

Example:

fab vbmstart:vmname
minimin.fabvbm.vbmstop(*args, **kwargs)[source]

Stop a VirtualBox VM.

Parameters :
  • vmname: name of the virtual machine

See vbmlist().

Example:

fab vbmstop:vmname

5.6. minimin.fabutil

Helper functions.

minimin.fabutil.firstreal(parents, child)[source]

First path pointing to child file in a list of parent directories.

Parameters :
  • parents: list of parent directories
  • child: file to find in parent directories
Returns :

full path to child file

Exceptions :
  • IOError: if real path to a child doesn’t exist in parents
minimin.fabutil.fmtprint(items, spc, **kw)[source]

Print a sequence of pairs with a certain spacing.

Parameters :
  • items: a sequence of options and values to print
  • spc: leading distance between the start of the line and value
Keywords :

if provided, keywords will format {key} items in string

minimin.fabutil.lsplit(s)[source]

Split a string based on newlines, strip non-empty lines.

Parameters :
  • s: string with newlines
Returns :

list of non-empty strings

minimin.fabutil.missing(paths, files)[source]

Return file names in files for which there are no matching path in paths.

minimin.fabutil.path(parent, child)[source]

Absolute path after appending a child file/directory to a parent path.

Parameters :
  • parent: parent directory
  • child: child file or directory
Returns :

resulting path of the child appended to the parent

minimin.fabutil.paths(parents, children)[source]

List paths of child files found in parent paths.

Parameters :
  • parents: list of parent directories
  • children: list of child files or directories to look for in parents
Yields :

(child, path) for each child in children, or (child, None) if no path was found

minimin.fabutil.printonly(s)[source]
Todo :Kill this and use puts() since we decided to document --hide?
minimin.fabutil.realpaths(parents, children)[source]

List of paths in parents that point to children.

Parameters :
  • parents: list of parent directories
  • children: list of child files or directories to look for in parents
Yields :

full paths pointing to children

Exceptions :
  • IOError: if real path to a child doesn’t exist in parents

5.7. minimin.fabqnd

Puppet assistance with configuration, service management and installation.

5.7.1. config class

Optional config class parameters:

  • ${pathid}: final, rendered string for the file (if set, ignores existing template and ${pathid}_tmpl)
  • ${pathid}_path: alternate file path
  • ${pathid}_tmpl: Puppet template, appended to default template
  • ${pathid}_{tmplvar}: for every {tmplvar} associated with a {pathid}, expose it within the class as ${pathid}_{tmplvar}

The config class checks defaults for:

  • $path_{pathid}: (optional) ... if not found, ignores (class parameters ${pathid}* are ignored if the $path_{pathid} was not defined in the defaults)

Todo

nested directories prior to actual file declaration.

5.7.2. service class

Optional service class parameters:

  • ${svcid}_state: Puppet service state (default ‘running’)
  • ${svcid}_subs: list of paths to subscribe the service to

The service class checks defaults for:

  • $svc_{svcid}_subs: (optional) list of paths the service subscribes to

5.7.3. install class

Optional install class parameters:

  • ${pkgid}_state: Puppet package state (default ‘present’)
  • ${pkgid}_provider: useful for manual installs and testing
  • ${pkgid}_source: useful for manual installs and testing

The install class checks defaults for:

  • $pkg_{pkgid}: name of the package to manage
  • $pkg_{pkgid}_provider: (optional) name of the Puppet package provider/manager, useful for automated updates setting state to ‘latest’
Note:${pkgid}_source is only available as a class parameter to prevent default versioning.

5.7.4. default class

The default class maps generic variables used by the other classes to actual values:

$path_{pathid} -> default path to file $pkg_{pkgid} -> package name as understood by the provider $pkg_{pkgid}_provider -> provider as listed in the Puppet package type

5.7.5. Notes

  • Lighter bootstraps are available using puppet-module:

    gem install puppet-module
    puppet-module generate $USER-mymod
  • qndmod() helps bootstrap a new Puppet-managed service by creating a quick and dirty (“qnd”) submodule; manifests that provide a simple layout that is easy to customize but works in the meantime.

  • There’s a lot of if-then-else boilerplate in here. It’d be nice to use something like a native Puppet ternary one-liner - I didn’t want to involve custom Ruby functions on the quick-n-dirty level. Unfortunately these patterns didn’t work (and I haven’t investigated further):

    ('x?y:z' and '(x && y)||z')
    'x = y or z' is 'x = bool'
    

Todo

Of pathid, svcid and pkgid only pathid is treated as an abstract identifier. svcid and pkgid are used as the actual system service and package names. While no checks are made on any of these strings, we assume a real ID wouldn’t use un-Puppet-friendly characters. However, we might not be able to say the same about a services and packages which might have dashes which “screw up qualified variable access” or invalid dots. Maybe add actual IDs for these later. It’s more readable, less redundant, and less likely than differing configuration file paths across operating systems.

class minimin.fabqnd.Config(*args)[source]

Bases: minimin.fabqnd.Manifest

add(tmplmod=None, user=None, mode=None, _id=None, path=None, *tmplvars)[source]

Add parameters to the ‘config’ class template.

Parameters :
  • tmplmod: name of the module containing templates/_id
  • user: default username of the file owner (also used for group)
  • mode: default file mode
  • _id: common ID/template name for the config file across platforms
  • path: platform-specific path to the config file
  • tmplvars: list of variables expected in templates/_id

Example INI option and value:

ntp.config.default = sitemod:ntp:0644:ntp_conf:/etc/ntp.conf:stratnum:servers
tmpl_class = 'class {mod}::{sub}::config (\n{params}\n)\n{{\n include {mod}::{sub}::default\n{files}\n}}'
tmpl_file = "\n $path_{pathid} = ${mod}::{sub}::default::path_{pathid}\n\n if $path_{pathid} {{\n\n if ${pathid}_path {{\n $final_{pathid}_path = ${pathid}_path\n }} else {{\n $final_{pathid}_path = $path_{pathid} }}\n\n if ${pathid} {{\n $str_{pathid} = ${pathid}\n }} elsif ${pathid}_tmpl {{\n $str_{pathid} = template('{tmplmod}/{pathid}', ${pathid}_tmpl)\n }} else {{\n $str_{pathid} = template('{tmplmod}/{pathid}') }}\n\n file {{ '{pathid}':\n path => $final_{pathid}_path,\n ensure => file,\n owner => '{user}',\n group => '{user}',\n mode => '{mode}',\n backup => '.puppet-bak',\n content => $str_{pathid}; }}\n }}"
tmpl_param = ' ${pathid} = undef,\n ${pathid}_path = undef,\n ${pathid}_tmpl = undef'
tmpl_tmplvar = ' ${pathid}_{tmplvar} = undef'
class minimin.fabqnd.Install(*args)[source]

Bases: minimin.fabqnd.Manifest

add(_id=None, name=None, provider=None, source=None)[source]

Add parameters to the ‘install’ class template.

Parameters :
  • _id: common ID for the package across platforms
  • name: platform-specific package name
  • provider: optional package provider name

Example INI option and value:

ntp.install.redhat  = ntp:ntp:yum
ntp.install.solaris = ntp:CSWntp:pkgutil

The class declaration also accepts a {pkgid}_source parameter that specifies the source for the pkgid as understood by the provider:

class {'qnd::ntp::install': $ntp_source => 'http://filesvr/ntp.rpm'} 
tmpl_class = 'class {mod}::{sub}::install (\n{params}\n)\n{{\n include {mod}::{sub}::default\n{packages}\n}}'
tmpl_package = "\n $pkg_{pkgid} = ${mod}::{sub}::default::pkg_{pkgid}\n $pkg_{pkgid}_provider = ${mod}::{sub}::default::pkg_{pkgid}_provider\n\n if $pkg_{pkgid} {{\n\n if ${pkgid}_provider {{\n $final_{pkgid}_provider = ${pkgid}_provider\n }} elsif $pkg_{pkgid}_provider {{\n $final_{pkgid}_provider = $pkg_{pkgid}_provider\n }} else {{\n $final_{pkgid}_provider = undef }}\n\n package {{ '{pkgid}':\n name => $pkg_{pkgid},\n ensure => ${pkgid}_state,\n source => ${pkgid}_source,\n provider => $final_{pkgid}_provider }}\n }}"
tmpl_param = " ${pkgid}_state = 'present',\n ${pkgid}_source = undef,\n ${pkgid}_provider = undef"
class minimin.fabqnd.Manifest(mod, sub, manifest, osname)[source]

Bases: object

append(seq, tmpl, **kw)[source]

Append a rendered template to an internal sequence.

init_class_macros(*args)[source]

Create lists for macros required by the class template.

render()[source]

Render the completed class string.

render_default()[source]
set_default(var, val)[source]

Append (osname, value) to funky variable key in defaults dictionary.

tmpl_val = ' {osname} => {val},'
tmpl_var = ' ${var} = $operatingsystem ? {{\n{vals}\n }}'
class minimin.fabqnd.Service(*args)[source]

Bases: minimin.fabqnd.Manifest

add(_id=None, name=None)[source]

Add parameters to the ‘service’ class template.

Parameters :
  • _id: common ID for the service across platforms
  • name: platform-specific service name

Example INI option and value:

ntp.service.default = ntp:ntpd

The class declaration accepts a {name}_subs parameter that should specify one or more configuration files by _id to subscribe to:

class {'qnd::ntp::service': $ntp_subs => File['ntp_conf']} 
tmpl_class = 'class {mod}::{sub}::service (\n{params}\n)\n{{\n include {mod}::{sub}::default\n{services}\n}}'
tmpl_param = " ${svcid}_state = 'running',\n ${svcid}_subs = undef"
tmpl_service = "\n $svc_{svcid} = ${mod}::{sub}::default::svc_{svcid}\n\n if $svc_{svcid} {{\n\n service {{ '{svcid}':\n name => $svc_{svcid},\n ensure => ${svcid}_state,\n subscribe => ${svcid}_subs }}\n }}"
minimin.fabqnd.esc_print(s)[source]

Print a line, escape dollar characters.

minimin.fabqnd.inigen(cfg)[source]

Use an .INI config to generate information for subgen().

Parameters :
  • cfg: path to .INI config file (must include [qnd] section with the appropriate options and values)
Yields :

(sub, manifest, osname), val for each option and value

minimin.fabqnd.join(iterable) → string

Return a string which is the concatenation of the strings in the iterable. The separator between elements is S.

minimin.fabqnd.qndgen(modhome)[source]

Create a quick-n-dirty script to generate submodules.

Parameters :
  • modhome: path to module directory (omit trailing separator)
Example:
fab –hide=running,status qndgen:/path/to/mymod > mksubs.sh
minimin.fabqnd.subgen(mod)[source]

Yield Puppet code sor submodule classes.

Parameters :
  • mod: main module name (submodules will be created as mod::sub)
Yields :

tuple of strings (sub, manifest, class) for all submodules