6. Implementation details

6.1. Minimin’s INI configuration

XML is too verbose and persnickety, JSON and YAML have an overhead that might frustrate Notepad users, and nested formats aren’t very friendly to line parsers.

The main trade-off with the INI file is that a lack of formal nesting requires us to fudge nests (as dotted option names) in the interest of consolidating related options (commands, installations, roles). This adds some redundancy in option prefixes, but this also makes it easier to grep the options. Lack of indentation and nesting makes it easier to copy and paste option blocks.

6.2. Fabric

6.2.1. Keyword restrictions

Fabric (specifically, fabric/main.py:516) prevents keywords host, hosts, role, roles, exclude_hosts from being passed to their intended functions. This is why we’re using node as a keyword all over the place.

6.2.2. Localhost output

I didn’t find an obvious or nice way to control output per function/task via configuration, decoration, or context managers. Specifically:

  • running messages precede function/task calls (preempting decorated or context-managed output),
  • the status Done. follows everything (by default),
  • with_settings and settings effects get rolled back after a function is finished (missing the final status message),
  • avoiding the restrictions of decorators and context managers with fabric.state.output would require every function to be output-aware or risk quietly overriding the intended output of those that weren’t.

An output-to-file option might be a good idea.

The end result is that functions intended to provide clean output just document the need to --hide undesired messages (see fab -d {func}):

fab --hide=running,status {func}

6.2.3. admin_ssh_home and env.ssh_config_path

admin_ssh_home is only here to support sshkeygen(), so that we know where to put our keys. We don’t want to assume that the home of our env.ssh_config_path is always where we store our keys.

6.2.4. SSH errors

  • EOF during negotiation is probably the result of a bad Subsystem sftp configuration in the server’s sshd_config. This is why sshput (for single files, no recursion) exists as a pure-SSH alternative to file copying.
  • Too many authentication failures (check your server logs) is probably the result of too many attempts to use invalid SSH keys. We might mitigate by setting IdentitiesOnly yes in our client ssh_config, but it looks like Fabric loops over our keys in fabric/network.py:280. A (practical) workaround is to limit the number of public keys used in the ssh_config. Help troubleshoot Paramiko/SSH connections in Fabric using ssh.util.log_to_file('logfile', 10).
  • KeyError: 'user@host:port' from fabric/network.py might be due to a misconfigured ssh_config. Specifically, I had a trailing space after a Port number in ssh_config. Running a test ssh host -- uname -a worked, but the trailing space resulted in a KeyError when running fab -Hhost -- uname -a.

6.3. Puppet

6.3.1. Why, why not ..?

Puppet feels like the best choice for system configuration because of its idempotency, built-in capabilities, number of supported platforms, online resources, and option for commercial support. Features of Puppet that weren’t attractive were easily avoided: SSL client-server setup may be replaced by SSH updates and native puppet apply calls and file serving may be handled by bundling configs as templates.

Some alternatives:

  • Chef, Kokki and cdist lean toward “programming as configuration” by exposing either more of the underlying programming language (Chef/Ruby, Kokki/Python) or the framework itself (Cdist).
  • Synctool is easy to get started with, but felt better suited for homogeneous environments.
  • Ansible and Salt are attractive alternatives. Both use a simpler INI or YAML syntax for most configuration. Ansible leverages SSH and Salt implements it’s own high-performance communication using ZeroMQ.

6.3.2. Modules everywhere

The quick-n-dirty implementation uses modules for everything (no /etc/puppet/puppet.conf or /etc/puppet/manifests) to reduce the number of locations we have to think about and the amount of bootstrapping we have to do. We copy all the modules we need to a common directory, and provide the module path and site manifest to puppet apply.

6.3.3. Preferring templates

The quick-n-dirty implementation assumes that configuration files are managed in templates/, regardless of whether or not the file or class (config.pp) uses configurable parameters because:

  1. templates/ and modules can provide automagic path resolution which works well for our push-based, copy/apply approach, and
  2. between templates/ and files/, a ‘template’ implies something that is dynamic will have its contents modified.

6.3.4. Role dependency

Fabric’s roles are named lists of hosts, and host specification via command line, configuration, or decoration is generally additive. I wanted group attributes (like required Puppet modules) bundled with the host list to consolidate group information.

6.3.5. Dirty boilerplate

The quick-n-dirty submodules have a lot of boilerplate due to the if-then logic that determines whether to use operating system variables defined in default.pp or class parameter values.

6.4. Manual installs

We avoid Puppet for manual installs (those which don’t use a package manager) because:

  • it’s difficult to parse manifests for external purposes (this typically requires fetching the results from node), and
  • including the installation recipes gains us uniformity only if we can assume Puppet itself is trivial to install (not likely on non-Linux nodes).

Since we must support manual, non-Puppet installs anyway (if only to bootstrap Puppet), we mimic idempotence using the commands we’d have to provide Puppet’s exec helpers unless, onlyif and refresh. The trade-off is between the convenience of using Puppet as a one stop shop for installation and configuration and the flexibility of INI-style syntax.

If possible, installations and updates should be configured in Puppet using a package manager (yum, apt, etc.).

6.4.1. Version comparisons

Sometimes you might want to compare the version of some required program, perhaps in an installer’s req option. GNU’s awk (gawk) has a match(string, regex, array) function, but the presence of perl is probably a safer bet:

[ $(java -version 2>&1 | perl -n -e '/"(.*)_.*"/ && print $1') \> 1.6 ]

Note that Bash’s string comparison does not support “greater than or equal to.” Bash supports greater than, less than, equal or not equal. In this case we’re taking advantage of the fact that a java -version has three components, and that 1.7.0 is greater than 1.7.

6.5. Things TODO


Figure out how the fabmin functions exists, support, and install should work together. exists functionality is duplicated in the other two, and the support directory in the [role] and [install] configuration sections is (or should be?) the same. For the time being, install and support are treated as separate concerns that just so happen to work with each other if they share the same support directory. The result is that after the initial SSH access is set up and Puppet is installed, we should run fab support, config, and manual install commands in that order.


It might be nice to allow roles to determine their list of nodes using other roles’ node lists.


Add command sec.setuid to look for setuid programs.


Add command sec.mount to check mounted filesystem options.


Add pointers in source to links/references in docs so we can see where we (supposedly) implemented a feature.


Note reasons for different error output - fabric.utils.error (bound to group of everything that can be dismissed with warn_only, log.warn (hints that you’re straying from the path of righteousness), and built-in exceptions (I wouldn’t work no matter what) vs. fabric.utils.abort (which should probably be used more than the exceptions).


Add dependencies and order to manual installs, and add a list of required installs per role: role_name.install = pkg1 pkg2.


Logging support.


A fallback mechanism for suffixes (so we don’t have to repeat for .f16 what would still work from .f15). Maybe a format for suffixes? Dashes might help but look ugly. How many alpha-only IDs do we need? f-Fedora, r-Red Hat, d-Debian, s-Solaris, o-OpenIndiana, h-HPUX, s-Slackware.. uh oh.


More documentation support and support for Doxygen specifically.


TaskJuggler support.


Pitz support. A project’s repository should probably include its technical issues, internal todo list, and “lessons learned” (a project should not include the full history of public issues and discussions). If issue management isn’t within the comfort zone of normal documentation, then a text-based tracker might be helpful.


Uniform output from [exe] commands to make it easier to use programatically, in an interactive session, or as input to a graphing utility.


Ansible looks like it might be an alternative to Fabric and Puppet (and it uses a simpler syntax by default).