1. Welcome

Minimin is a handful of administrative conventions expressed in INI-style configuration files and implemented using Fabric. Its goal is to help keep administration light, modular, and easy to reason about. Fabric executes remote tasks via SSH, and general system configuration is managed by Puppet.

Minimin is not a configuration management system - it is a way to think about, and ease the pain of, incorporating management components into an existing environment.

1.1. Audience

If you:

  • juggle personal and professional projects in exclusive environments and want to reduce the expense of context switching,
  • find your occasional-but-valuable commands getting lost among personal notes, project documentation, todo-lists and bookmarks,
  • wish it were easier to share your workflow (or as much of it as you’re willing) with others,
  • use (or want to make more use of) SSH as the default pipeline to remote servers but aren’t motivated to manage keys or an ssh_config,
  • don’t want to mess with Puppet’s SSL client-server “pull-based” system, or don’t have a happy recipe to bootstrap modules..

..then you may find something helpful here.

1.2. Admin environment

Three conceptual domains comprise an administrator’s environment:

  1. user: Everything specific to an administrator’s user account; files that needn’t or shouldn’t be shared with team members or the public. For example, exactly where an admin keeps off-line project repositories probably isn’t important to anyone else and private SSH keys should definitely be kept private.
  2. project: Specific sets of responsibilities, methods, and tasks define projects. Project information is shared between team members and may be available to others for the purposes of monitoring progress, making backups, or education. Projects should use some form of (preferably distributed) version control.
  3. support: Files that are shared by several projects or available publicly constitute support. Support includes everything in this module, additional pre-configured commands that you share with friends, third-party software packages, libraries, etc.. Support files should be easily accessible, preferably in a local cache.

1.3. Layout example

In the following environment, an admin user keeps private files in /home/admin, two project repositories in /proj/websvc and /proj/common, and third party support in /support. Project repositories contain Minimin configurations in their respective etc/*.ini files, and Puppet modules are found in the common project and as part of the support from a vendor. Support packages fall into one of three categories: sources, RPMs, and Solaris pacakges. All these locations are configured in /home/admin/.fabricrc:

/
+-- home
|   +-- admin
|       +-- .fabricrc                      (1)
|       +-- .ssh                           (6)
|           +-- config                     (7)
|           +-- id_admin_root_websvr
|           +-- id_admin_root_websvr.pub
+-- proj
|   +-- websvc
|   |   +-- etc                            (2)
|   |       +-- site.ini                   (3)
|   +-- common
|       +-- etc                            (2)
|       |   +-- share.ini                  (3)
|       +-- puppet
|           +-- modules                    (4)
|               +-- ...
+-- support
    +-- source                             (5)
    |   +-- tmux-1.6.tar.gz
    +-- rpm                                (5)
    |   +-- zeromq-2.1.9-1.fc15.x86_64.rpm
    +-- pkg                                (5)
    |   +-- nc-110-sol10-x86-local.gz
    +-- vendor
        +-- puppet
            +-- modules                    (4)
                +-- ...

The admin’s Fabric config (1, /home/admin/.fabricrc) tells us where to find administrative components in our directory tree. It uses option = value entries to define:

  • Minimin config file inclusion paths (2):

    admin_cfg_path = /proj/websvc/etc /proj/common/etc
  • Minimin config files to include (3):

    admin_cfg_include = site.ini share.ini
  • Puppet module homes (4):

    admin_puppet_path = /proj/common/puppet/modules \
                        /support/vendor/puppet/modules
  • paths to find software packages (5):

    admin_support_path = /support/source /support/rpm /support/pkg
  • a home for our SSH keys (6):

    admin_ssh_home = /home/admin/.ssh
  • and an SSH host configuration (7):

    use_ssh_config = True
    ssh_config_path = /home/admin/.ssh/config

1.4. INI configs

The Minimin configuration - the result of all the *.ini files above - tells us how to apply these components to our target nodes (hosts). Our node names map to an OS/platform identifier to determine the relevancy of configured commands (in the [exe] section) and “manual” installs (in the [install] section). Roles group nodes for the convenience of executing ad-hoc commands in bulk and applying a common Puppet configuration:

[node]
websvr = f15
mailsvr = s10

[role]
common.nodes = websvr mailsvr
common.puppet.mods = master qnd
common.puppet.modpath = /etc/puppet/modules
common.puppet.site = /etc/puppet/modules/master/manifests/site.pp

web.nodes = websvr

mail.nodes = mailsvr

[exe]
pkg-grep.txt = look for a package name matching {str}
pkg-grep.f15 = rpm -qa | grep {str}
pkg-grep.s10 = pkginfo | grep -i {str}

[install]
url-sfw10 = ftp://ftp.sunfreeware.com/pub/freeware/intel/10

nc.txt.s10 = Netcat, the all-purpose network tool SMCnc
nc.url.s10 = %(url-sfw10)s/nc-110-sol10-x86-local.gz
nc.src.s10 = nc-110-sol10-x86-local.gz
nc.dst.s10 = /tmp/nc-110-sol10-x86-local.gz
nc.pth.s10 = /usr/sbin:/usr/bin
nc.req.s10 = which pkgadd pkgrm gunzip
nc.chk.s10 = /usr/local/bin/nc -h 2>&1 | grep 'v1.10'
nc.del.s10 = pkgrm SMCnc
nc.exe.s10 = #!/bin/sh
    cd /tmp && \
    gunzip nc-110-sol10-x86-local.gz && \
    pkgadd -d nc-110-sol10-x86-local all

Two nodes, websvr and mailsvr, run Fedora 15 (f15) and Solaris 10 (s10), respectively. The platform identifiers are arbitrary strings.

The common role includes both nodes. The common role requires two Puppet modules, master and qnd, both of which should be found in admin_puppet_path. A configuration update copies these modules to each node’s /etc/puppet/modules directory and applies the /etc/puppet/modules/master/manifests/site.pp manifest on each node. Updates can be applied toward the group of both nodes via the common role or to a single node via the web or mail role.

One command, pkg-grep, conveniently wraps native methods to find installed packages.

Finally, a “manual” installation of Netcat is configured for Solaris 10. nc-110-sol10-x86-local.gz should be located in admin_support_path. An attempt to install the pacakge will first run a check for existence (nc.chk.s10). If it fails, the package will be copied to the destination (nc.dst.s10) and the installation script (nc.exe.s10) will be run using a specific path (nc.pth.s10).

1.5. Motivation

What we’re trying to achieve with the layout and configuration above:

  • Ease of adoption: We want team members to understand and be able to contribute to the site configuration as quickly as possible.

    The simple INI format reduces the potential for syntax errors, is friendly to line parsers, and lends itself well to visual pattern recognition, copied-and-pasted chunks, and manual editing in general.

    The tight-coupling between config files makes it easy to make sense of (and troubleshoot) the admin environment. “Where are my config files?” Look in .fabricrc. “Why can’t I contact a node?” Start with ssh_config_path. “Where is that installer?” Double check your admin_support_path. “How do I run that command?” Consult the [exe] section.

  • Minimal overhead: We want to install as little as possible and write as little as possible to accomplish as much as possible.

    Fabric (which includes SSH support) alone will get you started with ad-hoc tasks and Puppet will handle general system configuration. By evolving your custom Puppet modules and contributing to your commands in [exe], these tools may be all you need. [1]

    In terms of layout, separating a user’s environment (SSH keys) from shared projects (site-specific, group-accessible files) from publicly available third party packages should be considered minimal, but required, overhead.

  • Domain-less configuration: We want to be able to take our configurations, or portions thereof, with us. We want to share them between sites. We don’t want them locked in a binary format in a running service. We don’t want to define our systems in terms of screenshots.

    Puppet modules are one way of sharing reusable system configuration components, and layered INI files make it easy to splice and dice configuration sections and options. Admins can phase components in our out of an environment by editing the appropriate options in their .fabricrc (ex: admin_cfg_path and admin_cfg_include) or at runtime using fab --set ....

  • Encouragement: Finally, we want to make it easy to “do the right things” by reducing the startup expense for each administrative component. And the layout itself should be leveraged for convenience to encourage enthusiasm in it’s upkeep.

[1]The admin’s environment is light - both conceptually and in terms of required packages. Administered nodes can be laden with any and all software available to native package managers or found in the admin_support_path.

1.6. Use cases

“OK. What really motivated this?” In a nutshell: the desire to free administration from the domain.

1.6.1. Reading down, writing up

Barriers between networks at different security levels can lead to unncessarily divergent methods of administration despite their similarities; no matter where you’re at, you’re likely to manage DNS, NTP, and SNMP, and you’re likely to wonder what processes are resource hogs, and how much space is available.

1.6.2. VPNs and LANs

Unplug here, go there, same head honcho.