2012-04-11 by sourpoi in misc, tagged: fabric ssh

Short version: Don't use a lot of SSH keys.

For Fabric-1.4.1 using ssh (library) 1.7.13, my issue was that if I..

  • use an SSH agent (Fedora 15 and gnome-keyring), and
  • configure Fabric to use an SSH config (~/.ssh/config), and
  • configure multiple hosts by setting env.hosts prior to calling a task (function) in fabfile.py, and
  • call that task on a single host (-H $host)

..then I'm prompted (via Gnome's keyring pop-up) for passphrases to unlock all host keys in env.hosts rather than just the host provided by -H $host. To recreate my journey..

Change the default logging level in fabfile.py so that the ssh module will print debug logs:

import logging
log = logging.getLogger()
log.setLevel(logging.DEBUG)

..or set the ssh module's log level directly:

ssh.util.log_to_file('logfile', 10)

Compare the new debug log messages and available keys (ssh-add -l | tr -d ':'). The DEBUG:ssh.transport messages confirm attempts to use SSH agent keys that don't match the target host.

Edit fabfile.py to drop into a debugger:

import ipdb; ipdb.set_trace()

Run fab and step through the process until Fedora starts prompting for passphrases. Note the location:

fab -H $host -- date

ipdb> b ssh/packet.py:241
ipdb> c
1-> 241                 n = self.__socket.send(out)

Determine the current location in the stack:

ipdb> w

The following is the relevant part of the stack information:

> grep site-packages /tmp/fab-stack.txt | cut -d'/' -f8-

site-packages/fabric/main.py(712)main()
site-packages/fabric/tasks.py(298)execute()
site-packages/fabric/tasks.py(197)_execute()
site-packages/fabric/tasks.py(112)run()
site-packages/fabric/main.py(694)<lambda>()
site-packages/fabric/network.py(457)host_prompting_wrapper()
site-packages/fabric/operations.py(905)run()
site-packages/fabric/operations.py(815)_run_command()
site-packages/fabric/state.py(340)default_channel()
site-packages/fabric/network.py(84)__getitem__()
site-packages/fabric/network.py(76)connect()
site-packages/fabric/network.py(292)connect()
site-packages/ssh/client.py(332)connect()
site-packages/ssh/client.py(450)_auth()
site-packages/ssh/transport.py(1246)auth_publickey()
site-packages/ssh/auth_handler.py(82)auth_publickey()
site-packages/ssh/auth_handler.py(124)_request_auth()
site-packages/ssh/transport.py(1422)_send_message()
site-packages/ssh/packet.py(310)send_message()
site-packages/ssh/packet.py(241)write_all()

Somewhere up that stack is a loop over our hosts. After a little bit of a hunt, the culprit was found in fabric/network.py on line 280.

Is this a bug? I'm not sure.


Epilogue: I was using many (>30) SSH keys for local development, creating and managing them with scripts on a per-host basis between three user accounts. Many temporary development hosts meant many keys. This approach worked with loops over shell-scripted ssh calls but resulted in the issue above when I started consolidating the administrative user accounts and porting those scripts to Fabric. After searching for reassurance that my original approach was acceptable, I was left wanting. In the meantime, the path of least resistance has led me to a single, clean ~/.ssh/ home after reducing my SSH arsenal to about four keys. In theory I'd still like Fabric to stop looping over the keys. In practice I don't care any more.

Comments