SSH key loop in Fabric
Short version: Don't use a lot of SSH keys.
For Fabric-1.4.1 using ssh (library) 1.7.13, my issue was that if I..
- use an SSH agent (Fedora 15 and gnome-keyring), and
- configure Fabric to use an SSH config (~/.ssh/config), and
- configure multiple hosts by setting env.hosts prior to calling a task (function) in fabfile.py, and
- call that task on a single host (-H $host)
..then I'm prompted (via Gnome's keyring pop-up) for passphrases to unlock all host keys in env.hosts rather than just the host provided by -H $host. To recreate my journey..
Change the default logging level in fabfile.py so that the ssh module will print debug logs:
import logging log = logging.getLogger() log.setLevel(logging.DEBUG)
..or set the ssh module's log level directly:
Compare the new debug log messages and available keys (ssh-add -l | tr -d ':'). The DEBUG:ssh.transport messages confirm attempts to use SSH agent keys that don't match the target host.
Edit fabfile.py to drop into a debugger:
import ipdb; ipdb.set_trace()
Run fab and step through the process until Fedora starts prompting for passphrases. Note the location:
fab -H $host -- date ipdb> b ssh/packet.py:241 ipdb> c 1-> 241 n = self.__socket.send(out)
Determine the current location in the stack:
The following is the relevant part of the stack information:
> grep site-packages /tmp/fab-stack.txt | cut -d'/' -f8- site-packages/fabric/main.py(712)main() site-packages/fabric/tasks.py(298)execute() site-packages/fabric/tasks.py(197)_execute() site-packages/fabric/tasks.py(112)run() site-packages/fabric/main.py(694)<lambda>() site-packages/fabric/network.py(457)host_prompting_wrapper() site-packages/fabric/operations.py(905)run() site-packages/fabric/operations.py(815)_run_command() site-packages/fabric/state.py(340)default_channel() site-packages/fabric/network.py(84)__getitem__() site-packages/fabric/network.py(76)connect() site-packages/fabric/network.py(292)connect() site-packages/ssh/client.py(332)connect() site-packages/ssh/client.py(450)_auth() site-packages/ssh/transport.py(1246)auth_publickey() site-packages/ssh/auth_handler.py(82)auth_publickey() site-packages/ssh/auth_handler.py(124)_request_auth() site-packages/ssh/transport.py(1422)_send_message() site-packages/ssh/packet.py(310)send_message() site-packages/ssh/packet.py(241)write_all()
Somewhere up that stack is a loop over our hosts. After a little bit of a hunt, the culprit was found in fabric/network.py on line 280.
Is this a bug? I'm not sure.
Epilogue: I was using many (>30) SSH keys for local development, creating and managing them with scripts on a per-host basis between three user accounts. Many temporary development hosts meant many keys. This approach worked with loops over shell-scripted ssh calls but resulted in the issue above when I started consolidating the administrative user accounts and porting those scripts to Fabric. After searching for reassurance that my original approach was acceptable, I was left wanting. In the meantime, the path of least resistance has led me to a single, clean ~/.ssh/ home after reducing my SSH arsenal to about four keys. In theory I'd still like Fabric to stop looping over the keys. In practice I don't care any more.