FAQ:
Running jobs under rsh/ssh

Table of contents:

1. Can I make Open MPI use rsh instead of ssh?
2. What prerequisites are necessary for running an Open MPI job under rsh/ssh?
3. How can I make ssh not ask me for a password?
4. What is a .rhosts file? Do I need it?
5. Should I use + in my .rhosts file?

Yes. The method to do this has changed over the different versions of Open MPI.

1. v1.7 and later series: The plm_rsh_agent MCA parameter accepts a colon-delimited list of programs to search for in your path to use as the remote startup agent. The default value is ssh : rsh, meaning that it will look for ssh first, and if it doesn't find it, use rsh. You can change the value of this parameter as relevant to your environment, such as simply changing it to rsh or rsh : ssh if you have a mixture. The deprecated forms pls_rsh_agent and orte_rsh_agent will also work.
2. v1.3 to v1.6 series: The orte_rsh_agent MCA parameter accepts a colon-delimited list of programs to search for in your path to use as the remote startup agent (the MCA parameter name plm_rsh_agent also works, but it is deprecated). The default value is ssh : rsh, meaning that it will look for ssh first, and if it doesn't find it, use rsh. You can change the value of this parameter as relevant to your environment, such as simply changing it to rsh or rsh : ssh if you have a mixture.
3. v1.1 and v1.2 series: The v1.1 and v1.2 method is exactly the same as the v1.3 method, but the MCA parameter name is slightly different: pls_rsh_agent ("pls" vs. "plm"). Using the old "pls" name will continue to work in the v1.3 series, but it is now officially deprecated — you'll receive a warning if you use it.
4. v1.0 series: In the 1.0.x series, Open MPI defaults to using ssh for remote startup of processes in unscheduled environments. You can change this to rsh by setting the MCA parameter pls_rsh_agent to rsh.

See this FAQ entry for details on how to set MCA parameters — particularly with multi-word values.

In general, they are the same for running Open MPI jobs in other environments (see this FAQ category for more general information).

If you are using ssh to launch processes on remote nodes, there are multiple ways.

Note that there are multiple versions of ssh available. References to ssh in this text refer to OpenSSH.

This documentation provides an overview for using user keys and the OpenSSH 2.x key management agent (if your OpenSSH only supports 1.x key management, you should upgrade). See the OpenSSH documentation for more details and a more thorough description. The process is essentially the same for other versions of SSH, but the command names and filenames may be slightly different. Consult your SSH documentation for more details.

Normally, when you use ssh to connect to a remote host, it will prompt you for your password. However, for the easiest way for mpirun (and mpiexec, which, in Open MPI, is identical to mpirun) to work properly, you need to be able to execute jobs on remote nodes without typing in a password. In order to do this, you will need to set up a passphrase. We recommend using RSA passphrases as they are generally "better" (i.e., more secure) than DSA passphrases. As such, this text will describe the process for RSA setup.

NOTE: This text will briefly show you the steps involved in doing this, but the ssh documentation is authorative on these matters should be consulted for more information.

The first thing that you need to do is generate an RSA key pair to use with ssh-keygen:

1

shell$ ssh-keygen -t rsa

Accept the default value for the file in which to store the key ($HOME/.ssh/id_rsa) and enter a passphrase for your key pair. You may choose to not enter a passphrase and therefore obviate the need for using the ssh-agent. However, this greatly weakens the authentication that is possible, because your secret key is potentially vulnerable to compromise because it is unencrypted. It has been compared to the moral equivalent of leaving a plain text copy of your password in your $HOME directory. See the ssh documentation for more details.

Next, copy the $HOME/.ssh/id_rsa.pub file generated by ssh-keygen to $HOME/.ssh/authorized_keys (or add it to the end of authorized_keys if that file already exists):

1
2

shell$ cd $HOME/.ssh
shell$ cp id_rsa.pub authorized_keys

In order for RSA authentication to work, you need to have the $HOME/.ssh directory in your home directory on all the machines you are running Open MPI on. If your home directory is on a common filesystem, this may be already taken care of. If not, you will need to copy the $HOME/.ssh directory to your home directory on all Open MPI nodes. (Be sure to do this in a secure manner — perhaps using the scp command — particularly if your secret key is not encrypted.)

ssh is very particular about file permissions. Ensure that your home directory on all your machines is set to at least mode 755, your $HOME/.ssh directory is also set to at least mode 755, and that the following files inside $HOME/.ssh have at least the following permissions:

1
2
3
4

-rw-r--r--  authorized_keys
-rw-------  id_rsa
-rw-r--r--  id_rsa.pub
-rw-r--r--  known_hosts

The phrase "at least" in the above paragraph means the following:

• The files need to be readable by you.
• The files should only be writable by you.
• The files should not be executable.
• Aside from id_rsa, the files can be readable by others, but do not need to be.
• Your $HOME and $HOME/.ssh directories can be readable by others, but do not need to be.

You are now set up to use RSA authentication. However, when you ssh to a remote host, you will still be asked for your RSA passphrase (as opposed to your normal password). This is where the ssh-agent program comes in. It allows you to type in your RSA passphrase once, and then have all successive invocations of ssh automatically authenticate you against the remote host. See the ssh-agent(1) documentation for more details than what are provided here.

Additionally, check the documentation and setup of your local environment; ssh-agent may already be setup for you (e.g., see if the shell environment variable $SSH_AUTH_SOCK exists; if so, ssh-agent is likely already running). If ssh-agent is not already running, you can start it manually with the following:

1

shell$ eval `ssh-agent`

Note the specific invocation method: ssh-agent outputs some shell commands to its output (e.g., setting the SSH_AUTH_SOCK environment variable).

You will probably want to start the ssh-agent before you start your graphics / windowing system so that all your windows will inherit the environment variables set by this command. Note that some sites invoke ssh-agent for each user upon login automatically; be sure to check and see if there is an ssh-agent running for you already.

Once the ssh-agent is running, you can tell it your passphrase by running the ssh-add command:

1

shell$ ssh-add $HOME/.ssh/id_rsa

At this point, if you ssh to a remote host that has the same $HOME/.ssh directory as your local one, you should not be prompted for a password or passphrase. If you are, a common problem is that the permissions in your $HOME/.ssh directory are not as they should be.

Note that this text has covered the ssh commands in _very little detail._ Please consult the ssh documentation for more information.

If you are using rsh to launch processes on remote nodes, you will probably need to have a $HOME/.rhosts file.

This file allows you to execute commands on remote nodes without being prompted for a password. The permissions on this file usually must be 0644 (rw-r--r--). It must exist in your home directory on every node that you plan to use Open MPI with.

Each line in the .rhosts file indicates a machine and user that programs may be launched from. For example, if the user steve wishes to launch programs from the machine stevemachine to the machines alpha, beta, and gamma, there must be a .rhosts file on each of the three remote machines (alpha, beta, and gamma) with at least the following line in it:

1

stevemachine steve

The first field indicates the name of the machine where jobs may originate from; the second field indicates the user ID who may originate jobs from that machine. It is better to supply a fully-qualified domain name for the machine name (for security reasons — there may be many machines named stevemachine on the internet). So the above example should be:

1

stevemachine.example.com steve

If rsh does not find a matching line in the $HOME/.rhosts file, it will prompt you for a password. Open MPI requires the password-less execution of commands; if rsh prompts for a password, mpirun will fail.

NOTE: Some implementations of rsh are very picky about the format of text in the .rhosts file. In particular, some do not allow leading white space on each line in the .rhosts file, and will give a misleading "permission denied" error if you have white space before the machine name.

NOTE: It should be noted that rsh is not considered "secure" or "safe" — .rhosts authentication is considered fairly weak. The Open MPI Team recommends that you use ssh ("Secure Shell") to launch remote programs as it uses a much stronger authentication system.

No!

While there are a very small number of cases where using "+" in your .rhosts file may be acceptable, the Open MPI Team highly recommends that you do not.

Using a "+" in your .rhosts file indicates that you will allow any machine and/or any user to connect as you. This is extremely dangerous, especially on machines that are connected to the internet. Consider the fact that anyone on the internet can connect to your machine (as you) — it should strike fear into your heart.

Instead, you should use the full and proper hostname and username of accounts that are authorized to remotely login as you to that machine (or machines). This is usually just a list of your own username on a list of machines that you wish to run Open MPI with. See this FAQ entry for further details, as well as your local rsh documentation.

Additionally, the Open MPI Team strongly recommends that rsh is not used in unscheduled environments (espectially those connected to the internet) — it is considered weak remote authentication. Instead, we recommend the use of ssh — the secure remote shell. See this FAQ entry for more details.