3. Installation

3.1. Requirements

RADICAL-Pilot requires the following packages:

  • Python >= 2.7 (including development headers)
  • virtualenv >= 1.11
  • pip == 1.4.1

or * Anaconda Python 2.7

If you plan to use RADICAL-Pilot on remote machines, you would also require to setup a password-less ssh login to the particular machine. (help)

All dependencies are installed automatically by the installer. Besides that, RADICAL-Pilot needs access to a MongoDB database that is reachable from the internet. User groups within the same institution or project usually share a single MongoDB instance. MongoDB is standard software and available in most Linux distributions. At the end of this section, we provide brief instructions how to set up a MongoDB server and discuss some advanced topics, like SSL support and authentication to increased the security of RADICAL-Pilot.

3.2. Installation

RADICAL-Pilot is distributed via PyPi and Conda-Forge. To install RADICAL-Pilot to a virtual environment do:

3.2.1. via PyPi

virtualenv --system-site-packages $HOME/ve
source $HOME/ve/bin/activate
pip install radical.pilot

3.2.2. via Conda-Forge

conda create -n ve -y python=2.7
source activate ve
conda install radical.pilot -c conda-forge

For a quick sanity check, to make sure that the the packages have been installed properly, run:

$ radicalpilot-version
0.50.8

The exact output will obviously depend on the exact version of RP which got installed.

** Installation is complete !**

3.3. Preparing the Environment

3.3.1. MongoDB Service

RP requires access to a MongoDB server. The MongoDB server is used to store and retrieve operational data during the execution of an application using RADICAL-Pilot. The MongoDB server must be reachable from both, the host that runs the RP application and the target resource which runs the pilots.

Warning

If you want to run your application on your laptop or private workstation, but run your MD tasks on a remote HPC cluster, installing MongoDB on your laptop or workstation won’t work. Your laptop or workstations usually does not have a public IP address and is hidden behind a masked and firewalled home or office network. This means that the components running on the HPC cluster will not be able to access the MongoDB server.

Any MongoDB installation should work out, as long as RP is allowed to create databases and collections (which is the default user setting in MongoDB).

The MongoDB location is communicated to RP via the environment variable RADICAL_PILOT_DBURL. The value will have the form

export RADICAL_PILOT_DBURL="mongodb://user:pass@host:port/dbname"
export RADICAL_PILOT_DBURL="mongodb://host:port/dbname"

Many MongoDB instances are by default unsecured, and thus do not require the user:pass@ part of the URL. For production runs, and for runs on large secured resources, we strongly recommend the usage of a secured MongoDB instance!

The dbname component in the database url can be any valid MongoDB database name (ie. it musy not contain dots).RP will not create that DB on the fly and requires the DB to be setup prior to creating the session object. But RP will create collections in that DB on its own, named after RP session IDs.

A MongoDB server can support more than one user. In an environment where multiple users use RP applications, a single MongoDB server for all users / hosts is usually sufficient. We recommend the use of separate databases per user though, so please set the dbname to something like db_joe_doe.

Install your own MongoDB

Once you have identified a host that can serve as the new home for MongoDB, installation is straight forward. You can either install the MongoDB server package that is provided by most Linux distributions, or follow the installation instructions on the MongoDB website:

MongoDB-as-a-Service

There are multiple commercial providers of hosted MongoDB services, some of them offering free usage tiers. We have had some good experience with the following:

3.3.2. Setup SSH Access to Target Resources

An easy way to setup SSH Access to multiple remote machines is to create a file ~/.ssh/config. Suppose the url used to access a specific machine is foo@machine.example.com. You can create an entry in this config file as follows:

# contents of $HOME/.ssh/config
Host mach1
    HostName machine.example.com
    User foo

Now you can login to the machine by using ssh mach1. Please make also sure that your ssh keys are registered on the target resources – while RP can in principle handle password based login, the repeated prompts for passwords makes RP applications very difficult to use.

Source: http://nerderati.com/2011/03/17/simplify-your-life-with-an-ssh-config-file/

3.4. Troubleshooting

Missing virtualenv

This should return the version of the RADICAL-Pilot installation, e.g., 0.X.Y.

If virtualenv is not installed on your system, you can try the following.

wget --no-check-certificate https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz
tar xzf virtualenv-1.9.tar.gz

python virtualenv-1.9/virtualenv.py $HOME/ve
source $HOME/ve/bin/activate

Installation Problems

Many installation problems boil down to one of two causes: an Anaconda based Python distribution, or an incompatible version of pip/setuptools.

Many recent systems, specifically in the academic community, install Python in its incarnation as Anaconda Distribution. RP is not yet able to function in that environment. While support of Anaconda is planned in the near future, you will have to revert to a ‘normal’ Python distribution to use RP.

Python supports a large variety of module deployment paths: easy_install, setuptools and pip being the most prominent ones for non-compilable modules. RP only supports pip, and even for pip we do not attempt to keep up with its vivid evolution. If you encounter pip errors, please downgrade pip to version 1.4.1, via

$ pip install --upgrade pip==1.4.1

If you continue to encounter problems, please also fix the version of setuptools to 0.6c11 via

$ pip install --upgrade setuptools==0.6c11

Note

RADICAL-Pilot can be installed under Anaconda, although that mode is not tested as thoroughly compared to installation under non-Anaconda Python.

Mailing Lists

If you encounter any errors, please do not hesitate to contact us via the mailing list:

We also appreciate issues and bug reports via our public github tracker: